Most of the Amazon SES IP blacklisted by SpamCannibal
forums.aws.amazon.com
forums.aws.amazon.com
I am following all the best practises, hell, I even advocate them. It is just that these days it seems not to matter if you have SPF, Sender ID, DomainKey and DKIM, PTR, proper MX and even a normal to good IP reputation. There is still no guarantee what you will be able to reach the inbox of the likes of Gmail, Yahoo, Hotmail, etc.
I have been filling out huge forms for each and every major ISP for the past year because one or two users mark a newsletter as SPAM.
Conclusion: There is no common standard because every major ISP can set their own standards. This will eventually force everyone to use the same services worldwide.
Where's the freedom of choice here?
I'm one of them. Those newsletters are spam. I would never sign up for a newsletter and somehow I'm getting those too. If my intent was not to get the newsletter, it's unsolicited mail by definition, i.e. spam.
Stop spamming me and I'll stop flagging you. Period.
How not to be flagged as spam:
- There should be a checkbox clearly visible and it shouldn't be pre-checked.
- Your "kind" product reminders are obnoxious too and I'll flag them as spam as well. Did I ask you to remind me of your product? Nope. Unsolicited then.
- If you ToS say I agree to receive mail, guess what? I don't agree, I just want to try your product. I'll flag you in a breeze.
- Social reminders like Twitter's trending around me or people I might know? SPAM! I don't care if I can disable these, I didn't enable them.
- You want to offer me discounts but I didn't ask for them? Flagged!
- I submitted a paper to a conference and it got published? Dozens of "calls for papers" in my inbox. Flagged, flagged, flagged, flagged!
- Calling it a newsletter or adding a tiny "unsubscribe" link won't hide the fact that it's still spam. I didn't click subscribe, I shouldn't have to unsubscribe.
--
EDIT: Woah, this seems controversial. Lots of up- and down-votes.
Dear product owners, downvoting me here won't change the fact that me (and your fellow users) will still flag the shit out of your unsolicited mail. I guess it pays if you keep doing it, but you should direct your energy far from that downvote button and closer to "ways not to annoy my users".
Because the mail servers both share seem to allow sending unsolicited email, i.e. spam. You're the one responsible for terminating the spamming user.
If it costs you when your IP is blocked, insist on a deposit that's forfeit in the event of spamming.
Sadly, I've worked for shops with poor practices. How bad? Sending thousands of emails to domains which no longer existed (let alone stale accounts at existing mail service providers). Simply not a priority.
I also have catch-all addresses at my domain, so I register for things as SERVICE@mydomain, so I know exactly where spam is coming from.
Except I've been doing this for years, and 90% of garbage still just comes to my plain ol' gmail address that I've been using for over a decade, which suggests that businesses finally have wised up to the fact that sending garbage to everyone who puts their email address into a form is a bad, bad idea.
Then you're much more resilient than me :P
I used to do that but I thought, since they want to deliver their message, I'll deliver mine: "I don't want your unsolicited mail and I'll flag you without hesitation".
Apparently it works.
The difference is more important to me because I get catch-all mails for addresses on a domain that are no longer good (former employees) and they MAY have actually signed up to some of these things on purpose. This doesn't mean I won't both unsubscribe and also mark as spam completely useless e-mails from companies I don't care for. But I do notice that the RIDICULOUS volume of spam I get actually goes down sometimes when I do spend some time unsubscribing from what I find seems to be from legitimate companies, the frequently repeating contacts in the spam buckets. YMMV.
It's a pretty nifty interface that will automatically look up all relevant abuse contact emails for the ip ranges and let you file and register complaints with a single click.
1. You are not interested in their product or service (Obvious)
2. You read your mail, and it is not a throw-away box. (Non-Obvious)
Unsubscribe and your email address suddenly gets commercial value that it did not have before.
I just went through that a few weeks ago. I ordered something and the merchant decided to start spamming me with product info on a daily basis. I thought I unchecked the "spam me" button, but I guess I was wrong.
So I unsubscribed. And got two more emails.
They were required by law to stop sending me emails within 10 business days. They carefully waited exactly 10 business days. In the mean time, they sent me two emails a day in case I might change my mind.
After two days I started marking the messages as spam. Luckily they were kind enough to provide me with an opt-out confirmation email. I was ready to sue them when they stopped right at the boundary.
Just a pure jack-ass move by the company. I'll never buy from them again.
Edit: I am very confused by the downvotes.
Were the down-votes for something else? I honestly think some naming and shaming would be a good thing in situations like this. Especially in this day and age when complaining on twitter is more and more likely to get a response from the company in question.
It was spam, plain and simple. I purchased flowers because someone passed away and suddenly I was on the list of people who are just on the verge of buying chocolate covered strawberries, but just waiting for the right discount.
Thinking about it now, do chocolate covered strawberries and price conscious consumers even go hand in hand?
This is a great technique. I was receiving Nigerian-style spam to my santander@ e-mail address, but Santander ( a big Eurobank ) denied that they had leaked the address. They blamed me, stating that I must have entered that into some web form somewhere.
So I changed my address with them to santander_2014-03-20@ and guess what.. .within a couple of weeks spam came to that one too.
No subsequent response from them as to how these are being leaked / compromised. None of my other e-mail addresses of the form companyname@ are being spammed.
I'm gradually closing my Santander accounts, I just don't trust their IT systems and processes.
More frequently, I've been taking to /dev/nulling entire domains for crap. Usually recruiters (no, I'm not interested in your underwater basketweaving SEO marketing position in south-west Obscuristan, it's well outside my search and skill parameters).
I haven't got around to writing a good set of whitelisting scripts, but that's next.
Figuring out if I've subscribed to something, and/or how to MAKE IT STAHHHP really isn't worth my time.
No. You literally did agree.
That you don't like that it's a package deal, and you're exchanging getting their sales email for trying their product just makes you an asshole, but it doesn't make their message spam.
You established a business relationship with them, in which they told you ahead of time they'd do follow-up contact, and then you have the gall to complain about it in a way that damages the reputation of people who relay the messages you agreed to receive.
Way to make the world a worse place.
Yes, I agreed, but if I stop using your product, can't rescind the implicit contract and I am no longer interested, I will flag without compassion.
A product reminder is implicitly out of the terms of service, since I'm actually not using the service anymore.
Also: the other six points still stand.
> Way to make the world a worse place.
Seriously? I should've added that sentence to my post for dramatic effect too.
The issues is that spam filters learn. I fucking hate spammers, too, and like you I consider unsolicited emails to be spam. But when you flag mostly-legitimate, but still unwanted, emails as spam, gmail learns the wrong thing. Suddenly really legitimate emails get flagged as spam. My domain renewal emails from my registrar recently started getting filtered to my spam box, and I suspect it's due to users flagging any unwanted email as spam.
In my opinion, the best way to deal with these unsolicited emails is to use the unsubscribe link, delete the email, and then swear at them on Twitter or find their CEO's email and send them goatse or something. Fuck 'em. That way you get your revenge, and you don't muck up the spam filter for everyone.
This is totally absurd. My choices are "click one button" or "do a whole lot of bullshit that sounds like a lot of annoying work".
I hope you can understand why "one button" is taken more often than "raise hell on and offline".
The real solution is to separate spam and unwanted emails. Gmail and services need to add a separate button for non-spam unwanted emails, so they can categorize and learn about usage habits effectively.
But it's horrific, insanely bad UX design to create a flawed system then blame the users for using it naturally. I'm sorry but the user is not wrong, the system is wrong. The solution isn't "user training", it's "system redesign".
If the system were designed correctly, users would naturally gravitate to the correct option without training. That's good UX. Until then, it's perfectly acceptable to use whatever tools are available to achieve the desired outcome. That's software for you.
You're punishing the phone company because a company you had a business relationship with can't magically read your mind that you don't want further calls, after you agreed to a couple sales calls in exchange for a product demo.
That makes you an asshole.
You couldn't even quote me. You couldn't even use my words. You just invented a pathetic little fantasy and then attacked ME directly based on your fantasy world.
Pathetic, dude, pathetic.
If you genuinely did not give permission to contact you, sure, go ahead and flag as spam. If you regret giving permission (maybe because you didn't really want to give it, maybe because the communication you're getting isn't what you'd hoped, whatever), the sensible thing to do is just unsubscribe.
A follow-up contact and putting someone's email address into an automated system that spits out boilerplate junk are two very different things.
Any company that relies on the latter clearly doesn't give a flying fuck about their reputation.
The way I see it, when you send me an email, there are two perspectives at play. Yours as the sender, and mine, the receiver.
You do not see the email as spam, according to you I signed up for this explicitly when agreeing to your ToS (for example), it was requested and is not spam.
I do see the email as spam. I did not make the conscious decision to receive email from you about your products or anything else: to me, I clicked a box that said I read and agree to your ToS in order to get your product.
Terms Of Service themselves are a discussion for another day, my point is we both are fully aware that nearly nobody reads the ToS and that you're leveraging that to send me emails that we both know I do not want (if I wanted them you wouldn't be resorting to these tactics, IMO the fact that you are is a tacit admission of guilt).
Here's the crucial part. When it arrives in my inbox, the choice is mine about whether or not it's spam. You have no say in the matter, ToS or not. This is a matter of perspective, and my perspective is that the email is spam.
Argue it if you want to, but understand what you're arguing against is perspective and that I don't share yours.
Edit: Typo
Downvotes are not for disagreement, disagreeing with somebody is a natural part of that human process that we call conversation. Disagreeing is good. This is not reddit where people upvote what they like and downvote what they don't. Use downvotes for flagging inappropriate comments that do not contribute to improving the quality of the site, not as a personal argumentative weapon.
Yes, I get that you don't want to be responsible for what you agree to with other people.
However, you punish the middle man - the mail carrier - because you regret your own decisions you admit were made in ignorance.
> Argue it if you want to, but understand what you're arguing against is perspective and that I don't share yours.
I think you're simply unreasonable: you're whining about getting a sales message from someone you proactively established a business relationship with and turned your contact information over to, and that they disclosed your information would be used that way.
In no way was that message unsolicited. You just wish you could get the product without even having to pay the meager amount of receiving sales literature in return.
I think that makes you an asshole, because you're punishing people for conducting reasonable business rather than taking some ownership of your behavior and simply unsubscribing.
> down-voted you
This bolsters my view that you're largely just an asshole: you're trying to punish my internet points or hide my comment because you don't agree with me, while you yourself admit that there's nothing in my comment but a difference of opinion.
So, really, I wish mail carriers would just ignore people like you when they submit spam reports - since you admit you're not using it how it's intended, but to flag solicited emails you agreed to receive, which damages the reputation of the mail relay, even though they're not doing anything wrong. They're just delivering requested mail.
It's like you trying to get the phone company that a second company uses to call you in trouble because they had the audacity to connect a phone call after you gave your number to that second company and told them it was okay to call you at the end of your free trial.
I really wish someone could present a argument for your view that didn't just make the person sound wildly entitled and assholish.
It sounds to me like neither of us is willing to discontinue what the other side sees as deceitful behaviour.
On your side you assert that my agreeing to ToS is sufficient to start sending me "solicited" email, and on my side I assert that the fact that you have to hide the opt-in inside the ToS is evidence that your emails are spam.
I do concede that I could have carried on with our conversation without down-voting you, that wasn't necessary to make my point.
Other comments in the thread point out that a "Unwanted, but not spam" button could be useful, I think that's a great idea but wonder if it could be taken one step further. A spam filter that monitors who reports what email as spam and assigns them a rating based on what they report as spam.
Eg. I would have a high rating because anything I did not explicitly request is spam. You may have a low rating because you are much more lenient with your use of the Is-Spam button. This could then allow users of that service to set which rating to use when filtering spam.
Given the widely varying differences of opinion on this topic I can't help but wonder if the other commenters are correct about this being a UX issue instead of a technical one.
It requires 2 clicks to report something as spam and 4 clicks to create a filter which automatically deletes messages from a particular sender (or routes them in a way of your choosing; can also be used to selectively stop messages, eg, receiving bills without receiving ads; option is in the drop down menu).
I can't help but feel like you're saying you should be allowed to file harassment reports against the people standing behind sample booths, since you didn't explicitly ask them to talk to you when you grabbed a sample from the table, and well, harassment reports are just so much easier to file than asking them not to talk to you! (Okay, not actually true, but would be the analogous thing.)
I suppose there isn't a lot more to say, but I just want to ask this point blank one time to be sure I really understand what you're trying to say (even if I don't agree): are you really saying that it's entirely unexpected that a company which you're getting a sample or service from sends you a sales message and that you think the best response is to report them for harassment (in the process, attacking the reputation of the middle man in the communication for enabling harassment) rather than just informing them directly that you don't want further messages?
Edit: Corrected click count to account for menu hiding; tidied up comment a bit.
If there is a check-box that's pre-checked and all I have to do is un-check that box as I'm signing up to opt-out, I respect the company for being up-front about the choice and will un-check the box. On the other hand, if they do anything I consider "shifty" like trying to hide the opt-in anywhere (eg in ToS), then the answer to your question is yes. I would not expect those emails so in my opinion they are unsolicited, at best.
Just because someone shoved a statement they get to email me somewhere in the small novel I'm expected to read -- and your argument is disingenuous because you know damn well nobody reads those things -- doesn't mean I actually, you know, agreed.
A recent example: those dbags at ziprecruiter decided that, since I applied to a job at a single company that used them, they should now email me daily lists of jobs I may like. Was that buried in a tos somewhere? Probably. By any reasonable usage of the phrase, though, I in no sense opted in. And it's not my responsibility to find their unsubscribe link and figure out what username/password I used. spam
- Yes: not spam.
- No: spam.
And, since I never solicit promotional email to my personal address, my method is 100% accurate.
In my opinion the companies sending unsolicited promotional email are the ones gaming the system taking advantage of the "well this might be unsolicited but it's not v14gr/\" grey area.
edit: I guess spam is by definition commercial email. Still, I think it's possible to have a business relationship with a company where it's acceptable for them to send an occasional email that you didn't specifically request.
It is unlikely that the business is your friend ;-).
That's only some definitions. There are more hardcore people who'd view anything they didn't explicitly request as 'spam', just some of it originating from people they know vs companies they know.
They have proven themselves to me and I like them on an irrationally human level.
What bothers me, however, is when the other 99.9% of the brands I interact with automatically assume that they are the 0.1%.
Yes, it's possible and acceptable to have that kind of relationship between a client/business, but the problem becomes when a brand believes that they can control that relationship, or define it themselves, or simply assume that it exists.
SPAM = UCE + UBE.
UCE = Unsolicited Commercial Mail
UBE = Unsolicited Bulk Mail
What's common? Unsolicited.
A week or two ago I sent hundreds of messages to my customers, warning about Heartbleed and noting what measures we'd taken against it.
That mail was unsolicited. Not overtly solicited, anyway. Sure, it falls under the "we may contact you from time to time about yadda yadda" but people only know that if they actually read our Terms & Conditions. A normal person with an important mailbox and a convenient spam-button doesn't always have time for such careful consideration and nuance.
Heck, to make sure the mail reached the recipients, and to avoid getting my domain flagged (and this is for a very legitimate message!) I used a 3rd-party mail service. It sucks that I even need to do that just to communicate with my customers.
I suspect one or a few of my customers may have marked that message as spam, too. Because, whatever it was (didn't read, just glanced at subject) they didn't expect it so it must be spam.
Hahaha! YES! I have a personal vendetta against unsolicited email.
If you send me an email that I am not expecting, then I will flag it as "unsolicited".
I do not care if you provide other mechanisms for handling unsolicited email. I do not care if you used a dark pattern to technically ask me to solicit the email without my conscious knowledge.
The way you call it vindictive is empowering and enlightening.
It's your way of calling it unfair, of attacking the person as immoral and undermining the point of their behavior. You're trivializing them as petty. Instead of saying "I understand why someone might be unhappy at me sending them email they do not want" you say "it is vindictive and petty for someone to treat unwanted advertising from for-profit businesses this way". What a sick joke.
So you know what? GOOD! If "vindictive" means "control over your inbox" then I AM VINDICTIVE. I will PUNISH anyone who gets into my inbox without my approval. Email is WAR and I am fighting for Inbox Zero. Send those emails my way lightly ... tread carefully with that send button, because you might just find your messages are unwanted and end up clearly marked as unwanted.
Insult us users all you want, trivialize us all you want, attack our behavior all you want: I DO NOT WANT UNSOLICITED EMAIL and will HAPPILY and "vindictively" (lol) mark unwanted email as "unsolicited".
And again: I don't send email from my service; I don't even send receipts... so most i your rant is fundamentally mistargetted. I am a user who doesn't appreciate people's emotional responses to something that isn't really fixable anyway making email fundamentally more complicated as a protocol, breaking use cases like shared mailing lists (see the recent issues with Yahoo DMARC on the IETF list), and mis-training spam filters. If you ask me about real-life "actual" criminals I would frankly have similar responses against people who prefer vigilante, vindictive punishment for behaviors they dislike :/.
As the OP said, this is just reality: people have different criteria for reporting something as spam. I could even imagine malicious users flooding false positives using bots in efforts to break spam filtering. The state of spam is so much better than it was 10-15 years ago, but it's not yet a completely solved problem.
In short, this person isn't breaking spam filtering all on his/her own--his/her use of flagging may even be arguably justified (if more liberal than average.) He/she just represents part of the problem space.
These are not "false positives". Just because they are legal, and for some even common business practices, it's all spam and always has been.
Unethical business practices are causing the problem, not the few people that resist them.
So you don't agree to the terms you just read before giving out your email address. That's insane! It's like smiling and saying to a guy, "Here's my number", and not saying: "But if you call me, for any reason, I will report you for harrassment", without any indication that this is how you feel. Absolutely insane.
Why would you give out your email address with that attitude? "Here is my email address, but the only thing you should use it for is to let me ruin your communications with others who agree to the same thing I just agreed to, if I receive anything I might conceivably have given it to you for."
If you don't want to get ANYTHING, EVER, then what is the purpose of giving out a means to communicate with you??? What did you think you are doing by giving out an email address??
It's just so bizarre. Do the world a favor and register an email address for what you consider spam, which is everything.
Then never look at it again, while that email address gets invitations and calls to publish in journals, updates on the product offerings you are interested in, informative newsletters, free money on things that you're already spending on, and so forth... while you get nothing.
Boy you sure are showing them though!
Do I agree to get emails sent to me from <random company> as long as they're product-related and I'm using their service? Yes I do. Will I mark these as spam? No I won't.
Do I agree to getting promotional emails or reminder emails when I stopped using their service months ago? No I don't. Will I mark these as spam? Yes I will.
What you're describing is just so different from real spam.
Not at all, check this other post of mine on why I think it's still spam: https://news.ycombinator.com/item?id=7621722
I'm curious though - after you give out your email address, which you don't consider "soliciting" in this sense but I do, how do you imagine "soliciting" any specific communication?
Nothing can possibly be 'solicited' under your definition.
I just don't get why you even give out an email address, if anything you get thereafter is still considered unsolicited.
And then states pretty clearly why I'm not in his "EDIT" which you chose to ignore.
Because you pretty much have to buy anything online nowadays.
My personal solution is to use the '+' trick to give customized emails out to everyone and then to block them at the SMTP level in my /etc/mail/access file if they don't stop sending mails after the 1st unsubscribe attempt.
Absolutely, and what's worse is that people marking ham as spam means that email providers can't take a spam report as seriously as they otherwise might, thus reducing the potential benefits of a 'report as spam' feature.
So we all get more spam because some people misuse 'report as spam'.
Maybe downvoter didn't get that you're using ham as meaning non-spam.
If you signed up for the ham, and it has an unsubscribe, it's absolutely immoral to flag it and deprive other people of the benefit.
I wonder what would happen to false spam reports like the GP's if every email field had a button next to it, "Don't ever email me, I don't know why I'm even giving you this." (i.e. "I'm insane.")
EDIT: cleared up that the ToS isn't what establishes the relationship - the fact that you're giving them your email does.
But I know from industry experience that some people who explicitly sign up to mailing lists then go on to report those emails as spam, and this dilutes the effectiveness of 'report as spam' features.
An unsubscribe button = "now stop emailing me." i.e. the opposite action of signing up and giving out an email.
flagging the kinds of things OP talks about after giving out your email address is ridiculous. Giving out your email address is opting in to communication initiated by the other side, without further requests - otherwise you wouldn't be giving out your email address, they would be giving you theirs: "Email us to get a reply with our newsletter" or whatever else the OP imagines in this bizarro-world where you opt in to a specific communication.
And that's why I wouldn't flag them, because I solicited the email.
But I wasn't talking about that case. And you know it. Please stop.
To this day we have never sent a marketing email, or even non-transactional email - mainly because we suck at marketing, so it's not like they got confused about which email...
^ We do audio transcription, a customer sends an mp3 in, we have someone listen and type it up, and we emailed the customer the results. These days you can collect the transcripts from the website, but for years you couldn't.
I routinely deal with people complaining about not getting a newsletter they requested after they marked it as spam (once they do that, we are required by agreement with the ISP to not send them any more messages ever).
So she and I had two questions that we never resolved. First, is it really true that hitting the "Spam" button on a site like Yahoo or Gmail informs the sender of your email address? (Doesn't this lead to the usual concerns about confirming a valid address?) And second, what is a user supposed to do in the case of an accidental bump of the "Spam" button? Is there really no way to undo the damage (both to the sender and to the willing recipient)?
Yes. The feedback loop shows the sender who marked the mail as spam.
> What is a user supposed to do in the case of an accidental bump of the "Spam" button? Is there really no way to undo the damage (both to the sender and to the willing recipient)?
There is no good solution, it kind of sucks for both the sender and the receiver. When you click that spam button, intentional or not, I can no longer send you email until going through a sometimes laborious process of working with the email service to get the email unblacklisted. If you really want to receive email again, send an email to the company/person and let them know you accidentally clicked the spam button so they can work it out with the email provider.
Gmail handles this the best IMO as it gives the user ~5 seconds to click "undo" before reporting the spam. As far as I can tell most other email clients are instantaneous.
I think the fear of "confirming an email address is valid" is unfounded. A list of valid email addresses of people who actively report messages they don't want as spam is not worth anything to a spammer and it would be illegal for a US company to do anything with the list of unsubscribes or spam reports anyway.
I'm not aware of any simple and universal solution to "undo" an accidental spam button click.
But if we're talking about newsletters your don't remember signing up for or marketing emails or invites to a new social network, the unsubscribe usually works and will almost never harm you. AFAIK it's a myth that there's this huge black market for "valid" email addresses. Spammers don't want a list of people who click unsubscribe; they ain't gonna buy anything. There are way easier ways to find email addresses on the internet. And it would be crazy (and possibly illegal) for any sort of semi-legit company to sell their unsubscribe list.
Anything send by an actual company -- especially anything sent through any of the major email providers -- will almost certainly have a working unsubscribe. It's almost impossible to send a message in MailChimp without a working unsubscribe.
(edit: Replaced "parent poster" with username of poster, to avoid confusion.)
I will henceforth carefully choose my pronouns.
Its people like you that make it so hard to send emails to people, even when they agree to receive emails.
You have no idea if the parent sends out a single unsolicited email. All you saw was "newsletter" and you flew off the handle.
Not that you're alone; this happens in every single HN thread about email marketing, and is in my opinion one of the worst killers of signal to noise ratio on HN. We love to talk about "growth hacking" but every mention of email marketing is a race for HN comments denouncing all email as spam.
The fact of the matter is that even the most carefully run, innocuous, double opt-in email newsletter is occasionally reported as spam. I think it is because paranoid tech folks have spent years telling people to never use an unsubscribe link because it just makes the spam worse.
But if it's an email you actually signed up for, the unsubscribe link is the correct and appropriate way to unsubscribe.
I think spam reporting is made too easy. I also hate over-sharing sites but not as much as I hate true spam. Unsub should be one click, spam reporting should be four clicks. Or hopefully gmail gives you an internal "spam report reliability" score so it knows you're a more trigger happy than I am.
And no, I've never sent a product mass mail out ever.
* Meetup Messenger A fun, informative newsletter to Organizers and anyone interested in running a Meetup
* Meetup HQ Announcements Get promotional emails from Meetup HQ
* Weekly Personal Calendar A once-weekly email of your Meetups and top Meetups in your area
* New Meetup Group Announcements Get email alerts about new Meetup Groups that match your interests
* Meetup Surveys An occasional email survey asking your opinion about new or existing features, your Meetup Group(s), Meetup sponsorships, and other requests for feedback.
* Greetings Send me an email when somebody posts a Greeting
So that's a minimum of a weekly email that I don't care about, plus promotional emails that I definitely don't care about: Joy unconfined. No mention of all these is made during the signup process that I can recall. Logging in with Facebook instead and letting them mine my FB friends graph instead is looking tempting frankly.
I can totally understand some people responding to this kind of 'dark pattern' email signup by block marking everything that isn't directly relevant to what they signed up for in the first place as spam. Marking it as spam means they never have to look at it ever again & is practically effortless on their part.
Sure, they could log back into the website in question (if they can find the password / can be bothered to log into Facebook) and faff around looking for the email subscription settings, but marking it as spam in GMail is quicker & easier: it's win all round from the user's point of view.
The end-users choose which email providers (Yahoo, Gmail, Hotmail) they want to use. The email providers are motivated to provide a good user experience, which includes blocking unwanted email because they make their money through user engagement.
The key to delivering into the inbox is sending mail that your recipients both want and expect. Provide a good user experience, and you'll build a good reputation. Push the limits (for example, use a "pre checked" checkbox on an order confirmation page to put people on your sales mailing list) and you'll be putting out email delivery fires all of the time.
Most of the things that you describe (SPF, SenderID, DKIM, PTR, MX) are all technical requirements, which are just the baseline for delivery. These are required, but any sender of unsolicited email can configure them. They don't earn you access to the inbox. Just like properly formatted HTML does not earn you great SEO results.
I like to break email delivery down into four areas:
* who -- send to people who requested your email and are expecting it
* what -- send something of value to these people
* technical foundation -- (SPF, SenderID, DKIM, Feedback Loops, etc.) required, but having it does not give you any points
* monitoring -- (open ratio, complaint ratio, ISP response codes) you need to know when something goes wrong
(My company, www.drh.net, has been providing email server software, services, and deliverability consulting for over 10 years.)
---
[edit; added the below]
Another way the freedom-of-choice is in the hands of the end users is this: the big ISPs (yahoo, gmail, hotmail) make most of their filtering decisions based off of end-user behavioral data.
For example:
* what percentage of your email is opened
* what percentage of your email is complained about (the "this is spam" button)
* what percentage of your email is deleted without reading
* how long is your email read
* how much is forwarded
* how much is replied to
* what percentage of your email that was placed in the Spam folder when seen by the user received a click on the "this is not Spam" button.
This is the end-users voting on if they want your email or not. This isn't the entire email deliverability equation, but it's a huge part of it.
The ISPs treat this data so importantly because: (a) it's hard to game unlike content filtering, and (b) it directly correlates to good user experience which they want to provide.
[edit to make bulleted list look right]
> Provide a good user experience, and you'll build a good
> reputation. Push the limits (for example, use a "pre
> checked" checkbox on an order confirmation page to put
> people on your sales mailing list) and you'll be putting
> out email delivery fires all of the time.
Mail recipients are not perfect. They forget that they signed up for things. They accidentally click the "spam" flag on their messages. They get lazy and instead of unsubscribing they click the spam flag.
The real culprit here is that email messages rely on blacklists and not whitelists, i.e. recipients are required to give all senders full access and then block them when they misbehave, instead of giving them no access and giving them more access as they build trust.
So: What would it take to implement email whitelists across the industry?
We have one customer that's cleanest-of-the-clean (confirmed opt-in, valuable content, solid brand) sending 600k emails/day, and we see hundreds of spam reports. But their email gets delivered to the Inbox.
If you're a good actor with a solid technical setup you're still going to have an occasional delivery problem. This is why monitoring is so crucial. But you're not going to be putting out fires left-and-right, which is what it sounded like what _asciiker_ was saying he is doing.
> Conclusion: There is no common standard because every
> major ISP can set their own standards. This will
> eventually force everyone to use the same services
> worldwide.
>
> Where's the freedom of choice [of ESP] here?
email users want and expect
As an e-mail services provider, I cannot or should inspect what my customers are sending. I can suspend them due to complaints of abuse but the damage is already done.
Same goes for tracking. I still say, block domain names, not IPs..
As an ESP, since you are letting customers send through your IP space, then a bad-apple can hurt the delivery of your other clients.
This is one of the big jobs that an ESP has. MailChimp, for example, has invested a ton of effort into detecting bad-apples as early as possible. (There are some really neat big-data techniques.) This is also why SES requires that you start with a smaller quota and build-up.
Some techniques:
* manually reviewing new clients before they send
* giving a new client a limited sending quota, so they build reputation with you over some time
* detect clients/campaigns with high complaints, high bounces, or low opens and take compliance action
* detect a partially-sent campaign with a high bounce rate and suspend it
* don't give any client an unlimited sending quota, so they can't hurt you too badly
> I still say, block domain names, not IPs..
There's a minimum amount of mail volume required to build a reputation. Many of your clients might not have this so they benefit from being lumped-in on an IP reputation.
I don't think IP blocking will ever go away, as it's an effective technique. The threat of an IP block also places some reasonable pressure on ESPs to police their client base.
If you mean that on the same server the e-mail IP and server hostname will be the same for all the domains, then yes I understand.
But filtering can be changed to achieve domain level validation, not IP. It is done already on our servers.
Problem is, there are plenty of large sites out there who aren't so conscientious, and you can keep up that level of commitment for only so long.
There is no one on this entire PLANET that hates spam more than I do, and for a time I could rightfully claim that there was no other single person on this planet who had done more to fight spam than I had. But even I have my limits. I haven't been active in CAUCE or any other anti-spam effort for many years, and I haven't even been an active mail system administrator for a few years.
As one of the members of the Postmaster Team for python.org (currently inactive), I have seriously hated some of the stupid shit that has been done by the likes of Google, hotmail, and Yahoo! I've been sorely tempted to just ban them outright, because of whatever might be their latest stupidity. But it hasn't happened -- At least, not yet.
SpamCannibal can cause the originating mail server to get caught in a 'tarpit' by slowing it down. Given the AWS CUSTOMER was sending a significant amount of measurable email to a given destination server (which was running SpamCannibal) it's possible the sending servers are being slowed down. In that particular scenario, that would affect open rate over a short period of time.
I run a fairly large website, and I block all traffic from the likes of Amazon AWS because it's full of dodgy bastards who think they're entitled to run however many HTTP requests they like. Webmasters, look at your web logs. Don't be surprised if the majority of hits are coming from bots pretending to be web browsers.
As far as I can tell, 90% of all that crap still comes from the shitty cheap home user ISP networks and el-cheapo web-hosting services.
Never seen much bot traffic from AWS.
(The bot that is currently pissing me off is Netcraft. The practice of just "guessing" domains and then firing http-requests at them is annoying.)
And see here how to add them to your checks (and rank them accordingly) if you're using Postfix: http://www.postfix.org/POSTSCREEN_README.html
Bitcoin or some other cryptocurrency would be ideal for facilitating micro-transactions like this. Interestingly, the Hashcash concept was originally designed to fight spam, and later became one of the important ideas that made the invention of Bitcoin possible: http://en.wikipedia.org/wiki/Hashcash
I sign up for a mailing list.
The mailing list mail server contacts my mail server.
My mail server presents to me a request for free delivery.
I approve it.
My mail server gives the mailing list server a secret token.
Each mailing list message includes proof that the message has been blessed by the sender and that the sending server knows the token.
Right now I give out tagged addresses to most vendors, and I know others do it for lists. When a tag goes bad, I just route any further mail to my spam trainer.
Dunno, I feel like we should just charge for mailing lists too. Or use usenet!
Even with a charging scheme, some spam will still happen. Give the amount of BTC stolen so far, and given the number of compromised computers that could be used to generate cash, looks like there will be plenty of money to spend on sufficiently profitable spam.
I've seen some suggestions that instead of using money you could use some proof-of-work computation. This is something we could scale the difficulty factor of as computing power increased. Something that takes, say, an average computer 30 seconds (or longer) to calculate. It will attach this proof-of-work to the email.
There are some interesting (I'm not sure "compelling" or "strong" is necessarily the case) arguments to be made for enabling open relays and other forms of unauthenticated messaging. John Gilmore of EFF has fought that battle for a long time, and still runs an open relay on toad.com.
Signing and authentication measures (particularly on header data) have to be both standard and quick to process.
Methods which increase the costs of delivery -- pacing receipt rates from a given IP or block, can help. Being able to specify receipt priorities: high for IPs and ranges with which frequent legitimate business is transacted, very slow for most others, would also be useful. Along with a lot of built-in support for this.
Killing :80 and moving to entirely secured ports wouldn't be a bad move either.
By "secure port", I mean forcing encryption of all over-the-wire traffic. It's happening now in many cases with STARTTLS (modulo utter brokenness of the CA and SSL/TLS systems), but that's still only opportunistic.
And of course, encrypting payloads would be vastly preferable. Headers as well other than absolutely required for delivery.
Splitting these up converts one impossible problem into a tricky but mostly solved problem (delivery) and a hard problem that has to be addressed anyway (internet identity)
There is sadly not much options. Either I can accept more spam, or use blacklist and put the control of the filtering in the hands of a third-party with none of the responsibility attached.
"unsubscribe" links vary in position, language and visibility in various clients. Making something beyond "this is spam" part of most mail clients, perhaps with reporting back to the originating sender, would help.
http://gmailblog.blogspot.com/2009/07/unsubscribing-made-eas...
So many don't dare to use such links and rather click on spam.
The only solution could be some "trusted" functionality that goes via the own mail provider of the receiver. But of course the mail provider can not simply send information to the sender of the eMail .... So the thing gets complicated. As much I learned, for spam clicks there is something like that available ... some kind of trusted feedback chain that gives information to trusted senders, that some mails where labeled as spam. Thus those senders can (indirectly) adopt their eMail campaigns.
That's only true of the dubious "viagra" style spam, where they got your name from a list. I don't think those even bother with "unsubscribe" links any more. I only see unsubscribe links from places where I've had to give my email up to buy something or sign up to a site. Those are generally legit and most of the techy/startup web sites will unsubscribe you immediately.
The next tier are the sites that unsubscribe you but take more than a week and will keep spamming their dumb newsletter in the meantime.
The final ones are either broken by stupidity (it's amazing how many web developers cannot grasp that "+" is a legit email character), or willfulness and will keep spamming no matter what. I block these at the SMTP level with 503 messages (usually containing some personal insults and swearing) as soon as they "RCPT TO" the unique email address I gave them.
I'd call it "Sender Reputation Checking as a Service". Where said service is paid for by the sender, but provided to the email recipient. Anyone can send email directly; but knowing that email has been sent through SES and Amazon hasn't killed the account yet provides a greater degree of trustworthiness.
In a sense, it's like a bond rating service.
On a more serious note, it seems like the "greater degree of trustworthiness" is only very slightly greater. SES might be better than some server in a domain nobody ever heard of, but it's still not as good as a provider with a long history of responsible email use. Many people can and do block SES and its ilk, as is the subject of this story, because the aggregate amount of spam is so great even if the individual spammers are transient (like they care).
Amazon could raise the bar, thus raising their own reputation and thus making the service more valuable to those who can still afford/qualify to use it. It's probably just not worth their while to do so. I'm not even criticizing them for that. I'm just observing that online business has a shady side, and Amazon isn't afraid to partake.
Google's Play app does the same thing for a bit of their content. Certain headers get localized, despite everything else in the app being English. Netflix has the same problem, and then to further add insult, they send you to non-English phone numbers for support.
The most annoying thing is that someone probably got a raise for these "features".
If an otherwise normal email happens to come from a blacklisted computer, it'll still have a chance to get through, which is the correct thing to do, in my opinion.
I still believe that everyone has the right to run their own SMTP server, and I dislike that so many places blacklist someone just because they are on a cable modem.
(1) keeping in mind that the definitions of "spam" can be quite subjective
If you're running a bulk mailing server you simply must respond to abuse reports, otherwise your service will get blacklisted and be essentially useless. Other providers such as mailchimp are much more proactive about getting rid of spammers.
By appending in the Posfix configuration file line smtpd_recipient_restrictions = ... spamcop and spamhaus , spam decreases in like 95% without even touching your server further (spamassassin I'm looking at you).
I you add greylisting you get rid virtually of all of spam.
I'm all for cleaning up email.
I've never had a case of a false positive complain but yes, they can happen and I don't terribly mind because that means the sender's mail server is blacklisted and they should know.
Also in any case my recommendation is to use a 3rd party mail service, for deliverability reasons mostly.
Point taken, but I've been on the receiving end of what I would consider false-positive blocks by Spamhaus & co. They sometimes have policies about what's considered spam that I don't think all their end users would agree with. I've been blocked for having an IP address on the same provider as someone else who allegedly advertised their website via spam. If you're running a mailing list, dealing with the anti-spam stuff is at least as big a problem as the spam it was supposed to solve.
I don’t really like greylisting, as it takes longer for email to get through. What did help without any perceivable loss whatsoever is being extensively strict about SMTP specifications (e.g. proper hostnames in EHLO) and things like PTR records. I really like watching these ‘5xx: Client <something> rejected’ flying by in my mail.info :)
Not sure about "vague comment"s but anyone who has run an email server, used an rbl/rhsbl, and followed the logs <http://www.postconf.com/docs/spamrep/> would say the same. Having done so for years and run reports on dozens of servers daily it is clear that blacklists are the most effective form of spam blocking, by at least an order of magnitude.
Could be a bug in your web browser. The first sentence is a quote as indicated by the ">" character at the beginning of the line.
I got that part, I'm talking about your first sentence.
... anyone who has run an email server... would say the same.
This reads as though you agree that "IP blacklists are a waste of everyone's time" as OP said. And maybe you do (and that's fine) - I'm just unclear given your second sentence.