The Hostile Email Landscape
liminality.xyz
liminality.xyz
Greylisting still works amazingly well. With a long, long whitelist and greylisting plus DNSBL, I don't even bother running a spam filter, since the little bit of spam and emails from new senders ends up in its own directory as it came from a non-whitelisted sender.
Comcast finally started blocking residential mail server ports inbound a few years ago, so I had to migrate to a smarthost environment using a VPS as email server for $15/yr.[1]
Last year for a few months, Gmail was dropping everything I sent into the spam folder, even after recipients were marking it not spam. I eventually discovered the "Authentication-Results:" header that Gmail adds to every inbound message. It is under the "Show Original" dropdown menu. That showed that I "hadn't changed anything"(!) on my mail server, but suddenly Gmail was connecting to my mail server over an IPv6 interface, and I had never bothered to put the IPv6 block into the SPF record. Gmail was nice enough to explain exactly what it didn't like about those emails.
[1] http://lowendbox.com/blog/top-provider-poll-2014-q3-the-resu...
Any good tips on this section in particular? If I'm running my own mail server, how would I get started making sure this is in order?
I also have a small script that puts a new sender on my whitelist of sender email addresses. My whitelist is 12000+ lines right now, collected over many years. Procmail sorts to mailing lists and vendor folders, and finally puts things that are not on the whitelist into a "possible spam" folder. From the five or ten items a day, it is easy to spot legitimate emails and I add those to the whitelist. The majority of spam is blocked by the combination of greylisting and DNSBL lists, as the delay of greylisting (ten minutes for me) is enough for them to make the blackhole list, if they happen to ever attempt delivery again.
I was thinking recently that I should be collecting statistics on the use of a lot of those aliases and whitelisted emails, and maybe start garbage collecting my lists.
There are various reputation reports and services that can tell you how your mail is doing in the major ISPs, but a lot of those require higher traffic than a personal or small business generates. There is one service, DMARC[1], that is free and can give you some visibility into how email from your domain is being processed. I put the txt record in my DNS, and Google, Facebook, Comcast, Yahoo, Fastmail, and a few others send me reports about email they have processed from my domain. It's not that interesting at the moment because things are working, but it might help to debug issues if your email was being rejected. At least I see a few spammers are trying to use my domain from their servers.
Whoa, I think you just solved the problem I've been having with my outbound e-mail for the last two months! Thanks so much!
It is very easy to set up.
The full setup for inbound mail is - OpenBSD spamd (greylisting) -> internal postfix
Outbound is - internal postfix -> VPS postfix
Nitpick: when sending email, the sending mail server establishes a connection to the receiving server, not vice versa. The sender is responsible for looking up the domain name's MX record, choosing a hostname from those listed, initiating a connection, establishing an SMTP conversation, and negotiating protocol enhancements like SSL.
That said, I could see a change in behavior like this resulting from action on the receiver's side. Though the sending server makes the choice of how to connect, that choice can be influenced by the presentation of the MX record. I haven't specifically run into this before, but I could see this happening if, for example, the MX record advertised a host with an A record first (IPv4), and then was changed to list a host with only an AAAA record first (IPv6).
There's quite a lot of arcana in the email space that's difficult to manage without careful, deliberate effort. Sending email is harder than it should be, and a lot of that difficulty results from anti-spam or general anti-abuse efforts. If it was easy to bring a server up and begin sending with a new domain name and IP address while successfully reaching the inbox, then spammers would do exactly that. So new identities are put through the wringer to prove they're legitimate. Well-known platforms have an easier time sending email because they can relay traffic from infrastructure that the ISPs already trust; and in turn they maintain that trust by keeping abuse off their platform.
Also, do you just have one server or a backup as well?
Using postfix + dovecot(imap) + thunderbird. 1 user, set to check mail every 9 minutes. Not even sure if I have IDLE on my server (unless it's on by default).
The SolusVM control panel shows 15-30 bytes/second (in+out) when it's idling. Now add how much your email weights and you got your bandwidth.
Once upon a time, to send email, you needed to use SMTP. Now, you must use SMTP, from a IP block that isn't categorized as residential, and which has never before had any association with outgoing spam, and you also must implement several ad-hoc identification protocols like SPF and reverse DNS. You should also use a domain name that you've owned for some time and which is not expiring soon. Every system to which you want to send mail will give different weights to all these signals. If they don't like you, their behavior is to report successful delivery and then silently hide messages from their intended recipients.
Spam is no fun, but the present situation is pretty weak too.
For instance, several months back some of our account holders suddenly stopped receiving important account info as well as newsletters from us. Tracked it down to a third party filtering service used by various email providers. Contacted them and they indicated that we were sending to honeypot addresses and/or doing other things that landed us on their list. We refuted all of their assertions (all of our recipients are double out-in, etc.) But , they insisted, as if their processes were infallible and refused to remove us.
Here they were, a third party with whom we had no agreement, yet they were interfering with our ability to do business (and profiting from it). We also had paid advertising in the newsletters per agreements with advertisers. Consulted counsel and the next step was to send a cease and desist, followed by an injunction on the grounds of tortious interference. Then, just as suddenly, our emails started going through again.
So, all of this, just to send email.
Yet, I still have to deal with a ton of spam personally when wading through my quarantine folder for routine false positives. Current processes are harder on the good guys than the spammers.
Dangerous path to go down; because so many spammers throw around legal threats too, doing so can get you on a different set of blacklists.
Filing for injunctive relief isn't a threat. It's legal action. Unlike spammers, we could easily demonstrate our legitimacy and best practices, etc. They would be compelled to remove us and not retaliate if they are a legit business and care about the integrity of their service for their own reputation and customers.
If, however, that does not describe them, then the outcome would seem to be bleak in any case.
Sending messages in the millions to domains which have been inactive and unregistered for over a decade, for example, suggests a certain lack of diligence in bounce-tracking or list-cleaning.
The emails had been initially legitimately added, but clients and/or email providers had long since gone out of business. Re-validating email every so often isn't a bad idea. Especially if you're relying on it for other purposes (e.g., account recovery).
For instance, our ESP detects hard bounces and auto removes them from our lists. We also use Webhooks to update internally on unsubscribes, etc.
For transactional emails, our ESP auto-adds bounces to an outbound block list, so any future attempts to send fail. We only send those to accounts with recent activity anyway, but either way, one bounce stops future sends.
Our customers have email accounts with providers like Comcast. Comcast uses a third-party filtering service to handle spam.
So, our customers had no knowledge of the third-party, and some contacted our support staff to question why they weren't receiving our emails.
And, of course we attempted to identify the problem with the third party. If you read more carefully, you'll find that in my comment. Only after they insisted their classification was correct and refused to remove us from the list did we explore our legal options.
My point here is that the recipients of the email and the senders of the email have very different ideas as to what constitutes a "correct evaluation." - and in this case, it's the recipient who matters, so complaints from senders of email should be simply ignored by spam-filtering companies. Now, if the recipients can't complain, that's a problem, but it's not like people don't have a choice of email providers.
>recipients of the email and the senders of the email have very different ideas
What does that mean? They sign up for our service and expect to get e-mails from us, which they don't receive. We have the same "idea", but the third-party is interfering with that "idea".
>if the recipients can't complain, that's a problem
I will repeat that by definition of the business model, the recipients don't complain to the third-party because they have no knowledge that the third-party even exists.
>but it's not like people don't have a choice of email providers
Even if recipients did know about the third party, do you know the switching costs of e-mail? And, you're suggesting that we should wait until all of our affected members get fed up and change providers (hoping that provider doesn't use the same service, etc.)? All of this, when the problem is obviously being caused by the third-party's error, yet they have no responsibility?
Sorry, but of course the third-party is absolutely compelled to provide some recourse for legitimate senders that are mis-classified. Too hard with all the spammers out there, you say? Sorry. Pick another business.
That's not quite how it works from the end-user's perspective.
Say I buy a product or service from you. Of course, you send me billing emails, etc. and that's fine.
The problem is that companies take this further and start thinking they have the "right" to send all sorts of garbage to that customer.
Now, they of course let the customer opt out... if the customer is willing to spend ten minutes digging through their interface. But most don't.
I know you love those newsletters; they make you money. But don't kid yourself that your customers do. Many are going to click 'report spam' if you send too many. Even if it's not spam by the definition that marketers use, it's still very clearly unwanted email, and marking it as spam is so much easier than going through the unique-to-you opt-out procedure. And if your list is like most, some of the members signed up years ago (or bought something from you years ago) and long ago forgot doing so, so of course they are going to mark it as spam.
If you want to send email to your customers, you need to understand this.
Note, I'm not saying that you can't send that email; you can. I send email to my customers, and some of it borders on being "newsletters" - I'm just saying that you need to be aware of the cost- that cost is that it will annoy some of your users. Getting your email marked as spam is only one part of that cost.
>Even if recipients did know about the third party, do you know the switching costs of e-mail? And, you're suggesting that we should wait until all of our affected members get fed up and change providers (hoping that provider doesn't use the same service, etc.)? All of this, when the problem is obviously being caused by the third-party's error, yet they have no responsibility?
My point is that third party has no responsibility to you. They have a responsibility to their customers, but none to you.
I do agree with you on principle. We've been doing this for over a decade and have learned best practices. We also know what our customers appreciate and are very mindful of content, frequency, etc.
But, you seem to be overlooking a key point, so I will state it for at least the third time: the problem was NOT the content, and it was NOT being marked as spam by recipients. Instead, we were erroneously classified by the service as sending to honeypot addresses and/or defunct addresses. So, your assumptions about our customers, newsletters, etc. are just not relevant here.
>the third party has no responsibility to you. They have a responsibility to their customers...
Their customers complained to us that they weren't receiving our emails, so they weren't serving their customers very well in this instance either.
But, what you really don't seem to be getting is that they do have a responsibility not to harm us. That's why there is a legal cause of action known as "tortious interference". I sincerely don't know how to make that point any clearer. Perhaps researching that term a bit might help.
Yes, yes, reading up on that is probably a good idea. I am not a lawyer... but in the case of a mail-filtering company blocking emails that their customers have asked them to block, they are harming the senders of those mails. That is a direct result of what their customers are paying them to do.
Or ad-block programs- really, it's exactly the same thing; what the ad-blocker does for the user does harm the advertisers. There's many, many places in the free market where helping one side of a transaction hurts the other.
That probably means that "tortious interference" is more complex than you are making it out to be. There are a lot of situations where helping one party necessarily hurts another, and helping one party over the other is generally not illegal.
Neither is it "exactly" nor even remotely the same as ad blockers.
>That probably means that "tortious interference" is more complex than you are making it out to be.
No, it means that your analogies are invalid. Further, perhaps you should look up the term before musing about what it "probably means".
I'm an Amazon customer. If Amazon can't reach my gmail address because Google started using some third party filtering service, what exactly would you expect me to do? Even if I had the ability, I certainly don't have the inclination to spend time resolving this issue. I'll go so far as to mark something "not spam", but that's about it. The cost to me of not hearing from Amazon is virtually zero. The cost to Amazon of not being able to communicate with all its customers that use gmail is huge. It's an Amazon problem, not a customer problem.
But GP did figure out what the problem was and took steps to resolve it without involving lawyers before finally issuing the C&D and getting an injunction.
Glib, yes. Obtuse? less so.
>I'm an Amazon customer. If Amazon can't reach my gmail address because Google started using some third party filtering service, what exactly would you expect me to do?
If your free email address doesn't receive mail you want, I expect you might move to a different mail service. I think that if you want anything but automated support out of a free service, you are expecting far too much. Because you still seem to think sticking with your free email provider is the only reasonable course of action, I can only assume that said free provider is meeting your needs; that the automated support is good enough and that you don't mind the odd false positive.
Sure, if you want service, you are going to have to pay a few dollars, but there are plenty of spamfiltering services that are fairly cheap that do have a real person who will fix it if you tell them you aren't getting legitimate mail.
In reality, users consider "legitimate" newsletters to be spam... spending a bunch of time digging through the "legitimate sender" settings to get the mail you want and not the "legitimate newsletters" is often so much work that users just mark the newsletters as spam, and hope their provider will figure it out. Of course, this works really badly when it comes to reputation systems, as the billing stuff and the "newsletters" often come from the same server.
(really, if you must send "newsletters" in addition to your billing, you should do them from a different mailserver and different email address.)
more problems for people who send bulk email, sure...
the senders of bulk email vastly overestimate how much of a problem that is for end users.
And... this is exactly the point. If the filtering company works for the end user, and the end-user doesn't want the mail (or at least doesn't care if it doesn't get delivered) - then the filtering company should block it.
My point here is that you don't have a fundamental right to my attention. If I'm paying someone to sort my messages for me, and they throw out a message that is important to you, but that I don't really want to see... as far as I am concerned, the filtering company has done it's job.
(obviously, if I want that email, it's another matter entirely - see my other response.)
In fact, we learned of the problem when customers suddenly began to complain that they weren't receiving their password reset emails from us.
Well... that's an oversimplification. You don't have an agreement, sure, but your recipient has an agreement with them --- your recipient has decided that it's with paying the spam filtering supplier to filter their mail for them.
I realise that this doesn't help you, and frankly they don't sound like they're doing their job very well, but it's important to remember that the recipient chose to use their service.
EDIT: Not sure why it would preclude their liability in any case though. Their customers would be paying to have spam blocked, not emails from legitimate companies with whom they've agreed to do business. If they were blocking these demonstrably legit businesses in error and refused to do anything about it, then their "service" is helping no one, but is damaging those businesses.
It is simply a risk inherent to their business model and it is absolutely their responsibility to address it.
Note that I'm not suggesting you did the wrong thing here --- I don't see anything else you could have done. I'm just saying that pinning the entirety of the blame on the third party provider is wrong.
...
Having read the rest of the thread: I'm sorry to say but the reason why you're having so much difficulty, and the reason why you're getting hostile replies here, is that as an email advertiser people are going to automatically assume you're in the wrong. You are on the edge of an astonishingly dirty industry, and anyone who works in email has been so burnt by the Neptune-sized tidal wave of diarrhoea that is spam is likely to spit in your face before you have the chance to explain that you are not, in fact, a spammer. And chances are that they won't actually care. To a most people, email + advertising = spam.
I really don't have anything to suggest. I have written antispam software, and I've read some of the forums of both sides, and frankly I don't know who's scarier.
We disagree. It's certainly understandable that the third-party might raise false-positives. However, what is wrong is when a business can demonstrate that it is not engaged in such practices, yet the third-party provider still refuses to cease penalizing them.
So, who else's fault would it be? Mine? The customer? The customer's e-mail provider who contracts with the third-party? Some unrelated spammer who makes the third-party's business really difficult to execute well? I don't think so. The third-party is in the business of profiting from the proper classification of senders. It's their risk and responsibility to fix these problems.
>the reason why you're getting hostile replies here.
Well, I didn't think the replies were particularly hostile. A little misguided maybe, but not hostile.
Anyway, I think you're also jumping to a lot of conclusions and, frankly, not reading very carefully.
Firstly, we're not an "email advertiser" in the sense you're implying. We're a member-based site with a newsletter, into which only members double opt-in out of their interest in our service. The ads I referenced are occasional sponsorships by businesses that are also highly-relevant to our service and members' interests. But, we don't just push ads and we follow all rules regarding proper email management.
Secondly, our content was never the issue anyway. As I mentioned, the issue was that we were falsely claimed to have been sending to honeypot addresses and/or defunct addresses. I enumerated the reasons why we knew this to be in error.
I hate spam as much as anyone. In fact, my reaction to it is probably borderline irrational. But, there are legitimate businesses with legitimate customers between whom e-mail is a mutually beneficial channel. There's nothing "scary" about that. And, my initial point was in response to an ancestor comment, wherein I agreed that certain "anti-spam" practices make it far too difficult for those who use the channel properly and respectfully.
I use email for a slightly more spammy purpose (double opt in but mainly to promote sales to existing users) so I wouldn't be able to make this claim if it happened to me - my recipients won't notice if they don't get my email and many won't care. I have to deal with these problems too, usually by manually asking the blacklister to unblacklist me, which they seem to do.
It's a good thought. Just wish it were simpler to pull off.
Of course if they don't really want it or just half-heartedly signed up because it seemed like it might be useful at the time then they're not going to make the effort. And I guess that's the situation almost everybody is in when they find they're not receiving emails due to false positive spam filtering. So it's basically a terrible situation for everyone :(
At least we have these magic solutions like MailChip or MailJet (what I use because the free plan fits my usage pattern) so you could turn to something like that if you really had no luck chasing the spam list company.
They don't have any responsibility to help you do business. Comcast and your recipients are not required to accept your email. It's in their interest to do so, of course, because that's what their customers are paying them to do so... but their customers are also paying them not to accept email, because most email is spam, and they don't want to receive that.
Email is not a common carrier medium. You can't demand people accept your traffic. It just doesn't work like that.
> The ads I referenced are occasional sponsorships by businesses[...]
...which means, unfortunately, that people are paying you to send advertisements by email, which means that you're in the email advertising business. Which means that people will automatically assume you're malicious. Which sucks, but...
No, but in the course of doing their business, this third party cannot wrongfully interfere with our business. It's a key distinction that you are missing.
As to the rest of your comment, you are assuming that the problem was the content (advertising). However, I have stated at least three times on this thread the actual issue; we were erroneously classified as sending to honeypots and/or defunct addresses.
Um... they're an email filtering company, and you're an email sending company. Interfering with your business is their business! That's what Comcast is paying them to do --- Comcast want your business interfered with!
> As to the rest of your comment, you are assuming that the problem was the content (advertising).
I'm afraid you've missed my point here. The content is important, because having advertising in the content means that they will have already written you off as a reputable company. I'm pretty sure the conversation at their end went something like:
A. Hey, Bob, I've just got an email from someone claiming we shouldn't be blocking their email, but they're on the honeypot list.
B. Sigh. Again? Do we have one of their messages?
A. Yup. Here.
B. <spends two seconds skimming message> It's got advertising in it. They're a spammer. Screw 'em.
(It may not even have gone that far. I have no idea what format your newsletters are in, but if they score badly they may not even have bothered to look at them.)
My point is: you may not be a bad guy, but to an awful lot of people in the SMTP industry, you're going to look just like one.
It's funny when people start their sentences with "Um...", as if they are about to unleash some potent wisdom of which their target is unaware, then instead immediately go on to prove that they have absolutely no idea what they're talking about.
And, now you're literally inventing imaginary conversations.
It's gotten really silly by now and I've repeated nearly half a dozen times on this thread why "points" such as yours are not applicable here. So, hope you don't mind that I'm going to get off this ride now.
Take care.
Maybe Google is stricter now. I often get captchas when using google.com now.
Internet Mail 2000 IM2000 is a project to design a new Internet mail infrastructure around the following concept: Mail storage is the sender's responsibility.
You could push spam blocking to the end user and client, and simply let them not download messages originating from unknown origins (similar to phone + Caller ID), without having to worry about dealing with spam in a centralized manner.
A very tight feedback loop for generating content that will trick you, and hosting it for just long enough to see if it works.
And in the new application: Address == Public Key, Address book == web of trust.
Your mail agent probably doesn't even bother to check or propagate incoming message notifications from addresses you haven't manually added to your address book (100% trust), or addresses within a certain number of hops from someone you manually added to your address book (friends of friend, boss's employees, etc.). And the mail client could further hide and ignore all of the notifications from below a certain user customizable trust score as well.
There would be more time to determine if the message is spam. If enough people mark it as spam and site is blacklisted or taken down, spam would be removed from everyone.
The big problem with IM2000 is that it doesn't solve the real problems. It's focused on the economics of storage of mail, which, yeah, are a thing for a few mail administrators, but generally is considered less important than the spam-filtering problem in general.
I mean, I'm not willing to pay for infinite storage, but I certainly would be willing to pay the cost of receiving 100x my current monthly spam load if I didn't have to read any of it. Maybe even 1000x.
The major benefit of the IM2000 standard is that if you can squash the spammer's mail servers, you squash the mail in question, which actually could change things, because it would mean that spammers would need to control the servers they use to send for longer than they do now.
This is the reason IM2000 is exciting. Spammers only survive using hit and run tactics. We might see a 99% reduction in total generated spam.
SMTP is not where email organization should happen. That's what email clients are for. Blocking DoS attacks and spam makes sense. Blocking Amazon Kindle promotions does not.
See, you still seem to think that it's a negotiation between the senders and the receivers.
My point is that you don't have the right to the end-user's attention. If the end-user wants to hire someone to filter out messages they don't want to read, using whatever criteria for "don't want to read" that they like, this is the user's right.
(more relevant to the discussion, the email that is technically spam is pretty easy to block already, so a solution that requires a lot of social/political change is inferior to the technical measures we have in place.)
> Users should be able to easily filter and ignore promotional email, but email hosting companies should never block it as spam.
i.e. Yes, it should be the user's control, not the hosting company's, not the sending server's.
But I agree, SMTP is not where email organization should happen. Blocking illegitimate senders at the protocol level is the most that should be done, and the blocking of unwanted content should be pushed into a more user oriented layer. Current structure isn't optimal.
I'm really glad that gmail sorts these emails into a place where I don't have to ignore them, because I used to flag them as spam whether or not I might have accidentally given consent 10 years ago by buying something once. It was making email as useful as physical mail(read: not at all).
Just because I bought something from you once does not mean we have a relationship, just as a one-night stand does not entitle you to ask me to help you move.
If you didn't ask my permission to send me those promotional emails, they absolutely are spam and anyone who sends them should have their emails blocked. This rationalization that spamming is ok because the recipient is a customer and there is an unsubscribe mechanism is delusional self-interest.
Its screwed up, sure. But its the state we're in right now.
That he focuses on the software instead of pandering probably makes his software better than the alternatives that aim to please even the most foolish of users. At least I think so.
Maybe I interpreted the proposal incorrectly, but I always saw IM2000 (minus the "notifications") as a "pull" solution.
By contrast, conventional email relies on "pushing" spam to the recipient (in practice, a middleman called an "email provider").
A smart IM2000 recipient perhaps would not pull spam from the sender's server.
As such, the spam would never enter the network. It would just sit on the sender's server.
Therefore, IM2000 not only conserves storage but also conserves bandwidth.
Based on what criteria? You wouldn't have the message headers/content, so you couldn't filter based on that. All you can filter on is the IP address. But you can already do that with SMTP by just responding with a 5xx code as soon as the sending server connects anyway... Or you could actually fetch the message and then spam filter it. But then, you may as well use SMTP...
What difference does it make which side initiates the TCP connection? If I was a spammer, IM2000 wouldn't concern me one little bit.
If the sender did not have at least one personal device connected to the internet 24/7, then they can perhaps purchase storage and bandwidth from a competitive online market, similar to the seedbox market for torrents, without surrendering their online identity or address to a commercial entity.
1 - http://tonsky.me/blog/streams/ 2 - https://news.ycombinator.com/item?id=9829614
3 and 4 would require a sort of token system
https://github.com/ssbc https://github.com/ssbc/secure-scuttlebutt https://github.com/ssbc/scuttlebot
That might have since fallen out of favor, I'm unsure.
I am surprised and encouraged to see IM2000 mentioned in this discussion.
As to whether such ideas have fallen out of favour, I offer this observation:
You are using a pull-style electronic communications system right now. Your WWW browser is the originator and receipient UA, and Hacker News is the message store. A copy of a message is only made and sent across the network when you request it with your User Agent. A lot of Internet-mediated communications are not SMTP-based any more. The shift to pull-style systems has, in some ways, actually already happened.
As such, many of the things that people ask about IM2000 are questions that they can answer from their own direct experiences, with a little reflection. How do you deal with senders who alter their messages? Well: how do you the receipient deal with the fact that right here and now I can hit an edit button and modify this comment after people have pulled it for reading it, with no edit history and nothing but the honour system for dealing with letting people know? The fleshing out of IM2000 set out the idea of downloading to folders in the recipient MUA the messages that one wanted to take off the senders' mail stores, by the way.
Not to get all handwavey, but I think this is why some people are super excited about bitcoin becoming 'part of the internet'. There are definitely some areas where we need a concept of identity & trust, and bitcoin seems like one of the first truly distributed ways of doing it.
Because the SMTP cabal is fanatically opposed to changing it.
The response I had 15 years ago was "It takes 10 years to deploy any serious changes to a protocol".
Uh... so? Does that mean we shouldn't do it?
Apparently, yes. 15 years later, there have been no substantive changes to SMTP.
Repeat after me, SMTP is perfect. It's perfect! Well, there are a few known issues. But it's deployed! It's too hard to change. You're trying to change it? You're a terrible person who doesn't understand SMTP.
I got that for 5 years before giving up and going to more productive lines of work.
It's been some years now that I'm thinking on this messaging and spamming problem. It is not a protocol or technical issue. The problem is the abuse of the communication system by spammer.
The worst is that people have to hide their contact address or use semi private messaging systems. This is totally contradictory to the evolution of the communication systems. We have smartphones and can't reliably send a message to any body in the world ? Internet is lacking a solution for that.
This one seems completely uninteresting in a world in which $15/year can get you your own domain name, complete with reliable email servers, IMAP, and as many email aliases as you like at that domain. Given that, why create an entire infrastructure to "decouple" email addresses from domains?
That being said my suggestion doesn't work from a privacy point of view as the domain GUID would become a tatoo and it wouldn't take long for directories mapping GUID to real identities to appear.
The X protocols have typically struck me as grossly overdesigned and complicated.
The latter didn't win because it was "better" just because it was easier to implement. Like a car with no brakes or seatbelts.
Since the 19th century, postal services have been monopolized by government institutions with a fee levied for storage, processing and transportation costs.
Physical mail delivery in bulk is costly, which makes it more profitable to send directly rather than en-masse.
But SMTP requires no fee. Originally, your ISP would provide you with e-mail in exchange for the fee you paid for your connection. Since the advent of "free" email services, people got used to not paying for mail anymore.
With electronic mail, it's cheap and easy to send in large quantities. Before we never needed to verify who was sending us mail, but now it's the only way we can filter out the noise.
To make electronic mail better we have to make it more expensive. Requiring a payment to send mail would reduce the cost-effectiveness of spam, and I would argue it would make the systems more verifiable. You could even preserve anonymity by having anonymous mail services, as long as you paid to send those mails, say with bitcoin.
Want to run your own mail server? Fine, but relaying will cost you. FedEx doesn't deliver mail from the USPS for free, and so relays shouldn't be expected to pass on mail without transportation costs.
[0] https://roj.is/blog/a-different-approach-to-online-account-m...
For a new businessman or professional setting out in the world, pre-Internet, "establishing your name" was a requirement. These days the concept's often referred to as "creating a personal brand", but the reality was pretty straightforward: how does an unknown quantity become a known quantity?
A common method was the professional or social introduction. This is still practiced, where a third-party _matchmaker_ will introduce two parties. The matchmaker usually knows both, and can vouch for the newcomer and smooth the path for introductions with the established party. Essentially, the matchmaker stakes _their_ reputation by speaking for another.
Lawrence Lessig describes a similar concept in his book Code and Other Laws of Cypberspace, in a passage describing a physical messaging system, the Yale Wall. This was a board onto which messages could be posted, with the proviso that that they be signed. Unsigned messages would be posted from time to time, effectively presenting an anonymous viewpoint. Removal wasn't instantaneous or automatic, but rather, at some point prior to garbage collection, another individual could review the piece, and, if they felt it warranted posting, sign for it. They weren't registering themselves as author, but as vouching for the merits of the viewpoint -- not necessarily agreement.
A messaging system in which a new peer might be able to indicate that, hey, peers X, Y, and Z, with established reputations can vouch for me, and for which those peers could confirm their endorsement, might address the "how to build reputation" issues of new mailservers.
I've also found that even large mail systems will frequently have some procedure for getting at least provisionally vetted, effectively a workfactor cost to coming online. Though the process absolutely could be improved.
The big providers need to build an accessible system for entering the email market or face tremendous fines from worldwide governments. There are plenty ways imaginable to do this, especially if you allow the process to involve real people and paperwork.
It shouldn't matter whether this is simple negligence or conscious work to keep out competition. The end result is that it's made nigh on impossible by a few market leaders to enter said market.
Heck, this doesn't even have to be a new company. A big host like Google could just start offering this service anyway, as a way to simplify their own handling of unknown senders, although I'd feel more comfortable if this was done by someone else.
I checked the spam folder. Gmail had plopped the apple email into the spam folder. Gmail's reason was that the email was in a foreign language, Italian. Didn't matter that the previous 2 weeks all my google.com searches were done from Italy.
1. Accessing an account one time over Tor. Subsequent reconnect attempts from the same device I'd previously used, same IP, etc., failed. Took a couple of weeks to get back.
2. Accessing accounts from different IPs while travelling. Ultimately ending up at the same IP as one of the email accounts I'd previously corresponded heavily with.
3. Accessing accounts from new device(s). Old (Android) devices could access email but not Web-based account recovery tools.
"Who are you" is the most expensive question in technology. No matter how you get it wrong, you're fucked.
https://www.reddit.com/r/dredmorbius/comments/3mo7l6/that_go...
The email architecture was started back when it was a smaller network of researchers at universities, governments, etc. Everybody basically trusted each other.
Once the "internet" is available to the general public and commercial interests, it becomes vulnerable to the "bad actors" problem (e.g. spam abuse). That's why we have the inevitable situation today of a few entities (e.g. gmail, hotmail) being "trusted", and random residential SMTP servers run by homeowners being "untrusted".
I haven't seen a realistic de-centralized trust proposal for email. Even if a proposal is theoretically sound, what incentive is there for other big players to adopt it?
Unfortunately, that would be less than decentralized, but it may turn out to be the best option in combating spam.
I've just opted to pay for sendgrid for my small hobby BBS server's outbound email, because it's easier than setting up an appropriate outbound system myself, and as TFA points out, even then odds are you'll be bitbucketted before you even start.
You can't remove "trust" or "reputation" from a viable messaging system. The concept of trust always exists whether informally or formally.
When the internet was a small private network between universities and government, how did a new unknown scientist "cold-call" an email? It was not an issue. The scientist just sent the cold email. There was already implicit trust within the system to allow that scenario. Everybody trusted that researcher@someuniversity.edu didn't send a million emails about Nigerian money transfers to researcher@army.mil.
Back then, email users didn't need "spam filters". The trust was informally created because the institutional "system" outside of the email system vetted email users. (The "system" being the hiring procedures of SomeUniversity, DepartmentOfDefense, etc.)
All those gates for reputation are gone when we expose that simplistic email architecture to the general public. Trust doesn't go away. You now must recreate trust at a massive scale.
>Spammers could create million accounts at once and fake trust build up between them.
That's not the type of "trust" people are talking about in this thread. That type of incestuous "trust" between bad actors won't work because it doesn't have an initial authority (good actor) to "bless" them which starts an acceptable "chain of trust". The concept is similar to Certificate Authorities. We trust Chrome because we trust Google Inc; and in turn, Google Inc trusts Verisign; and in turn, Verisign EV-certificate processing trusts government offices that register businesses.
Notice what you did there? You characterized spam accounts as "trusted" even though you didn't specify why or how they'd get trusted in the first place. There's still a gap in your thinking of how the origination of real (not fake) trust happens.
It's like a 1970s email user asking, "how does the system prevent a 'trusted' Nigerian money scammer from spamming everybody?" Well, why would Department of Defense hire a Nigerian scammer? Would a UniversityOfWhatever hire a Nigerian scammer? No. In that scenario, the trust begins with the institution. For the Nigerian scammer to be trusted, somebody authoritative has to "bless" him.
>Could you give an example how trust could be used to control abuse ?
Well, we're still using trust right now to help control abuse. Today, reputable corporations and also millions of honest users (that's us) "trust" Google Inc, Microsoft, Yahoo, FastMail, etc. Spam on a massive scale does not originate from gmail/hotmail/etc because those corporations shut them down.
The issue is that it's centralized trust concentrated in a few big players instead of de-centralized trust. That's the thorny problem that's very hard to solve. All the clever decentralized proposals (sender pays for storage, outbound payment bonds, cpu-proof-of-work, etc) are basically roundabout proxies for recreating "trust" on a bigger scale. I don't know what the final answer will be. I just know that trust/reputation is an unavoidable part of a new solution.
I rent some servers from the Rackspace cloud for personal use. I have my own sites on these machines, and my own email servers.
Meanwhile, I have a day job, and lately it has been consuming 12 hours a day. We missed a deadline and we have all been working like crazy to catch up. I have fallen behind reading my personal email.
Roughly a month ago, my friends who use Gmail stopped getting my email. Or rather, they did not know I was sending them email, because all of my email to them was going to spam.
After a few weeks, I finally had a free weekend to catch up on my personal life, so I did some investigations. Turns Rackspace had switched over to IP6 in a way that impacted my email. I did not have a Sender Policy Framework for IP6, only IP4.
It's likely that Rackspace sent me an email about this, though I never read it because I was busy.
This was easy to fix: I added a SPF for IP6.
However, these kinds of issues do make it harder to maintain a personal email server. Its tough for us to keep up with the changes.
* every time someone implies spam filtering is a massive conspiracy against "the little guy", drink
* every time someone suggests Bitcoin be used for something, drink
* every time a solution utterly fails to account for compromised end-user machines sending spam, drink
* every time a comment can be summed up as "assuming a PKI exists, this problem is trivially solved", CHUG THE REST OF THE BOTTLE
That reminds me of an old classic: http://craphound.com/spamsolutions.txt
There are a lot of obvious problems with email.
* It's hard to run your own mail server, as described in the original post
* Your address is tied to your provider
* The standards involved in delivering mail (SMTP, POP, IMAP, etc) are complex, layered with hacks (SPF, STARTTLS, etc), hard to implement correctly, hard to secure.
* The standards that govern the actual body of email (multipart MIME, a 90s-vintage subset of "HTML" and "CSS") are even worse
But despite all those flaws, email is built on ubiquitous open standards running on a federated infrastructure.
It's the best thing we have.
Proprietary standards and centralized designs like AIM, ICQ, MSN, FB Messenger, Slack, and Twitter DMs come and go.
Email is forever, and is more useful than all of those combined. The flipside is that it's very hard to "fix".
And this is exactly the reason I setup my own mail server. I'm only 1 man, but I hope more people will do so with time, thus requiring the "big ones" to work on better algorithms for filtering and not base it on reputation.
Every email would be digitally signed and encrypted. Certificate with keys would connected to email address (and bitcoin wallet).
Spam would die.
Go build it please.
Maybe it would just become more targeted?
In any case, I would love to only send/receive encrypted/signed email. I don't think the problems are technical.
http://craphound.com/spamsolutions.txt
and check all boxes that apply.
Other than that, I agree paying to send will reduce spam.
But also look at your postal mail box. There is arguably more spam there than in your digital inbox, and that one costs (stamps).
Are folks afraid of bitcoin? What can be worse than just having a 16 digit number printed on a card -- that is a 1000x more vulnerable.
You are part of the tiny percentage of the tiny percentage of the population that understands bitcoin.
I keep thinking instead about something that works more like a refundable bond:
To send a message to you, I have to include some amount. (I don't know how much. Not a penny, though. Maybe $10 or $100.) If you want to keep receiving messages from me afterward, you return the bond. If not, you keep the money.
And after a couple of minutes of Googling, I just found a name for schemes like this: Attention Bonds.
The bitcoin transaction can just go to /dev/null with a hash of the message as data, so don't have to pay somebody ( ;) ) and you cannot track who sent data to whom via tracking the bitcoin transactions.
In my experience if you get blocked by big mail providers it's almost always due to some reason. What's tricky is that it may be hard to tell what exactly is wrong, because they won't necessarily tell you (or not in an easy way).
Some advice what I'd do to try to find out what's going on:
1. Take a sent example mail that is like the blocked one (but obviously one that reached its target destination) with all headers and run it through spamassassin. Don't just look if it hit the spam score (then you did something terribly wrong), look at each individual rule that spamassassin hit. They might give you a clue. A proper mail usually shouldn't hit any or very few positive spamassassin rules.
2. Check your IP at a service like valli where you can query multiple DNS black lists. If it is on any blacklist try to find out how you can be delisted. There are some rogue blacklists that make it impossible to be delisted at all, you may ignore them (google for them, their behavior is well documented), but these shouldn't be more than 1 or 2. As already said by other commenters, don't forget IPv6.
3. Read whatever error message you can get your hands on. If you're blocked on the SMTP level read the error message. If your message got sorted into a spam folder look at all the headers. If the provider blocking you has some online docs about their spam filtering read that. If they have some sort of service for mail ISPs where you can sign up to get warnings sign up there.
Of course also the obvious stuff. If you do anything that is mass mailing you are in extra danger. Make sure that you allow people to unsubscribe easily, don't ignore manual attempts by them to unsubscribe ("I want to get off this mailing list") and delete invalid mail addresses.
Hashcash (also known as the precursor to Bitcoin) was proposed to solve this problem in 1997:
https://en.wikipedia.org/wiki/Hashcash
The trouble with it is that it requires computation for each sent message, which is bad for senders with low resource devices or legitimate mailing lists. I want to propose a variant.
Instead of creating an expensive hash against the message, create an even more expensive hash against the (sender TLS certificate, receiver domain name) pair. This implies using TLS but it works just as well even if the certificate is self-signed. Then each mail server only has to generate a hash once per recipient domain, ever. Every message that mail server sends to that domain is tagged with that hash. A legitimate mail server will have already computed hashes for all the domains its users regularly correspond with and rarely if ever need to do any more expensive computations.
If spammers do the same thing then the receiving server can mark all messages sent with that hash as spam. So there is a highly disproportionate cost to spammers (even if they have more computing power) because to avoid that they have to continuously generate expensive new hashes. Which can be made arbitrarily expensive because legitimate servers only need to do it once. And a new hash is much less valuable to a spammer than a domain name or IP address is today because each hash can only be used against one recipient domain. The required amount of computation can be set by the receiving server so domains with more users can require more computation.
If a legitimate mail server is compromised by a spammer then it will have to generate all new hashes (because the spammer will presumably immediately ruin the reputation of the compromised ones), but the reputation of the legitimate sender's email domains is unharmed because the reputation is tied to the hash computation, not the sender's domain name(s).
And adding support to mail servers would require no configuration whatsoever. You install server version N+1 and it starts tagging outgoing messages with hashes that receiving servers can verify.
So here's the traditional form: http://craphound.com/spamsolutions.txt
How'd I do?
With a memory-bound proof-of-work system like my Cuckoo Cycle, computing the hash could require the use of more than 4GB of memory for over 5 minutes on a 20-thread server, thus preventing the use of botnets for avoiding the expense.
But even if some large providers refuse to hash outgoing mail, they can still verify incoming mail. The interesting benefit is that a small email provider can burn some dollars in computation once to avoid having their mail discarded by a large provider's spam filter indefinitely.
I suspect that people having trouble are sending a lot of mail, like “newletters”, etc. But I can’t prove this hypothesis.
Even when I used to work at HostGator and handled many of their abuse issues, they had many issues with being blacklisted just because RBLs didn't recognize new HostGator IPs or the rate of email being sent from their new gateways.
So, at least with my personal and professional experience, I can attest to the issues with using self-hosted mail servers.
The one and only issue was with some AT&T block list - and judging by their supporting forums they end up with a boatload of false positives - I filed a complain with them and they removed the block few days after.
I suspect that OP ended up getting an IP from a netblock that has lots of transient customers, which included some spammers. It's not his server's reputation that was weak, it's the netblock's rep that was bad.
When people buy my book, I my server sends them literally one single email containing a link to the PDF they have purchased. And that's it.
Google delivers my email fine, in most cases. But hotmail/outlook was just deleting them most of the time for over a year.
I know I'm just a single data point, but I swear I'm not a newsletter!
That means that, if you have any configuration mistake, an Exchange server will reject your email. But if you have a good reputation, Gmail will deliver it anyway.
Hotmail sender support kept saying "I do not see anything offhand that would be preventing your mail from reaching our customers from this IP"
Also, my emails were and are text/plain.
I'm sure Exchange does reject malformed emails, but that wasn't my problem.
But, of course, YMMV. And I have no experience at all sending email to Hotmail. I just thought it was worth checking your server settings.
People talk about the reputation of their individual IPs, but networks matter a lot too. DigitalOcean for instance is an occasional source of spam; Linode is very rarely; 1and1 is a HUGE headache; and so on.
I have some custom software in place that temporarily drops entire netblocks for abuse, and I did that to keep up with the spam filtering offered by larger services. With so many spammers now switching between cheap VPS hosts, it was getting impossible to do single-IP-only bans.
Ever since the software was put into place the amount of spam actually making it into users mailboxes has fallen off a cliff.
You have to be very very careful about which network you choose to host your mail server from.
It's a pity there's no easy way (that I know of) to look up the reputation for an entire network block. I'm glad to hear your experience with Linode matches mine.
Then I got a VPS at Linode (for some other purpose), installed Postfix and emails were just working - no SPF or even reverse DNS set up.
IP reputation IS a thing.
The reason this is happening is as the blurb from the MS postmaster help page: Your IP doesn't have a reputation yet.
The reason these rules are in place aren't about email monopoly, it's about spam. If anybody could setup a SMTP server and start firing off large amounts of mail, spam would be even more endemic than today.
You can configure your server perfectly, but that doesn't mean much, since it's your IP that's the problem.
If you have legit objectives, it's a pain in the ass for sure. But you are not the only one having this problem, and there's a solution for it.
All the big email service providers (ESP's) like Neolane, Exact Target, Mailchimp, Campaign monitor etc share this problem when they onboard a new client, who requires their own IP.
Deliverability is a surprisingly deep, technical topic, and all major ESP's have entire teams of specialists working on this.
If you want to make such a service as Fastmail, you need to get really into deliverability. It's not a walk in the park, but it's not impossible either.
I'm not a specialist in this particular area myself, so I can't give you that much specific advice. I've just worked elbow to elbow with a lot of these guys, so I know what kind of challenges they work with.
One thing I know for sure is really important, is the "warming up" of IP's. Basically the IP you are sending from needs to accumulate some reputation over a period of time, typically a month or two.
If you send out reasonably small amounts of mail to email addresses that exists and the recipients does not explicitly report you for junk mail, your IP get whitelisted and you will get a much higher delivery rate.
There's no quick fix unfortunately, and email reputation is hard to gain and fast to lose.
But it certainly can be done. You sound very competent on the server side of things, so to get your fastmail-like service up, I think it's just a matter of a bit more persistence and studying deliverability as a technical subject.
Hope this helps.
I self-hosted from about 2007-2008 and 2011-2013, and had nowhere near as much trouble with deliverability. It came as a bit of a surprise how much more difficult it is these days.
But anyway yeah the barrier to entry has risen really much. It's much harder these days than just a couple of years ago, and it probably won't get easier either!
Real answer: It comes down to trust (duh). But, how do we manage trust online? How do we manage trust in real life? Real life trust is through association of groups. But, groups online are meaningless. A "gmail user" doesn't belong to a community, they belong to The Nation of Google.
How do we break down online identities into manageable, trustable communities? How do we bootstrap new communities into the system so we don't end up with a pre-selected list of blessed communities while others languish in obscurity?
Open questions, no answers, and if you say Facebook you have to run 20 laps and give me 1000 pushups.
Personally, I would like everyone to have their own rsa or ecc key. While not as expensive as HashCash, it does require more computational effort to send an encrypted email. (This precludes web portals from sending mail, but user convince seems to always trump security! :( ). Still not a great solution.
Web of Trusts are an interesting idea, but fails for the same reason not everyone has a public encryption key: bad ui, complex idea, and no one cares. I honestly think this is the way forward. If someone isn't a few hops from me, they probably don't have any business contacting me anyway -- if they really want to, a.k.a. read my blog and want to ask me a question, they can use HashCash or something else expensive to make the initial contact and slowly build up trust/reputation as we exchange emails.
In the end, everyone is always told they need to check their spam folder for false positives anyway. Maybe we should all just say "screw it" and read all our email. If some combination of the sender and subject doesn't make us think it's legit, then we can delete it.
Maybe we keep spam filters, but stop calling it a "Spam Box" and treat it like trash. Maybe we start calling it "Unknown Box" and treat it with a little skepticism instead of a dumpster.
Also, I guess the big ISPs have an easy enough time (in their view?) analyzing trust, which is probably a better solution in general if applicable. The problem is that it's easier to judge (newly created) internal users (accounts) than external ones (random IPs).
I would think improving the mail system would be a research topic, but perhaps not interesting enough from the point of view of researchers (would it be best done in social sciences or computer science?, and there are many real-world variables, taking time to collect and verify, and working out simulations to predict possible remedies).
All of our users have their own domain names, IP addresses, SPF records, and correctly configured (and up to date) Postfix SMTP servers. There is absolutely no excuse for not always delivering their email, and yet this is what the big companies do.
OP might have had an IP previously used by spammers or in a spam block. (See edit).
Important to check the IP before using it if it's been given to you by, say, the VPS Provider or the upstream. To see what, if any, reputation it has.
Here are two tools to use:
http://mxtoolbox.com/SuperTool.aspx
And there are a bunch of similar tools.
1. They set SPF policies of "-all" on customer domains. These customers have no idea about SPF and often use 3rd party software to send account statements that never get delivered. This breaks email for their customers and for everyone else. They should set "~all" by default as Google Apps does.
2. They claim not to, but they definitely block entire IP ranges rather than a single IP address. A noisy neighbour in the same datacenter as you can get you blacklisted. Use http://www.senderbase.org/lookup/ip to monitor your neighbours and report them to your hosting company. If you spot a bad neighbour, you will no doubt see your own IP blacklisted at Outlook.com's https://postmaster.live.com/snds/ipStatus.aspx within a few hours.
3. They are well known for accepting mail and then silently dropping it, without bouncing sender or receiver and without putting it into a spam folder, even when the sender is a longtime sender to that receiver and when the sender is sending personal anticipated email. They should either reject at the SMTP transaction (sending a bounce later is bad practice and leads to backscatter) or else deliver to a spam folder. Silently dropping email breaks email for everyone.
The sad thing is that many businesses are switching to Office 365 and Outlook.com.
But this kind of problem invariably arises when we go from a fragmented Internet with lots of small hosts/providers to an Internet of several walled gardens, run by the big guys.
The real answer is to offer a reasonable self-hosting competitor to GMail.
The difficulty in getting feedback is extremely frustrating, particularly compared with getting feedback from, eg, google's webcrawlers.
You could always deal with traditional viruses by blocking certain types of attachments. A false negative on spam just annoyed someone with some Viagra sale they probably didn't want.
Nowadays, you fail to block an email and suddenly a user loses his entire department's file share for the day.
help center post: https://goo.gl/7QHoqc Postmaster tool: https://gmail.com/postmaster/
(disclosure I work in Gmail)
I'm not blaming you personally, but the way Gmail handles email originating from personal servers is reprehensible.
In our case, a number of IP addresses we are using started showing up as "bad" in Postmaster tools, despite having no increase in complaint rates (actually, in Spam rate dashboard it shows 0.0% for the past 90 days). We have implemented Gmail feedback loop headers into outgoing emails to diagnose the issue, but the Feedback Loop dashboard also does not show any data.
All of these IPs have a Senderscore reputation in the high 90s and deliver perfectly fine to all ISPs, except Gmail. Contacting Gmail team through their form also yields no response.
My takeaway from this situation is that once Gmail starts not liking your emails for some obscure reason, you have practically no way to fix that, apart from getting a new IP / sending domain and starting to build sending reputation from scratch.
No data to display at this time. Please come back later. Postmaster Tools requires that your domain satisfies certain conditions before data is visible for this chart. Refer to the help page for more details.
The internet was supposed to be for everyone and anyone. Current internet email is terribly flawed. There's just no way around it and the current status cannot be defended.
That empire amounts to a junk email delivery service and later a way to gather information about email users. The later purpose is probably why you want to run a new email service?
However it is certainly feasible to run a new SMTP-based email service from residential IP's that does NOT interact with the existing email empire. One with no middlemen. The sender's SMTP server talks directly to the recipient's SMTP server. You decide what port you want to use. There are thousands to choose from.
There are multiple ways to do this, but I rarely if ever see this option discussed. I suspect it's because like DNS most users are not comfortable configuring mail servers nor with NAT traversal.
If indeed the motivation for running your own mail service is because you do not want your mail stored on third part servers (whether in the sender's mail folders or the recipient's), then the ability to interact with the existing store and forward email providers seems a counterproductive requirement.
Last August, I have added IPv6 to our mail server, so its address for sure didn't have any prior reputation. Once the PTR and SPF records were correct, sending mail over v6 to gmail was no problem at all.
I have yet to see any delivery issue over v6 that's caused by a lack of reputation
Also, I currently use mailgun (and dabbling with mandrill on some new projects) but I'm intrigued by postmarkapp.com take on dedicated IP addresses. Apparently they won't sell dedicated IP addresses because of the time needed to warm them up, so you get a shared IP and their TOS is stricter anti-spam than other ESP's: http://blog.postmarkapp.com/post/14127210172/the-false-promi...
The reason Mailgun and Mandrill can do this is because they belong to much larger companies with better access to IPv4 stockpiles: Rackspace and Mailchimp.
Source: I worked at Mailgun.
Furthermore I can't stand when I have to setup all of my accounts across the multitude of devices that I own. Way too many settings to configure regarding ports, domain names, encryption techniques, folder mappings, etc.
Don't get me wrong, I'm super impressed that email has managed to continue to be an essential tool in many people's day to day workflow despite being old enough to be found in a museum. But it's time for a change, and for us to put all we've learned about encryption, data serialization, and user workflow out on the table to design a better future for the universal method of digital asynchronous threaded communication.
And if we've learned anything from Google Wave and friends, these changes have to make sense, and not stray too far from what we're already familiar with and know that works.
Anyone want to weigh in?
If you're worried about the fact that your mail host and web host can now impersonate each other, we can just define a new X.509 extension for "I can only be used for email". You might even be able to get away adding a new option to the existing Extended Key Usage field, but it's possible that enough clients don't enforce it rigorously enough.
Ideally, you'd also want an SMTP equivalent of strict transport security.
My concern is that it doesn't get you anywhere. phishing sites can and do get TLS/SSL certificates. The process isn't particularly difficult or labour intensive if you own the domain. As far as spam goes, so what? This only proves I'm talking to the server I intended to, not that it's a reputable and upstanding member of the server society.
Postfix, for instance, lets you configure what it checks via the "smtp_tls_verify_cert_match" option. If you set it to "nexthop", it'll require google.com. The default, "hostname", will check against mx.evil.com.
http://www.postfix.org/postconf.5.html#smtp_tls_verify_cert_...
And yes, the concept of relays should disappear along with much of the other cruft SMTP brings. (And while we're at it, can we fix/replace IMAP, or at least make the spec say that message ids are eternal and can't be invalidated?)
As for a new protocol, there are some out there, but all fail in some way or another. I figured I'd throw my ideas out there and see how much they fail https://github.com/jimktrains/email_ng
Spam needs to stop and so too does the convergence of Internet services in general (not just email).
Frankly, the solution already exists, and just like how billion-dollar companies start with the tech community, techies need to embrace the idea of decentralization by adopting some trust-based model.
Frankly, I do not see why encrypted emails cannot be accepted by users through some peer-sharing agreement where the public-key is online.
There's tons of solutions that are easy for tech people and if this community embraces it, it trickles down, like how a lot of other things do in tech.
If there is a single abstract advantages, like 'more security', that might not be good enough.
We might as well throw in the kitchen sink and come up with a detailed list of every possible advantage an ideal replacement protocol should offer:
- no domain names required to create addresses
- 1 terabyte+ attachment file sizes
- ability to update and delete messages which recipients have not yet downloaded
- mail transfer and storage agent which don't have access to unencrypted message contents
- mail agents which can be distributed across multiple personal and home devices
- client apps which always and non-optionally use local encryption\decryption
- ability to purchase mail serving and storage resources from a competitive commercial market directly from client application.
- ability to switch to a different commercial serving\storage resource provider without loosing your previous identity\address.
So moving on from there, how do we fix it? Who is currently working on fixing it? What would a new protocol look like?
SPF, DKIM and DMARC are no indicators of spamminess of a source. These systems have a completely different purpose.
It turns out that creating a successful product in the email space requires you to build relationships and partnerships with the existing vendors/providers. It takes a lot of work, and is part of what you pay for when using Mailgun/Mandrill/etc.
These "greylists" and systems that drop mail from unknown IPs have been pretty carefully designed and tuned to combat the insane amount of spam out there. They work remarkably well, and for most of today's email users, spam is no longer an issue. (The current state of email abuse innovation is promotions/marketing/etc. which is a more subtle challenge.)
At the end of this article, the author writes, "This isn't how the internet is supposed to work." Ironically this system of obscure reputation-based email is a direct result of how the email system was actually designed to work, with a total lack of permissions or feedback loop. Many SMTP servers used to not even require a password.
Stuff like DKIM/SPF and DMARC is a step in the right direction. But the RFCs upon which our email system is based were written decades ago, and in many cases have fundamental flaws, like SMTP leaking metadata no matter what. I could go on and on about issues with email, and why I care about getting them fixed, but let's just say it was designed in a different era of Internet with different constraints and opportunities.
So how do you build a new email service and not get blocked? Well, you spend a few weeks or months emailing, calling, Skyping, and meeting with folks in the current space. You work your way up through marketing support and random protocol discussion lists until you are talking with the folks who can influence which IPs are blocked/unblocked. Then you convince them you are (1) building a legit venture, (2) are worthy of their trust, and (3) don't directly compete with them. Then you'll get a small number of clean IPs and you must not screw up!
There are a few hacks, like sometimes a single partner/vendor will sell you a block of clean IPs and help manage the spam reputation. But usually it's just a lot of sweat and annoying phone calls. It takes way, way longer than setting up SPF/DKIM. The challenge is more relationships than technical.
Once you have a new email sending provider, the burden shifts to you for doing abuse/spam prevention. And you find yourself implementing many of the strategies and systems you cursed when getting started. But that's the circle of rfc2822 life I guess.
Oh, and never use EC2 IPs for sending mail. Most of them have been burned by spammers.
That said, silently dropping messages without a notification is probably illegal! And pretty serious! So if you know what you are doing (and you are not a spammer) you should send a cease and desist!
Just make sure you use double opt-in and that providing an e-mail address in sign-ups/etc is optional!
Some will however put your mail in a spam-folder and it's not much you can do about it, just hope your readers complain to their provider.
So basically: setup your smtp relay correctly! Make sure you are not on any black-lists. Add extra headers like dkmi / precedence, add spf (don't froget ipv6) and ptr. Add your relays to white-lists. Publicly publish a privacy and e-mail policy (important! with opt in and optional clause); Link to them and fill out some "email provider" forms at gmail/microsoft. Send out a bunch of test mails. This will take a whole day, but if you do this, you will have no problems unless your IP or domain is perma-banned.
Do we really need a comment against the NSA in every single thread mentioning emails or hosting in general? Not that I'm a NSA supporter or anything.
Though that might just be because I have a 6-character domain (ajf.me).
There is a KB explaining that new senders get a high score.
If none of these major services ever learned that his server was OK, it's likely that users weren't unmarking the mail he sent as spam. And that leads to the question of why not.
For awhile I wondered why so many SaaS apps bothered to pay external (not cheap!) mail services when it's so easy to set up an email server.
This is why. My "easy to set up an email server" info was years out of date.
Our domain has all the correct SPF records, DKIM etc and we're using a reputable email provider (Mandrill). All is sparking clean. Validated etc. Still we see some activation emails to google get delivered, but severely delayed on the Google side (after we see an OK from google for the email being delivered). Our users can't reset their password or activate their account until a certain time passes...
Emails from my mum (who I've emailed on the same account for nearly a decade) suddenly get flagged as spam by gmail. Go figure.
The blacklisting ugliness is one more sign email was badly designed. One more nail in the coffin.
I once heard that my email got flagged as spam.
Of course, if you don't set up your mail server correctly it might be that it's flagged as spam, but it's not really harder than setting up various other things correctly.
It sounds like it was set up correctly by the author. Most people use SpamAssassin, also big companies do. So it should be good.
Maybe the network itself wasn't considered to be good by the mentioned big companies.