Email encryption is here – use STARTTLS everywhere
dwheeler.com
dwheeler.com
The problem that prevents such a standard from ever taking off is that, unlike TLS, there is zero interest from major providers to push end-to-end encyption, because they absolutely need access to the plain text for commercial reasons. So they advertise STARTTLS as "email encryption". By that token, CDMA and GSM are "email encryption" as well, it's not like they push the plain text naked over radio.
They are also the only possibilities to provide proper end-to-end encryption. The last thing we need is another proprietary “standard” (remember how those same companies pushed for drm in the w3c spec).
Yet the only real change can come when end users understand the implications of using the free email providers. I think this is harder for most than using a pgp email client plugin.
> Please indulge me: I think a small rant is appropriate here. There are some security specialists who think that only the perfect is acceptable. Nonsense! Requiring perfection is crazy. [...] For almost all users, email encryption with STARTTLS is a major improvement over what they had before. Let’s keep working to deploy even better systems, but let’s take partial victories where we can get them.
Edit: srsly guys, HACKER news people say that it's impossible for a person to have a proper mailserver set up? That's hilarious, if not sad!
Of course you can have a lot more downtime than that, even just 20% uptime would be sufficient if timed right to ensure that no mail gets lost.
Or in other words: My central heating system has higher availability requirements than my mailserver, so I guess everyone in non-tropical places has a non-trivial endeavour going?
The Internet is full of stories of people having trouble with large email providers (notably Google and Microsoft) accepting self-hosted users' mail. A self-hosted setup's IP starts with 0 reputation (or worse, depending on who owned the IP address beforehand) and will face throttling and outright blocking for a painful amount of time.
Furthermore, if you forward your email to e.g. a Gmail inbox, any spam sent to your address will go towards lowering the reputation of your forwarder, meaning that someone just has to send you a lot of spam to get your forwarder blocked from Google servers.
SPF, DKIM, DMARC and TLS all cannot mitigate the above.
More like, relying on SpamAssassin to reliably and correctly classify spam is the anachronism here.
Even if SpamAssassin could do a good job of filtering spam today (which it can't), the set of messages that SpamAssassin classifies as spam will inevitably differ from the set of messages that Gmail classifies as spam, so the only thing this achieves is another point where you have to worry about false positives and training the classifiers.
Furthermore, the problem is exacerbated by poor support of forwarding in modern MTAs. They will queue all received messages first, and when Gmail rejects certain messages during the IMAP session (spam with high confidence, or messages it considers invalid), they result in a bounce, and the thus ensuing backscatter further degrades your mail server's reputation.
The closest thing to correct behavior is Exim's cutthrough delivery feature, which has been implemented "relatively" recently, and even so it is not complete and I've run into issues with it with non-textbook setups.
The big players (Google, in particular...in case it's not clear, the bulk of this rant is leveled at Google; Microsoft is an offender on perhaps 25% of the level, Yahoo just doesn't seem to care) have ZERO interest in "playing nice" with small e-mail servers. My experience running my own e-mail for almost twenty years then finally throwing in the towel and migrating to Fastmail not only proves this for me, but is, according to other sysadmins I've talked to, quite common.
I did everything right when hosting my e-mail. I did it for myself and maybe four others. The volume of mail going out of my server was so low, I could read the outbound mail.log for myself, and routinely did. Every domain had SPF, DKIM, and DMARC records. The outgoing single IPv4 and IPv6 addresses I used had perfectly-matching forward and reverse DNS lookups. I had a script that checked every blacklist I could find every 24 hours. My IP addresses were allocated from my (professional) colocation provider and SWIP'd to me with valid contact information. My IP addresses had been unchanged for seven years and had been stable for ten years prior to that change. My colo provider had a draconian non-spam policy. Nobody ran an e-mail list or anything. I've used the exact same domain for e-mail for over twenty years.
Still, every couple of months, Google (sometimes Microsoft, but far and away most likely Google) would decide to simply stop delivering messages from my five users--remember, including me--to Google's users. The messages would just...disappear. Sometimes, Google would stop delivering the messages mid-conversation. Replies would be going back and forth but then my reply to a Gmail user would go to /dev/null while their messages to me would still arrive. Most of the time, the mail would go to spam (even with that little yellow tag meaning "verified sender" or some such) but 20% of the time the mails would just disappear.
Worse, the destination would say "250 OK" to every e-mail I sent them so I had no way of knowing that the messages had been sent to the big bitbucket in the sky. I got annoyed.
I signed up for Google's Postmaster Tools and Microsoft's similar offering. Microsoft would at least show me telemetry but it wouldn't do any good. Google? Google would proclaim that I wasn't sending them enough e-mail so I couldn't show up on the dashboard.
Finally, after three years of fighting this, it came to a head when I was, at all places, at a boardgame meetup during one of these "blackout periods" and got introduced to a Google SRE who worked "adjacent" to Gmail. "Oh, hey, you should talk to techsupporter, he does e-mail too!" was the opening line. We talked, I ranted, and he actually said he'd try to help. He escalated internally and the word came down from on high on their SMTP team or some such: "domain [mydomain] has insufficient reputation."
What did that mean? Apparently, their system--at least at that moment--thought my domain was "too new" and they tilted towards just binning the e-mail. I got red, saw stars, then gave Fastmail $390 for three years of service on three e-mail accounts and moved everything to them within five hours. That was the last straw, the final injustice. For me, the "decentralized Internet" died on that day.
Why was I so pissed? Observe these dates from WHOIS:
Domain Name: MYDOMAIN.ORG
[...]
Creation Date: 1997-03-10T05:00:00Z
Domain Name: GOOGLE.COM
[....]
Creation Date: 1997-09-15T04:00:00ZThere is a specific reason I recommend doing so: the reputation of the sending IP address almost always overrides the reputation of the sending domain.
A smart host operated by a third party, such as Google: https://support.google.com/a/answer/2956491?hl=en , would have solved your blackholing issues because it's the third party's job to maintain extremely good reputation of their IPs sending mail. Most importantly, they have high volume of legit emails, and volume is key to maintain good reputation. Finally, even if you have a low-volume domain, your emails will benefit of the good rep of the smart host, and thus would not be blackholed.
I will repeat it again: if you host your mail server, you definitely should also use a smart host, and all the spam/blackholing issues disappear.
Google also rewrites the "From:" address to your Google account's address, unless you jump through some hoops to add it to your account. This has become more difficult recently, and needs to be done for every address (which is an issue if you use single-purpose email addresses).
Many things wrong with this sentence:
• Don't like Google? Then use any of the hundreds of other smart host relays: Sendgrid, OutboundSMTP, etc
• The smart host can't read your incoming mail (typically more sensitive than outgoing.)
• Google, being a mail provider, can usually read your email anyway, because all your conversations with Gmail users end up in Google's hands.
«Google also rewrites the "From:"»
Google has multiple smart hosts offerings, but the one I linked to does NOT rewrite the From:.
This. I go to a lot of trouble to follow all relevant RFCs, SPF, DKIM, DMARC, and TLS. My intent is to help other servers know I'm transmitting in good faith, I'm hopes I'll get the same fair shake.
If it's still not good enough for you, please at least let me know that fact, with a message I can investigate. At the very least, don't outright lie, which is what I take a bogus 250 to be doing.
This was your actual problem. The large providers (gmail, outlook, yahoo) and appliance vendors (ironport, etc.) are all now treating "unknown" senders as spammers. If you don't send a large enough volume of emails, you will never make it into their reputation tracking lists because "it's not worth it". End result, you are considered unknown and will be treated as a spammer forever.
It is a huge organically grown cartel where the protocols work but you cannot enter a small player, no matter how well you behave. It is a shame, but there's nothing you can do about it :(
Yes, with mixed results.
For one instance, see the trouble Linux people are having with getting messages delivered to Linus' Gmail inbox. I recall reading that some have needed to sign up for Gmail specifically just to get their patches to Linus. Plus, the 2015 occurrence with Gmail having a 20% false positive rate: https://plus.google.com/+LinusTorvalds/posts/DiG9qANf5PA
Counter-anecdote: I have run my own mail server for several years. I use it for all my email communication, and also send out daily emails to a couple of thousand subscribers[1] who are on all sorts of email providers. The "canary accounts" I have set up on all the big providers all seem to have their subscription messages delivered straight to the inbox just fine, though I should probably monitor this better somehow.
[1] the Aphorism of the Day, https://www.aphorismsgalore.com/
This is pretty much what "hosting your own" really means. Having SSH access to some virtual machine somewhere is much better than letting gmail read all your mail, but it's still isn't self hosting. You could still use the remote host as a mere relay, but it's still not ideal.
Real self hosting means having your box at home, sending and receiving mail directly. And that is effectively impossible. Self hosting is dead. I'm not sure how we could resurrect it.
Of course, I don't do this because I don't want to rely on my consumer internet connection or domestic electricity provider, and my flat is too small for a rack anyway. Also the noise, electricity bills and waste heat play a part.
Some things are best done centrally. I am happy with servers in the cloud, I really don't miss having to travel half way across the country to replace a hard drive...
As I said, "You could still use the remote host as a mere relay, but it's still not ideal". And by the way, as much as I'd like to host my email at home, I still gave in somewhat and use a remote virtual machine (at https://gandi.net).
> my flat is too small for a rack anyway. Also the noise, electricity bills and waste heat play a part.
You need less than 5W, a small shoe box worth of space, and no fan at all to host your email at home. R-Pi, Sheeva plug… A web site with moderate traffic might be more problematic. But we're nowhere near a rack. For remote hosting, you don't need more than a fraction of a CPU core.
Reliability, though, that can be a real problem. And a big reason why I still use my virtual hosting provider. I'd love a redundant setup, where if something goes down, the backup takes over, and notifies you about needing to replace a piece of hardware. (Of course, it should be plug & play so my grandma could do it… one can dream.)
As are many cloud hosts / VPSs. AWS, OVH, Digital Ocean, etc I've heard plenty of horror stories from people trying to run mail servers on all kinds of cloud platforms (and personally experienced issues on EC2 instances myself)
Oh yes it is. I mean for professional sysadmins it may not be, but try talking any normal person through buying a domain, getting an SSL certificate, attaching that at your MTA, configuring all those fiddly DNS records that prove you're emails are who you say you are. And all of that isn't even taking into account how needlessly painful the actual process of just installing the MTA is.
And what do you have for show from all of that? Still no real assurance that half your mail won't end up in spam, no web front end (that step alone adds several additional layers of complexity), less guarantees regarding up time (since you no have a single server that you need to manage yourself), arguably worse security (since you're now asking a layman to harden a Linux box) and the whole thing comes with a heftier price tag than Gsuite.
We're not talking about self hosting an IRC server or other service you can throw up in an hour and then forget about. Email is a frustrating sequence of a multitude of steps where even just one of them could render your set up worthless. Hence why even many of us sysadmins have long since given up self hosting email servers. It's just not worth the effort
using certbot:
sudo ./certbot-auto certonly -v --standalone --standalone-supported-challenges http-01 -d mail.mydomainname.com
where mail.mydomainname.com is whatever is the full hostname of your MX.
then a pretty typical postfix configuration blob:
# TLS (SSL) private and public keys
# obtained from certbot-auto in /etc/certbot/
# sample CLI to obtain new cert
# sudo ./certbot-auto certonly -v --standalone --standalone-supported-challenges http-01 -d mail.mydomainname.com
smtpd_tls_CAfile=/etc/letsencrypt/live/mail.mydomainname.com/fullchain.pem
smtpd_tls_cert_file=/etc/letsencrypt/live/mail.mydomainname.com/cert.pem
smtpd_tls_key_file=/etc/letsencrypt/live/mail.mydomainname.com/privkey.pem
# TLS (SSL) configuration
smtp_tls_security_level=may
smtp_use_tls=yes
smtpd_use_tls=yes
smtpd_tls_loglevel=1
smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
# See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for # information on enabling SSL in the smtp client.
(Anyway, the most significant difference between the two is that STARTTLS Everywhere can also enforce certificate validation on recipient sites that request it, which Postfix won't otherwise do.)
The easiest way is to forget about HTTP and use DNS verification. Just create a dynamically updateable TXT record in your zone that the LE client can update and then point CNAMEs for the names that you want to create certificates for to that TXT record.
You simply set up a dynamically updateable zone and then have the LE client call nsupdate, it couldn't be any easier, you don't even need sudo.
You could also delegate DNS for your MX to the MX itself, of course, then you'd have pretty much the same setup as with the HTTP server on the MX, just using a DNS server instead. But that's just too complicated, really, using dynamic DNS is flexible and allows you to easily add other services that need certificates on any machine whatsoever with no need to have an HTTP (or any other) server running on the machine reachable from the outside.
The "STARTTLS Everywhere" project [0] -- the one thing this entire article is talking about -- is what's new.
> ... I've been using STARTLS on SMTP for almost 20 years ...
That's great -- I have too -- but I have some bad news for you: not everybody else has been! (I know, right!?)
The EFF is simply trying to bang the drums to get everyone's attention and then convince those that aren't already using STARTTLS (everywhere, hence the name) to consider doing so.
The good news, for you, is that you're already done. There's nothing for you to do, unless you want to add your domains to the "policy list" so that others who are participating won't send mail to you unless encryption is in use.
Fortunately, the EFF has already written about this extensively [0], so there's no need for me to rehash all the details (coincidentally, that same exact web page is also reachable by clicking on the first link in this article).
I submitted my domain and it’s not on the list, and there has been no communication about why or why not (given that it passes their security checker and requirements).
Couldn't we just put this in DNS?
A point that may be obvious, but I would like to make explicit, is that unlike on web, end-users have no good way to express whether they prefer security over deliverability on a connection, which makes downgrade attacks more difficult to deal with. Everyone defaults to preferring deliverablity (falling back to plaintext). In the absence of user intent, it's more important for an email sender to know the recipient mailserver's intent: what level of security should they expect? This can be achieved at varying degrees through DANE, MTA-STS, or a "HSTS Preload List" equivalent for email, but there is currently no reliably deployed standard for this.
(edit: something I forgot to mention: of course, you'd have to trust the initial DNS lookup as well)
Seems like (with absolutely no thought here) a new header could me minted for this - eg. "Transport-Security: Require". Tough part as usual would be getting MTA and client support, but switching to a per message approach would at least allow incremental rollout - and critically I think - allow introspection by following bounces.
Incidentally, that same university in the past year started injecting footers into people's emails, so I guess their primary focus is neither privacy nor security.
Yes, this is how my server is. When exchanging smtp with another domain's mx, if that other smtpd will not do TLS, it sends plaintext mail. For clients submitting mail via smtp with auth, such as thunderbird on my desktop PC, of course it does TLS1.2.
I remember reading they were really upset when this email providers started using this by default... always a good sign when warrantless/dragnet surveillance is made harder.
> The STARTTLS Everywhere project has created a “STARTTLS Policy List” where supporting organizations can assert that they always use STARTTLS using a valid certificate. Organizations should join that list and check that list before they send email.
We don't need a new separate place to publish the TLS ability of our mx. This can be done with a txt record in the DNS zonefiles same as spf.
It would need some sort of consistent rfc-guided format for what TLS policy re: downgrades, port 25 plaintext, etc, that you want to publish.
Not too dissimilar from publishing an HSTS policy, but done by the authoritative nameservers for a zone rather than on an individual httpd.
https://mail.sys4.de/pipermail/dane-users/2018-March/000445....
He also runs stats every month on DANE adoption.
https://mail.sys4.de/pipermail/dane-users/2018-June/000460.h...
We just need a way for domain owners to signal that their domains definitely accept email over STARTTLS. At that point, senders which recognise those signals can start to enforce that mail which they send to those domains MUST use STARTTLS. Fortunately, those signalling methods are beginning to rise to the surface. We have MTA-STS, and now also a preload list thanks to StartTLS Everywhere.
Unless you're recommending to not fall back to port 25 after a failed connection to the tls-on-connect port. In which case, see the comment you were replying to where I said it would be backwards incompatible.
Both of the solutions I mentioned are backwards compatible, and protected from a similar downgrade situation.
IMHO, being backwards compatible is necessary for a transition period. The messaging to SMTP operators should be... get on board, or you are going to start losing mail in 1 year.
Even Chrome is going to show all http as insecure in the very near future. Which reminds me... I need to get all my sites upgraded.