SMTP protocol basics from scratch in Go: receiving email from Gmail
notes.eatonphil.com
notes.eatonphil.com
On the outbound side, many accepters of mail will deny mail from SMTP servers on a residential IP address, as provided by the Spamhaus Policy Block List (https://www.spamhaus.org/pbl/) or equivalent. Unfortunately, there's not much one can do to get off this list - ISPs self-publish their own residential IP space onto it. In order to self-host from home, business-class service or higher is in order. Or, again, use the public cloud, and hope your compute instance's IP isn't on some other block list.
On the inbound side, though, many ISPs block inbound port 25. Since that's the agreed-upon port for mail receipt, there's no real way around this. Several times while I self-hosted off residential, I was able to call Comcast support and ask for the ban to be lifted, but not before fighting through several layers of level one support asking me to configure my mail client differently. The real kicker, though, is any time my modem was rebooted (I owned my modem, too!), I had to call in and request it _again_. My (very small local) ISP in upstate New York also performed this blocking, but _would not_, in the face of any complaining, lift the block.
Also I am using windows (smartermail) so even a cheap vps isn’t that cheap.
However, I don't run it from home over residential ISP connection.
I mostly run the mail for my family and myself, so the attack surface is reasonably small.
My network mostly only dies when one of the ISP-provided modems slows down too much without actually killing off the actual connection (so pings seem to keep working from my OpenWRT router to detect dead connections), which requires resetting that modem (happens only every few months, rare enough for me to bother debugging it: it's usually my dad who pings me early the next morning that his email is not working :)).
I am only mentioning bandwidth because someone could easily saturate my internet connections by extensively brute forcing, and that's another mild concern (perhaps those are things that kill my modems?).
In practice, biggest issue is that Gmail in particular will never rank your server as trusted because you are sending too low a volume (how's that for an anti spam measure?), and 1/3rd of recepients report emails ending up in spam folders. Curiously, only Gmail does that.
That's super easy to avoid. Use strong passwords.
Heat death of the universe will arrive before anything brute forces a 30 character password generated off /dev/random
I don't have any of my users (just family) set their email passwords, those are generated to be strong. These are not passwords that need to be seen or remembered by anyone, they just go into the IMAP client config.
99% of people don't use their home computer to write programs. Some people who write computer programs at home cause trouble. Should we therefore ban programming languages for individuals?
Of course, how well your support requests are handled is another matter.
A tunnel to a vm hosted on a cloud provider is far from "running SMTP servers on a residential IP address" that OP was referring to.
Outbound ISP SMTP servers should work that way. This makes email behave much like instant messaging.
Messages with many recipients may need to take a store and forward path. They can be redirected to a slow bulk mail server.
Historically, mail servers are store and forward because the destination might be offline, or on some store and forward UUCP system or something. Today, that's very rare. We don't have to emulate Sendmail forever.
I wrote one of these years ago, as a front-end to my Exchange server. At the time Exchange had this thing where it ALWAYS accepted messages as long as the domain part was an accepted domain, and then it would do a fuzzy match on the mailbox names and dump the mail anywhere it felt matched.
So I had to have this SMTP front-end forwarder to weed out mis-addressed mail... which I eventually added SpamAssassin to as well. It ran for over 10 years without issue.
What I never solved/wrote was the TLS stuff, as it always seemed to complicated to code from scratch.
http://smtpd.github.io/qpsmtpd/
I'd reject spam at SMTP time, but also capture the rejected messages, so that users could view them in an online quarantine. Worked pretty well, though I think these days the original perl-based project has been reworked into a nodejs thing:
Cool article.
I setup a wireguard tunnel back to a mail server hosted at home once, gotta have that PTR and my ISP isn't giving me one. I cobbled the instructions together from 3 or 4 different blogs to get exactly what I needed in my particular situation. There was all this stuff about source NATing which was like a foreign language to me and I was in "get it to work" mode so I just kind of blew past it. It seemed interesting and I got the gist of it but in the end I had a script full of iptables commands and I'm not really sure what they did, other than that they did what I wanted.
I love that he mentions something like that here. There's a real temptation to sweep that kind of thing under the rug to puff yourself up and look more knowledgeable, and it certainly would have gone unnoticed by me if he chose to do that. Props.
My problem is the opposite: I would have stopped and not continued until I knew what zones were – and any other part of the technology stack that might potentially have security consequences. With my (overly) cautions/conservative approach, I might still learn a lot but I don’t get the same sense of excitement – or accomplishment – as the author.
Does the latter actually happen? Do smtp clients actively check this? What’s to stop some middleware performing a tls downgrade?
Adding the appropriate option in Postfix fixed the problem, and I think there were further options to require TLS.
I think you are talking about MTA-STS (rfc8461) [0]. It is currently being adopted by the larger email services (Google, Microsoft).
MTA-STS is not 'hosted' in DNS, but published via a policy service over HTTPS, so it uses PKI.
I work for Mailhardener, we offer MTA-STS policy services as a service [1]. We've seen an increased interest of MTA-STS recently, likely due to Microsoft announcing MTA-STS support [2].
[0] https://datatracker.ietf.org/doc/html/rfc8461 [1] https://www.mailhardener.com/blog/introducing-hosted-mta-sts [2] https://techcommunity.microsoft.com/t5/exchange-team-blog/in...
Anything else would probably not render correctly, especially if they use other encodings which would make the message unreadable without extra parsing (think koi8-ru)
This program responds 250 OK to absolutely every command other than data which doesn't exactly make sense, so I assume it's a toy example. You don't send 250 to QUIT, for example. Nor to STARTTLS.
You need to pass DKIM and MTA-STS specifically as two things most tutorials on 'sending email in golang' didn't tell me. I'm considering writing my own article on sending -deliverable- emails, because it's probably pretty important your stuff doesn't end up in spam.
This is the second sentence. What does this even mean?
What do you mean by this?
If you send spam through Google, they will send your administrator an e-mail telling them that they are sending spam. If you send "a large volume" of this spam, they will stop relaying it until you promise not to send spam (or follow their guide to make your spam not look like spam).
More generally: How does this weird trust system work? Is it just an inofficial, loosely connected club of large actors who trust each other enough to whitelist?
Now, regarding the "trust" system: the reality is that every e-mail provider that is receiving e-mails chooses its own policy. In general, a provider receiving an e-mail will choose one of:
a) Accepting the e-mail and delivering it to the users' inbox
b) Accepting the e-mail and delivering it to the users' spam box
c) Rejecting the e-mail at SMTP time (at least the sender realizes it hasn't been delivered)
d) Accepting the e-mail and quietly deliver it to /dev/null (the sender thinks it has been delivered, but the recipient cannot ever know about it).
Providers try to be as smart as possible to pick one of the 4 actions above. They do it by using a combination of rules that happen at different times of the delivery process. For instance:
- Noticing that an e-mail is being sent to a non-existing address is easy (a simple check against the addresses db). Hence, most providers do this during the SMTP conversation (and choose option c above when the recipient doesn't exist).
- Checking a zipped powerpoint attachment for viruses may take a while. Hence, most recipients do this _after_ having accepted the e-mail where option c above cannot be used anymore. For viruses, most providers would then go for option (d).
Now, when people speak about the "cold IP" problem they are mostly speaking about one of the common techniques used by large providers (gmail, outlook, etc.). Do note that business domains that use Google Workspace, Outlook 365, etc. are also managed by these large providers.
What these providers do is they keep an internal "reputation" map for every IP (and/or network) that they've received e-mail from in the past. These lists are private and the precise effect they have are treated as internal secrets. However, between the scarce statements made by the providers plus the observed results by many senders the following "common understanding" ensued:
- When they don't have information about an IP, they treat that as a negative signal. No reputation for a specific IP + bad network reputation (for instance, anyone sending e-mails from OVH has bad network reputation because there are many spammers in their network) probably leads to the recipient applying (d) or (b).
- When an IP that hasn't built a reputation starts sending large volumes of e-mail, they treat it as a _very_ negative signal. (d) is practically guaranteed here.
- Unfortunately, most providers _also_ require a minimum sending volume to start tracking the reputation of an IP. This is what bites self-hosted users the hardest (whether they know it or not). For instance: outlook only updates an IPs reputation when that IP has sent more than 100 e-mails (to outlook-managed addresses) that day. If you send less than that, your IP will never build a good reputation there even if the recipients mark every single e-mail you've ever sent as non-spam.
Finally, notice that this is an ongoing arms race. Spammers try to adapt to these measures, and e-mail providers keep changing their techniques, scores and effects. The really bad place you can find yourself in as a self-hoster is the following:
- You've taken all known measures to be a nice sender (you have a reverse record, you send mails through TLS, you've got SPF/DMARC policies setup, you are DKIM-signing your e-mails, you _never_ send spam e-mails, you've signed up for the feedback-loops at all major providers and everything).
- You've had regular e-mail conversations with john@outlook.com over the last few months without any issues.
- You send an e-mail to john@outlook.com and outlook's servers accept it.
- John _never_ receives that e-mail (not in their inbox, not in their spambox, they just don't have it anywhere). For the avid reader: outoook has taken the (d) option above.
Now, if John was expecting the e-mail he will probably complain at some point and you'll both find a way to get the information to him. However, if the e-mail said "Hey, I've convinced my wife to stay at your place for the weekend! We'll arrive on Friday around 8pm." you can imagine how this might be a problem.
The fist time something like this happens you contact outlook's "support" to complain about it. If you are lucky, they reply saying that "your IP qualifies for a temporal exemption" or something like that. Then it doesn't happen to you for a while, until it does. If you are unlucky, the response is on the lines of "send good e-mails and tell your recipients to mark them as non-spam and this will stop happening to you". But the recipients aren't receiving any of you e-mails, so they can't mark them as non-spam. Now you waste a few hours going back-and-forth with the "support agents" until you give up in despair.
Then you see an article on HN about how easy it is to self-host your e-mail. Sadly, you know that self-hosting (to receive e-mails) is easy enough and trouble-free. However, self-hosting to _send_ emails is another story entirely, a story where you are all but a small bug in a world of elephants that may (accidentally or not) stomp you without even realizing nor caring the least bit.
Low reputation traffic gets temporary failure codes at smtp time, not silent acceptance nor permanent failure.
It's unfortunate that we rely on opaque proprietary systems and informal deals between providers for what is arguably fundamental infrastructure.
It feels awfully hard to fix this within email without breaking the end user experience. Do you have any thoughts on how to improve it?
Has anyone got a Python take on a bare bones SMTP mail server/relay and client because that would be super useful for teaching email concepts in a sandbox where full-blown Exim/Postfix etc us too clunky.
(irony)
lol