DIY Mail
computer.rip
computer.rip
Mail is hard to set up properly, but it's not ridiculously hard to set up. Anyone with average Unix administration skills can do it. I might add that OpenSMTPd is a lot easier than postfix and makes a dangerous open relay misconfiguration hard to do, so it's my recommendation to the uninitiated.
It requires maintenance attention, but not a ridiculous level like some suggest.
What is ridiculous is the effort you will go to to stop your connections being bounced, or (worse) your emails going to spam folders. It's this third reason why I use Fastmail to run my email. Technical accumen is of some but limited assistance here. In essence you will you will be struggling to build an IP reputation with low traffic flow. This article has some interesting insights on this topic.
Most things are easy if you’re willing to accept an end result that doesn’t do one of the two (3? Maybe? Send, receive, search?) primary functions.
I know probably 4 or 5 people who are far more than capable of running their own mail server and have given up on it these days, bowing to the inexorable market pressure of gsuite/google workspace/office365 for their MX, because they just can't be bothered by having outbound mail deliver-ability issues. These are people who were running smtpd for regional ISPs in 1997. It almost seems like the google and microsoft spam filters are a bit of a protection racket, forcing people to buy their services.
The author of the article covers this:
>
If you self-host email, you will run into an elevated number of delivery problems. That is a guarantee. Fully implementing trust and authentication measures will help, but it will not eliminate the problem because providers weight their IP reputation information more than your ability to configure DKIM correctly. Whether or not it becomes a noticeable problem for you depends on a few factors, and it's hard to say in advance without just trying it.
Federated systems like email tend to rely on a high degree of informal social infrastructure. Unfortunately, as email has become centralized into a small number of major providers, that infrastructure has mostly decayed. It was not that long ago that you could often resolve a deliverability problem with a politely worded note to postmaster @ the problematic destination server. Today, many email providers have some kind of method of contacting them, but I have never once received a response or even evidence of action due to one of these messages... both for complaints of abuse on their end and deliverability problems [3].
I don’t have any evidence for this view but have believed the same for several years.
I guess what I'm saying is that I'm not conspiratorial enough to think that they created this situation on purpose, but now that it exists they sure are benefiting from it... and that undoubtedly reduces incentives to allocate more resources to abuse prevention/response at the source.
I mentioned in a footnote in the article my anecdote about a university switching to G-suite, more or less because of a serial abuse problem originating with gmail. That's not the only reason of course and there were institutional politics, cost optimization, etc involved... but to be quite honest I personally feel that Google has various ways to strong-arm universities into G-suite that are reprehensible and an abuse of taxpayer/tuitionpayer dollars. In my opinion, and at that time, even constraining ourselves to the inevitability of going SaaS the Microsoft 365 offering was both superior in quality and more cost-effective (given specific requirements of the institution like desiring interop with on-site AD and US data custody agreements... YMMV). Microsoft was also amazingly easier to work with from a pre-sales perspective, with a very accessible sales rep who got quotes and answers to detailed engineering questions very quickly, including setting up some meetings with customer engineers to plan architecture. Google was, well, more or less a brick wall... we spent more than 5 months waiting for an updated quote at one point (we were looking for some features that Google said they offered but not on their published pricing, although it later became questionable whether they even offered them).
I don't want this whole thing to turn into a rant, but, well, the university chose Google, and it really felt to me like that was largely a result of some fairly dirty tactics by Google that involved leveraging their partnerships with other academic software vendors to basically box the university out of features of other products it used if it did not select Google. Google and the vendors presented this to us in a way that made it feel remarkably intentional (i.e. the vendors were surprisingly candid that they offered integration only with Google due to commercial incentives). I also felt that Google was outright deceptive about their pricing and capabilities at points, although it felt like it came mostly from the incompetence of their sales staff (promising before they found out if they could deliver, which did not happen until post-contracting). At the time I wondered if any of this rose to a possibly actionable violation of state purchasing regulations but from my perspective now I doubt it... nonetheless it was an extremely frustrating experience that left me with a very negative impression of Google's enterprise offerings and business practices, despite their products generally being more polished and user-friendly than Microsoft's (although frankly I feel that for the last two years or so Outlook Online has surpassed gmail in terms of UX, Microsot has visibly put a lot of work into it and Google has done little except for somehow make it slower).
I suppose Google's success in the enterprise has not changed the fundamental observation that Microsoft gets enterprise sales and Google does not. Google just has the bluster and name recognition to sell anyway.
That is expressed in such understated terms that one might be forgiven for suspecting the OP might be blowing smoke (he wasn't, of course; he just seems to be the sort of guy that doesn't care to rant).
I thought it was a very good, well-written article.
I think it's a shame you can't reliably self host mail without jumping through hoops around IP reputation. But it's just the reality. You can be an idealist or you can be pragmatic.
In a similar vein, I now only ever use transactional email services (SES, Sendgrid) for automated emails. The sacrifice you make with self hosting is also significant in this context.
In general I've found Apple Mail to be probably the most aggressive major provider in terms of rejecting at the SMTP level. Microsoft seems to virtually never do it unless there is a major problem (i.e. SPF exists and prohibits the sending mail server). Google is somewhere in between, but seems to stop SMTP hard rejecting at all once it's seen the IP in use for a couple months.
The unfortunate thing, of course, is that hard rejections at least give you feedback. When they accept the mail you still don't know if it's actually made it to an inbox.
The IP reputation issues that will prevent your emails from being delivered are not really a technical matter.
Almost none of the mail gets through.
Now I am researching outsourcing the site's mail to one of the usual suspects: mxroute, FastMail, ProtonMail, Exchange, etc. Perfectly configured mail that no-one gets to read is worthless.
I'm now down to the point where I only get one or two ham messages in my spam folder a month. I don't get very much actual spam to that address at all; presumably they have an even higher level of filtering where it doesn't get delivered at all.
>Do not cheap out on providers. Run your mailserver in the most reputable IP space you can find. Most likely you will use some type of cloud instance or VPS (probably in such a way that the line between the two is blurry). Unfortunately, per-month cost tends to correlate directly with IP reputation quality, as does name recognition. But neither are any guarantee. AWS has plenty of IP reputation problems. Sometimes a little-known, brand-new VPS provider can be a surprisingly good option because they got their IP space off of some staid corporation that maintained an impeccable security program. Do some research to try and find out what kind of results people get from different providers.
> From an IP reputation perspective, consistency is key. Mailservers and IP reputation are like credit cards and your credit report: once you open one, you want to keep it as long as possible so that it will build a good IP reputation. Do not move your mailserver around between providers to save money, it will result in headache.
> Do not run a mailserver off of IP space that is not intended for commercial services. For example, running a mailserver on a residential ISP is a terrible idea (if it even works, many residential ISPs drop outbound on port 25). You will be guaranteed to have IP reputation problems and, moreover, IP space allocated to end-users usually gets reported to a "policy block list" such as the one Spamhaus maintains... meaning your outbound email will be blocked by recipients simply because it's coming from an IP that should not have mailservers, so it's most likely to be from a compromised end-user device.
===================
Now to add my own 2 cents:
If you are serious about doing this, find a medium sized long-established ISP where it is still small enough that you can get in contact with the people who run the routers. And that ISP should be one that really cares about its IP space reputation.
If you can afford it, anywhere with long-held IP space belonging to its ASN that has never had $5/month VPS customers on it, or any form of shared hosting environment, is much more likely to have "clean" IP space.
https://www.seattleix.net/join
Don't have to connect to the SIX or have an ASN, just buy default routes from somebody with a presence near the "core" of the internet in Seattle. I'm using the SIX page there as a handy list of westin and westin-adjacent colo providers.
That's what the "2FA mule" is for.
It's a stock android phone with no google account and no apps installed except for "SMS Forwarder"[1].
It is configured to forward all SMS to an email address via encrypted SMTP. This means that I can receive these 2FA codes anywhere I have Internet access - such as an airplane or newly arrived in a foreign country where my SIM card does not work.
The "2FA Mule" itself is plugged in at my office in a corner.
I'm not employing this for anything sensitive but it's interesting to consider that I can use SMS based 2FA while divorcing it from my day to day SIM identity ...
[1] https://play.google.com/store/apps/details?id=com.frzinapps....
- F-Droid: https://f-droid.org/en/packages/com.github.axet.smsgate/
- GitLab: https://gitlab.com/axet/android-sms-gate
A client of mine had corporate clients who relied on MS's hosted email solution. Often emails would be accepted from our MTA onto theirs, but never show up in inboxes or spamboxes. They were just silently discarded, even though the IP had pretty good reputation (and had been used to send email for years).
Eventually, the problem got so bad they just had to switch to a hosted email solution, since there was no way to work around it. I would have insisted the other company deal with it, but they wouldn't pay my client if the invoices weren't emailed, so we didn't really have a choice.
Maybe it's not too late. You'd be doing all of us a major service.
Cite?
Of course, they can't do that with other big players (like Gmail) so they're allowlisted, because they would be afraid of lawsuits and their users would complain about not receiving email from Gmail. By applying this to smaller providers only, they maintain plausible deniability that the problem lies on the other side, a sentiment which is reinforced by maintaining their tech support's lower levels unaware of the scheme (the only ones you can talk to for months after initial complaint).
Citation for this? We have a witness in the room! WhyNotHugo's comment is a primary source.
The integration is extremely easy to setup [2], and solves the deliver-ability issue. Most of these services also take care of the DKIM and DMARC setup, so it is much easier to get things right. And they all offer debugging tools if something isn't sending or returning correctly.
On the other hand, they are not in privacy abusing position that Gmail is, and I also don't have to worry that they will close my account with no recourse, as Gmail once did to me.
The biggest downside is no email forwarding. There is no way to do email forwarding using a service, since they only send mail from approved senders. (There are Bad Idea solutions, such as rewriting headers with SRS. Don't.)
And of course, I need to pay for sending. Mailgun used to have a generous free tier, and Amazon SES is decent at $1/10K mails, but things add up. I chose Postmark for many reasons, but the pricing of different companies is worth looking into if you are sending enough mail. (Sendgrid and Mailchimp Mandril, for example, have a unlimited per deliveree charge, that could conceivably be cheaper for some use cases.)
OT, the only reason that I support PoP3 is for Gmail: The only way to "check an external account" to write and send mails in Gmail is PoP3.
[1]: postmarkapp.com [2]: https://postmarkapp.com/support/article/832-can-i-configure-...
More people getting familiar with how reliable email systems work is a good thing.
With regard to sending mail, a few things go a long way. Check your IPs on dnsbls before sending email from them. Don't expect to send mail successfully from a public cloud address. have at least one backup mail system that isn't listed in any dnsbl. Use separate IP addresses for transactional email, user email and bulk email. Deploy SPF, DKIM and DMARC for your domains.
The article pooh-poohs iRedmail and some others, and while he's not wrong, I can say that iRedmail is about as close as you can get to learning all about the inner workings of a mail delivery system and still be reasonably one-click install. It is opinionated to some degree, but it doesn't lock you in. I've used it for years and have been happy. Unfortunately, delivery problems (as the author noted) meant I have to use Mailgun for their relay service.
I think this is a fight we should continue to have. It's worth it.
I don't think we're trying to dissuade them. I think we're fighting the insidious memes of "Look, it's easy! I did it in 5 minutes!" which perpetuates a potentially hazardous misunderstanding. If you don't do e-mail right, you can end up sending or receiving an important piece of official mail that nobody will ever read. Job offers gone unseen, billing reminders turned invisible, customer service tickets lost, warranty redemption missed, urgent family notices not gotten in time, etc. We're just saying it's much more nuanced and unintuitive than just turning on the software.
It's like the blind leading the blind: somebody's going to end up in a pit. We're putting caution tape around the pit. They can still choose to go under it and explore.
[1] https://jschumacher.info/2021/05/running-a-private-mail-serv...
PS. It’s not mentioned often but when you send out emails you need to send at least 100 emails per IP per large destination _every day_ (100 to Gmail, 100 to O365 etc). Otherwise you’re not even on the radar and can’t build any reputation. You need to have a constant mail flow going out for the big guys to take you even remotely seriously.
IPv6-only for email is probably still a pipe dream, I guess...
On the plus side, since it's all pretty fresh - you can do things like "MUST be DKIM signed from an aligned reverse DNS" on IPv6 and have a chance of it working.
I'm curious how you came to that conclusion.
Messo's question above made me curious, so I just did a quick query on the database of Mailhardener. >99% of email coming from Gmail is delivered over IPv6 (from ~10k samples).
This appears to be provider specific though. For example: Yahoo seems to mail exclusively over IPv4, Zoho also IPv4. Most exchange based services appear to prefer IPv6. Microsoft 365 also uses IPv6.
So your conclusion doesn't really reflect what I am seeing in our database. Obviously, we process mostly DMARC reports, so maybe regular email is more IPv4 biased to support mail services that are not IPv6 capable.
IP addresses (especially IPv6) are ephemeral, email providers know that. Just about every email service has a TTL on IP reputation.
Usually, if an email is DKIM signed (and aligned) then domain reputation is used as the key indicator, the sender IP becomes less relevant in that situation.
Obviously, each implementation is different. There are no silver bullets here.
But what I'm trying to say here is: do not worry about inheriting a 'tainted' IP address. With the limited number of IPv4 addresses available, and the huge volume of spam, just about every publicly available IP would have been 'tainted' by now.
I had actually given up on mail months ago, in the sense of not considering it a reliable channel anymore. But I haven't been strict about enforcing it, and instead live with the risk of missing important messages.
* You probably shouldnt use email for sensitive communications ever again.
* While it's not worth self hosting a mail server to send and receive email there's still a huge privacy value in hosting email storage and using a service that only passes on emails via POP3/SMTP (ideally with minimal logs).
When I write new posts, I use a shell script that runs them through aspell, then runs the site generator, does a simple concatenation to build the email version, and calls pandoc to build the secret hidden pdf version.
Anyone using Protonmail with a custom domain?
https://www.migadu.com/pricing/
You do lose the layer of encryption protonmail provides. But it is essentially boils down Encryption at Rest and e2e when between two protonmail users. It can be mimicked by pulling down the mail from the servers to your machine and using pgp.
It’s not an all- or nothing question.
Similarly, if you’re worried about the risk of potentially losing incoming email if you have issues, you can keep your existing mail provider choice as fallback in the DNS and fetch+delete mail going there into your primary mailbox every 5 min or so using mbsync.
Nationalized digital mail isn't that much of a stretch. You can still run private digital mail, just like you can hire private physical mail couriers.
But probably it won't happen. Capitalist countries prefer private industry does things. E-mail could become a regulated utility, though.
The really painful things are either places that subscribe to dodgy blocklists and never check their configuration, or... yeah, places like t-online.de and free.fr.