Exploring Self-Hosted Email Services
synergeticlabs.com
synergeticlabs.com
Then I'll remember I just spent 6 hours of my Saturday fucking around with it and now have to live with it and maintain it. So I go crawling back to outlook.com like the pathetic corporate whore I am and go and do something less painful the day after.
I'll just keep my private comms off email.
In other words, I can only protect email from my provider as long as everyone else on the email protects it from theirs. And it only takes one gmail user on the thread to ruin it for everybody.
If I'm corresponding with people who also care about this kind of thing, we aren't using email. If we absolutely have to, then we can encrypt it with PGP.
My email use cases are: (1) b2b professional emails, which you expect to be retained for compliance, etc. and (2) transactional emails, which my provider unavoidably has access to and which I protect with strong passwords and 2FA. (That said, I would like to obscure the list of my internet accounts from google, so maybe one day I'll migrate my transactional emails somewhere else.)
Realistically an absolute fuckload of spam and not a lot of genuine email comes out of residential and cheap VPS hosts IPs so they already have a negative rep.
The good reputation IP range is the biggest problem. I have a more expensive ISP at home that takes its network seriously, so I look more like a commercial interest in that regard. There are decent services that will act as relay for you and probably cost less than such an ISP (or a similarly more expensive VPS provider that takes priority care of related things) if outgoing SMTP is your main reason for considering one.
I send a couple newsletters (one tech related, other for marketing for a coffee shop/bakery I own), and it was fairly easy to land on inboxes. Now, I don't know with a 100% certainty that they were delivered, but if DMARC records and bounces are anything to go by, self hosted emails aren't all that hard.
When you buy a VPS, it helps to check if the IP you receive is already blacklisted. After that, it's easy to use some outside service to monitor DMARC responses and blacklist. There are about 40 black list databases. My domains and the IP address are on none of them.
I use Outlook with a family account, and they used to allow custom domain aliases. All of my email is addressed to addresses on my domain. It's actually a pretty good value for 4 people.
If they ever decide suddenly I'm out or they're out, I'll spin up a mail server somewhere else while I figure out my next move
They’ve been working hard to remove features from that capability. I assume it was either more than they could justify based on adoption rates, a nightmare to maintain, or seeing more abuse than they want to manage.
My anxieties over email match yours. I used to run my own email server but the entire IP range I was in got blacklisted by Proofpoint, and they were thoroughly uninterested in helping unless I owned the entire IP range and could prove it. The ISP didn’t seem to care, and at the very same time Microsoft added the same range to their lists, which meant I wasn’t able to correspond with an attorney. That was in June and by August I’d moved completely to Fastmail. I haven’t looked back, though I do sometimes miss managing my own email server.
I've used Fastmail in the past and it was a very good experience. Pricing for four people tipped me towards the MS Family plan.
A solid intermediate solution is to have your own domain and host your own email server for incoming email. Then google/microsoft/yahoo/et.al. will never be able to lock you out of receiving important email.
You can still use a third party to relay outgoing email if you're worried about deliverability like in your example. But you'll never get locked out of receiving.
No immediate confidentiality is achievable by using a trusted mail server while interacting with untrusted ones, but doing what seems right often works as an improvement. I find it analogous to, among other things, not throwing garbage on the ground: by carrying it to a trash bin, you don't make all streets clean at once, and it may be an inconvenience, but once more people do it, streets do become cleaner. And even if others keep just dumping trash on the ground, it may be nice to know that you don't contribute to the unpleasantness. Likewise with a multitude of other areas, including good practices in programming and administration, good manners: it is unfortunate that not everyone follows them, offsetting the effort put by those who do, but it is not necessarily a reason to give up.
Running my own mail server for 25+ years. 50+ domains, quite a number of messages daily.
Most recent maintenance was few years ago caused by Apple phasing out procedure for push certificate generation (I use apple push for mail delivery).
e-mail has a learning curve for sure, but once everything is set up properly, there is no maintenance whatsoever, if you know what you are doing.
It's not just about keeping your private communication private, it's about relying on your e-mail address being there. As long as you use someone elses domain you're vulnerable.
The big providers can shut it down whenever they want.
One solution is to use a custom domain that guarantees some sort of ownership. For example certain ccTLDs have more protection for private owners of their domains. And then use providers who support custom domains.
I've had the account since 1999.
I've been running it for ~5 years this way and it hardly ever lets a spam through (Very rarely the spams that are just images get through) and I've rejected 2 mails that I'm fairly sure were legit (just bulk marketing ones though)
People always say "Use Mailcow!" and I say "But why?". Adding the additional complexity of docker with the random firewall rules it adds, individual containers that need to be fed and watered. It's a mailserver, all the processes are doing the same mail related function. Don't get me wrong, there's a very real place for Mailcow etc, but for a small end-user mailserver I think it adds overhead that makes running a mailserver more complex, not less.
Anyway, great in depth article.
For me the reason I use mailcow is it's a few commands and I have a fully functional mail, calendar, and contacts server with a web UI. It can then be updated with a single update.sh.
Using mailcow I don't need to mess around with virtual domains, connecting it to mysql, DKIM, etc which is far more than a few commands.
I also get a nice admin UI, web mail and CalDAV and CardDAV which you don't get with the above.
Also I'd consider the separate containers a good thing so that if rspamd, or clamav, for example, has a CVE they can't pivot to other parts of your mailserver.
Why? three main reasons: (a) security (as you have identified isolation matters, but that is not the only thing), (b) get the benefits of "battle-tested" setups and (c) features
On security: in its default config, Mailu scans emails for malicious macros via oletools (and optionally viruses via clamav). It also uses a hardened-malloc, Snuffleupagus (a security module for PHP), gates all PHP code behind an authentication wall (webmails), ... and does both DANE and MTA-STS validation to ensure your emails are delivered to the right place. The authentication stack handles "smart" rate-limiting: you get to limit the number of authentications with distinct credentials over a time-period (a misconfigured thick client won't trigger it), you have plenty of ways to avoid running into it (application tokens for thick clients, per-device cookies that give you a way out, whitelisting of "used" addresses, ...) and you also get to rate limit the number of sent emails (useful if a spammer gets their hands on the credentials of one of your users)
On the importance of "battle-testing" setups: well, there are plenty of non-subtle ways of breaking an email setup. Experience has shown that all the layers in the stack can be problematic... I can give you a bunch of examples of what we ran into recently if you want.
On features: your setup might be simpler but your users are missing out. Whether it's enhanced filtering (like with oletools), better indexing (full text search), indexing of attachments (with OCR! via Apache Tika), configuring server-side rules with managesieve or just "having an interface" to configure ooo, change their passwords, configure aliases or delegate permissions.
I have started spending time on Mailu because I don't like the bloat that comes with Mailcow. Give Mailu a shot; it is reasonably easy to debug when things go wrong (and not written in PHP :p).
In fact, I just added another domain to it.
The key here probably is, most people don't want to spend time learning enough about Postfix, Dovecot. Etc. To do that.
mailinabox
Just postfix + rspamd
I’ll have to give self hosting a shot with one of my domains :)
Fixed title: "Self-Hosted Email Setup Guide"
To be fair, there's this bit (1935-ish words into the article) where there's some exploration going on
> A number of free and open source MTAs are available for system operators including Postfix, Exim, QMail and OpenSMTPD. Each comes with their own limitations, quirks and security history.
> I ended up selecting Postfix
Too bad the exploration wasn't shared in this post: that's what I had clicked for
I'm very aware that I'm in a bubble with my peers, so these are certainly opinionated.
I would pick Postfix because it's more maintained, and what most of my peers use it too. It's a PitA to configure, but there seems to be many resources online. Most of the configuration are quite common, so that one-time configuration is manageable.
Exim is pretty robust too. I don't have a strong reason to not use it, except that I don't have many close peers who use it. Really nice configuration. If you have a multi-tenant setup, Exim would serve better.
Qmail, despite being unmaintained, is praised for being quite secure. I suppose it's the low number of CVEs.
There is also msmtp that provides sendmail compatibility, but uses an external smtp server (like AWS SES) to send the actual email. This is what I use when the server I'm sending from has low IP reputation but I don't want to change my app to use a REST API to send email.
sendmail would like a word...
I find postfix is super easy and intuitive to configure. Still have nightmares of maintaining sendmail systems in the 90s.
Exim seemed ok from the times I've used it but I use postfix now everywhere. Feels like it has the best combination of wide support and excellent documentation, good maintenance and being widely used.
There's a much higher sense of achievement when putting things together with deliberate configuration and sorting out how to glue things together.
No one RTFM’s before doing stuff anymore. I have learned over the past many years that this has been key to my own journey and something others are willing to pay for.
I've never had deliverability issues. Because it's nix, upgrades are atomic. Installation is dead simple. Also, the entire setup was like two minutes. I eventually added LDAP authentication and roundcube which took some work, but because it's nix, I did this development on a separate VM before I simply replaced my config on my main server (actually just copied the closures). Adding domains is simple: just modify the nix config file. As is adding new users. Very easy. Highly recommend to run all your critical infrastructure on Nix. Any mistakes are easy to fix... just go back to the old generation.
I did consider solutions like mail-in-a-box but adding the complexity and statefulness of such a system just didn't seem worth it. Also, remote control panels scare me. My e-mail server is heavily SSH protected (Only my private key plus wireguard) and hardened against malicious IPs (fail2ban) among other things. Again, NixOS makes this easy. For storage, simple-nixos-mailserver keeps all data on a separate ZFS partition, which gets zfs 'send't to a remote backup (several).
You bet it does. If you care about the security of your data, run your own server and use the VPS simply for its public IP.
It's a good, lengthy article with lots of meat.
"In order to deliver email to other mail providers without being marked as spam, I ideally needed to implement all three policy frameworks with the mail server."
Then a few lines later:
"Inbound and outbound message delivery was working."
He goes into great detail, and if anything edge-case arises, you can almost always find a comment relating to a fix before the main article gets updated.
I know the risks of taking over someone's old IP to host email on of course, but with TF you could quickly re-deploy somewhere else until you find an IP that is good.
For my own needs, I essentially let others do the heavy lifting and used docker-mailserver, which includes Postfix, Dovecot and some other software in an easy to deploy package: https://docker-mailserver.github.io/docker-mailserver/latest... Actually, that documentation of theirs is a nice read as well!
I've been using this for a low volume of mails for a few years and haven't had much in the way of issues, deliverability, or otherwise. I even got lazy to the point of only configuring SPF, not DKIM or DMARC and have no problems there either (though that might change in the future). Throwing something like a Roundcube container into the mix when I need a web UI is exceedingly easy as well, as is setting up Apache (with mod_md), Nginx (with certbot) or Caddy as a reverse proxy.
It's not beyond the wit of man. I cannot believe that my IPs are blessed or whatever. My mail systems are behind "business" IPs but nothing more fancy.
At work I have six WANs - a 1GBs-1 leased line, a 1Gbs-1 FTTP and four 80/20 FTTC. Our MXs are on two of the FTTC connections. We only have SPF set up, I toyed with DKIM and DMARC but disabled that and left as is.
I run several "vanity" email domains and I never have problems with them. My own is an IP on the leased line, mentioned above.
In the UK, it is possible to run an independent email system. I have been doing it for over two decades.
A friend who also has their domains going through my server did have Gmail give some recipients "are you sure this is the person you think you are taking to?" warnings a while back due to us but having DKIM setup for that domain, so I would say that is becoming necessary these days.
The biggest issue is mail volume to establish reputation. You need a fairly high number of emails before Google or Microsoft's email Postmasters tooling even registers you exist. This leaves new servers stuck in a chicken and egg conundrum. You need to send X number of good emails before the emails are excepted, BUT you can't get emails through without having X number of good emails go through. Old servers like yourself have the advantage of already being "trusted" to an extent.
Mail deliver-ability has been discussed multiple times here on hackernews with the poor state of the postmaster tools being explicitly mentioned in the articles. For example Google is eating our mail: https://news.ycombinator.com/item?id=19756125
> Most of the Postmaster Tools dashboards will only display data when there’s a sizable daily volume of email traffic (up to the order of hundreds) coming from your Authentication Domains and/or certain other conditions, in place to prevent abuse.
https://support.google.com/mail/answer/9983020?hl=en_#zippy=...
Postmaster tool links for additional FAQ's:
You really don't need volume. You need to do it right by following the rules and also not sending crap out. The likes of mail pig and co will never work as I do ... Americans use the phrase "Mom and Pop" for a small family business - that is what I do.
I have a VM at work that acts as a SMTP/IMAP post office for a few vanity domains. I registered another one a few months ago and added it in for a friend and they have no problems conversing with the hyper scalers etc and nor do I and the rest of my small holders.
This may be an artifact of being in the UK but I doubt it. The big sods are run by people like me - professionals, who try to get the messages through, despite a sea of hate, viruses and the rest.
If you really do SMTP seriously, you will discover a world of people who really care about delivery.
I am not an "old server" per se. I refuse to deliver crap and my peers seem to appreciate that. I own my IT company and I keep a pretty tight rein on my S&M department - they do not get to randomly spam people. Strangely enough we still manage to function and profit without being arseholes that blatt the world with spam.
--------------------------------
This is, apparently, my certificate from HE:
<img src="//ipv6.he.net/certification/create_badge.php?pass_name=gerdesj&badge=3" style="border: 0; width: 229px; height: 137px" alt="IPv6 Certification Badge for gerdesj"></img>
I agree that the setup is a complete PITA but to be fair, postfix, at least, is very old and didn't have the benefit of what we've learned about best-practice configuration like having a single conf.d directory so things sit alongside each other nicely. The documentation is also very terse but Wietse did tell me that he's busy enough maintaining the software and porting all the docs is a mammoth task. I did do something myself converting the text files to markdown and using a SSG but there is too much stuff that I didn't understand well enough to re-word. There are also a tonne of compatability which has been added over the years which is no longer needed but still makes the docs feel a little overwhelming.
My top-tip though, is to invest time in the Ansible setup script to avoid the scourge of following out-of-date blog posts line by line. We got to the point where we can provision a new server (manually) run the script and have everything setup ready to go, which is super helpful.
The other important part is to monitor what is happening. Blacklisting can come upon you quickly and not always for an obvious reason but if you don't notice, it might be harder to resolve later and you might have lost a tonne of email. Instead, I semi-regularly spot an error in a log file and have to email someone or login to some ancient web page to request removal from a blocklist. We are generally always successful but it is also worrying that we have no direct control over it.
Honestly, the original setup of everything is a big lift if you've never done it but once you have ldap+krb5/PKI for auth{n,z}, DNS working properly, and a working POSIX setup (what used to be sysadmin 101) it basically runs itself in perpetuity. The underlying tech is old, feature-complete, and rock solid. Not to mention you're in full control of the entire stack. I honestly haven't had an issue after the first year and I use email HEAVILY.
Though to be fair, I do have a bunch of ldap entries and routing rules in place so the power users can choose sieve+dovecot and all the plebs can use the Gmail webui.
idk. what the fuzz is all about:
yes, running a mail-server is a complex task - but nothing to write home about!!
personally i prefer exim over postfix - because of its configuration: exim has a really flexible configuration-language which enabled me for example to operate qmail-ldap and exim-ldap from the same LDAP-tree - for migration purposes a few years ago -, using the qmail-ldap.schema vs postfix which insists heavily on the posix.schema ...
additionally i prefer courier-imap over dovecot, for no apparent reason, just because i use it since ... ever. and i don't care about peoples performance who keep something like 10 k emails in their inbox/IMAP-folder ... yes, for this use-case dovecot is significantly faster / because of caching ;))
just my 0.02€
ps. i used qmail-ldap and qmail for over 20 years, and it was a really great piece of software - kudos to djb, but even brilliant code ages over the decades - for example, i didn't want to implement IPv6 support for outgoing mails myself / the available patch didn't cooperate well with the ldap-patch itself if i remember this correctly etc.
pps. i would recommend: use an LDAP-directory for email ... openldap is an extremely lightweight and easy to operate piece of software. and its easy to replicate the db around - if you have multiple nodes in your mail-system - etc. ;)
If and when the Mailcow project goes away, it’s all still standard components, so moving to i.e. vanilla postfix and dovecot should be straightforward.
please use maininabox or something similar. it is also selfhosted and foss but you update it in one click and has 6 monthly 10 minutes updation.
i have been using miab for over 3 years now. extremely satisfied with the result.
Self-Hosting email isn't that hard as it used to be (as long as you don't have to deliver to MS)
i gave up and started to use a secondary domain hosted with a paid email provider just so i can reach people that my selfhosted server can't reach.
aside from that my selfhosted server is maintenance free.
But if you don't want to struggle with delivery to MS, you can always give up and use a (paid) relay. Although this feels like losing tbh.
I don't think it'll work on your ZX80, though ;)