After self-hosting email for 23 years I have thrown in the towel (2022)
cfenollosa.com
cfenollosa.com
But eventually it got stable enough and did its job well enough that I stopped messing with it to learn things. And then it became a job instead of a hobby: friends texting me because my ISP was having an outage and they couldn't read their email, rolling out critical security patches ASAP, coping with the occasional DOS attack, and so on.
At some point, I realized that I'd started to resent running it. It was eating little bits of my time for no additional benefit to me.
I also, as one does with age, started thinking about what happens if I die or become incapacitated: my friends and family all lose their email service the next time there's an outage that requires manual recovery.
So I moved it to a commercial email service and have zero regrets about the move. I make sure I've always prepaid for at least 6 months of service so that if something happens to both me and my wife, my friends and family will at least have half a year to make other arrangements.
Search on HN for either, there's lots of discussion and reviews from accounts way older than mine.
Have a look at their pro/cons (in general really good): https://www.migadu.com/procon/#in-switzerland-but-servers-ar...
I will never understand people who see it as an all or nothing proposition. You can perfectly self-host while using an outgoing SMTP provider. This would solve all the problems the OP is having while still preserving the main benefit of self-hosting, meaning: managing your own MX records, you own IMAP server, so that government authorities don't have a single organization they can target to get access to your emails. Heck, if you are paranoid you can even have multiple outgoing SMTP providers and round-robin between them.
Edit: the author does mention not wanting to pay for an outgoing SMTP provider, but there are many free option: for example Gmail lets you use their SMTP server as long as you authenticate with a (free) Google account. That's possible with Yahoo, or many other providers.
That's mostly selfhosting, not fully selfhosting. I full selfhost my email, I 100% control as much of it as is possible to do. When I send an email to someone, it goes from my client to my server and out to the recipient's server.
I genuinely think some people are not aware you can have free, reliable, outgoing SMTP providers.
And you can have an additional route just for some recipients, that doesn't make it less (or more) selfhosting.
It seems to me that you are proposing exactly this: routing all emails through one of these companies. How can this be a solution to "self-hosting"?
Sure, it's a never ending battle with the behemoths but there are options and the most beautiful thing is what most of built-in in the SMTP itself.
It's just not worth the overhead in people and hardware to try to be surgical in your approach - you don't win anything. It's the closest thing to working at the DMV for tech.
Notes, cc:Mail, Groupwise, Exchange, Sendmail, Exim, qmail, postfix, etc... I have no interest in running my own email ever again.
The article is not about blackholing spam sources. It's about blackholing mail from sources that have never sent a single spam message since the dawn of the internet.
By definition, every new source has never sent spam -- but it's reasonable to assume that an unknown source is likely spam, however unfortunate that may be.
Uh, yeah, you can, I did it again recently. I've done it at Oracle Cloud, AWS Lightsail, Linode, GoDaddy, and several ISP's before that. I could start right this minute with Digital Ocean, Vultr, etc. pick one and be up and running, delivering email to Gmail, Microsoft, etc. in a day.
Each time one of these articles is posted, I feel I must be some kind of email savant.
- Setup blacklist monitoring (e.g. HetrixTools/MXToolbox)
- Check if you can email Gmail
- Check if you can email Hotmail
- Check if you can email Office365
- Check on Microsoft SNDS that you are not blocked: https://sendersupport.olc.protection.outlook.com/snds/
Gmail, Hotmail and Office365 are the largest email providers and also the most strict ones.
I have accounts on all of these providers so testing deliverability is trivial. You could argue that testing deliverability with one account is unreliable but in my experience it simply is not. Usually you send a couple emails across a week and if they all go through you are good.
If you are paranoid, you can ask if any of your friends have Office365/Hotmail and email them. They probably have company or university accounts on there.
My server has only been blocked once after all this, and ironically it was by another company that self-hosts their email...
No, it's not trivial, and no, it doesn't work like that. It's fairly easy to get a test email to your own gmail/hotmail/office365 account delivered. It doesn't mean your other emails get delivered.
For example, when I needed to send out a link for my wedding photographs to my wedding guests, I did exactly what you suggested: I sent a few test emails to Gmail and Hotmail test accounts that I had set up, confirmed that they delivered, and then proceeded to send the actual email which had a link and like 50 people in the BCC field. Guess what happened: all the Gmail accounts placed my email in the spam folder. So then I had to later send another email, using a different email provider, asking people to check their spam boxes for the link (this time not including the link in the body of the email with the hope that it would increase the chances of delivery).
Note that this particular anecdote is not of self hosting email, it was using Migadu (I was initially not trusting that they can deliver email properly, so I ran those tests like you suggested, concluded that they seem to be delivering email, and then my actual real email was not delivered).
Actually I initially got placed in spam too in Gmail/Outlook but I always replied to my own emails and marked them as "Not Spam" and the problem just kinda disappeared?
When I initially setup my server I was paranoid (as you should be) and so I had the following protocol:
- Send all urgent email with Gmail
- Send non-urgent email with my own server. If I don't get a reply within 24h, send again with Gmail, explaining my email troubles. Usually the other party would happily confirm whether or not my email landed in their spam without me even asking.
It was never a cold-turkey migration to my self-hosted setup. It was more of a slow process over a couple months after I gained more and more trust in my server.
Did you continue to use those same Gmail/Outlook accounts for your deliverability testing? Because that would explain why you're not encountering any deliverability issues in your testing.
Try to create fresh Gmail and Outlook accounts and see if you can actually deliver to them.
I had a couple Hotmail/Microsoft Live/Outlook accounts too.
But deliverability for me was always quite consistent. If it went to spam on one account, it would go to spam on all of them.
At this point I no longer do deliverability testing because I simply don't see the need to. No one has ever complained about my emails going to spam so the thought has never really crossed my mind.
I've also had a lot of trouble creating Outlook accounts for some reason. New accounts seem to get suspended real quick. My previous Outlook accounts have mostly all been suspended too. Not sure what's going on there...
EDIT: From the output of `dig` I can see my landlord uses Outlook. I just emailed him a couple days ago and he replied so I guess all is well?
You mentioned that your emails on Gmail and Outlook initially went to spam, but that you clicked "not spam" on those emails. This action should prevent your emails being flagged as spam for that particular receiver who clicked not spam. Obviously it can't work such that one person clicks "not spam" on an email, and then everything from that sender is whitelisted for everybody. Because then the spammers could just register a single email account, click "not spam" once, and then spam everybody. So my point is, if you test deliverability on email accounts where you have already whitelisted the sender, of course it's going to look like email is delivered.
> I just emailed him a couple days ago and he replied so I guess all is well?
Presumably you and your landlord have already emailed each other in the past, so any anti spam system should allow emails between those addresses to reach their destinations. It doesn't really prove anything.
Obviously I'm aware that me clicking "Not Spam" would not instantly whitelist me in Microsoft's systems, but the hope is that it would influence the spam algorithm somewhat. It doesn't even whitelist me on that account because even after clicking "Not Spam", my emails still went to spam for some time.
> Presumably you and your landlord have already emailed each other in the past, so any anti spam system should allow emails between those addresses to reach their destinations.
Maybe, but my entire university runs on Office365 and I never had any trouble sending to friends and professors there (who have never sent me emails previously).
I'm not really sure what you are trying to say here? Should I have some sort of commercial deliverability monitoring system setup? Do you have insights into what "real email providers" use?
Or, if I were to put a larger point behind it, I would say that getting good deliverability is hard and it doesn't make sense to try it unless you are delivering email on behalf of a large amount of people (rather than merely yourself).
We read from different cookbooks and my reading comprehension is high. My sent email stats shows over 20K successfully sent emails just for my personal account, since 1999, when I started tracking it.
Ok, since you claim to have measurements, let's hear what they are:
- What is the total number of emails sent?
- What is the number of emails bounced?
- What is the number of emails that landed in inbox (as opposed to spam)?
- What is the number of emails blackholed? (recipient confirms delivery, but email doesn't show anywhere, not even in spam folder)
I feel like for every pro host-your-own-mail on HN there is one that contradicts it.
After many hours of this over many months, I finally looked for a comparison of the actual taste of coffee from cheap grinders versus 10-20x more expensive burr grinders. (As opposed to comparisons of grinder technologies, which are everywhere.)
There are almost no published side-by-side taste comparisons; when I did find one, the cheap grinders had won!
I can imagine, because why would taste matter for a coffee /s ?
Setting up a server is trivial, yes. Keeping it going is a never-ending treadmill of not really technical problems.
Not magic, but when I managed outgoing Postfix servers for a few companies I had to set rate limits for yahoo.com an a couple other domains to reduce concurrency or they would block one of the SNAT's for a while. It probably sounds tedious but it really wasn't. There were not many MX that were as strict as Yahoo. I never ran into issues with Gmail but I think they cut some slack for corporate IP addresses and domain names.
For my own personal email servers I never had issues because I never sent at a rate that anyone cared about. The closest I got to that was running a forum that would email when threads would get updated and people subscribed to them but my solution there was to suggest to the people on the forum not to do that.
I can't see the future, if the big email providers who likely have some of their trolls posting in the comments ever decide to start choking out us personal email server runners, then it'd be game over. If things remain for the next 20 years assuming I live that long then deliverability would be 100% for the next 20 years.
> Do you know some magic incantation for staying off the naughty lists?
I don't spam, that and don't make a finger fumble edit like I did the one time in over 20 years and didn't check to make sure it was working correctly first.
Also, I use http://www.mxtoolbox.com/ to keep an eye on my server.
Put up an ASK HN and see what happens.
The cases where my email got blocked, I always received an email back from the box, so I think there are no cases where my email just disappeared into the dark. Reasons for blocking where: - Some strange spamlist blocked my IP. I was able to resolve this by filling out a i-am-not-a-spammer-form on the spamlist provider - Some email server required from me that I add my personal address to my website, so that they know who maintains the server
Never have I send an email but it was not delivered without notice
I have never been on a spam list. My IPs have never been on a blacklist. And yet, my mail was often delivered to spam folder (as opposed to inbox) on Outlook and Gmail.
> The cases where my email got blocked, I always received an email back from the box, so I think there are no cases where my email just disappeared into the dark.
In the vast majority of the cases where your email is not delivered to the inbox of the recipient, you will not get any notification. In these cases your mail will usually be silently placed into the spam folder, and sometimes it will be blackholed completely (not even landing in the spam folder).
It's times like this that I wish HN had better moderation, you are insultingly rude to the parent and to one of my comments.
While setting up your own mail server is fun, rewarding, and pretty easy, there are serious problems that anyone considers it needs to be made aware of. Over any serious period of time, the odds that not all your mail is getting delivered approaches and equals 100%. It's a sad reality but a true one unfortunately. If you okay with the fact that some percentage of your email will never make it to your destination (and in many cases you won't know that's happening) then it might be for you. Most people are not okay with that for obvious reasons.
No, they are hand waving and being rude and insulting while delivering misinformation.
The GOP sued Google for blocking their campaign e-mails and lost.
These cries of protest were just more ridiculous outrage fuel, from a party that loves portraying themselves as some kind of victim.
For incoming mail, I'm hosting it all virtually on a home server. I set my MX record to my home IP address, point a CNAME of "mail.domain.com" to it. I do letsencrypt on a cert for each domain. This has been working fine for the 3 domains I've tested.
For outgoing mail, because I send so few from these domains, I'm using Amazon SES. I looked at setting up DKM to avoid exactly what the author describes about blackholing. Setting it up for multiple domains on one IP made my head spin. I might send 1-2 emails per month on these domains, so at 10 cents per email, that's quite affordable.
> it is very hard to remember how you set things up, and almost have to start over.
Same issue. I have an absolutely terrible memory. That's why I do all the setup and configuration through Ansible.
If you do have 10 domains that would be quite a lot still.
The biggest error in blackholing, that I never hear mentioned, is that no mail server sending several hundred email per day is a spam server.
If traffic is below a threshold level, several hundred or thousand emails per day, there is no way that server is part of a commercial spam distribution network.
Small system operators should not be banned or blackholed in the same way as servers producing millions of email messages per day...
I like hearing about what impacts contrarian nerds too, but that discussion usually happens so we may as well get that out of the way ;)
Google is another beast altogether.
* First, I disable IPv6 in my MTA. Yeah, I know, other people managed to do it but it never worked for me and it's not worth the hassle.
* The next step is SPF. I set it up correctly and verify it actually works using several online tools, not just one.
And, lo and behold, this was enough to get first emails to Gmail spambox (Actually getting it to the inbox needs time and never works reliably). First for one account, then to others.
So yeah, the oligopoly is real, but as of now, December 2023, you can still set up your email server and send messages to Outlook and Gmail recipients (Yahoo too, but they are easiest to deal with than the preceding two in my experience.) I encourage everyone interested to experiment and at least try if they can set it up for personal purposes, because if everybody just gives up it will be the end of email.
After running the service for 20+ years, I was tired of it. Tired of the continual war against spam. Tired of annoying corner cases that I had to fix on my end because the giant corp sending demonstrably malformed messages to me wasn't going to correct their system. Tired of this week's Mysterious New RHBL breaking connectivity. Tired of not being able to effectively filter email the same way giants do, because I only see a tiny sliver of messages and can't identify bulk senders.
I appreciate and respect anyone still hosting their own email. I did it for many, many years, and feel their pain. It wore me out.
(The nail in the coffin was Apple offering free email hosting for iCloud+ users. It checked the right boxes for my family's needs, at an incremental price of $0.00 over what I was already paying them.)
After self-hosting my email for twenty-three years I have thrown in the towel https://news.ycombinator.com/item?id=32715437 (733 comments)
This is almost certainly false. If this is not false, please do show how you measured your deliverability (across all those years) and how those measurements show "almost no" issues.
I can tell because I also run a mailman mailing list (~500 people) on that same server and domain, I measure initiall discoverability by how many bounces I get (almost always none) and also that the high percentage of read receipts I get, and the fact that engagement with the list is pretty even across domains (IE, people with gmail, outlook, yahoo, and aol emails all frequently read and engage with the list).
> I'm very surprised if the % for gmail and outlook truly is not lower than the % for yahoo and aol, as you indicated.
the distribution of domains is far from even, but I can tell by checking my inbox from today that one of our most frequent posters with an aol domain sent a news article around and it started a thread with multiple gmail users chatting with the original poster about it.
Thats not hard numbers, but everyone on the list has agreed that the list works better on the self hosted server then it did on google groups.
This might be indicative of deliverability issues affecting some domains only.
The fact that some gmail users can see emails doesn't mean 100% of gmail users are delivered your email.
I mean maybe? I remember when we switched from google groups that I checked and the % of opens and that it seemed fairly evenly distributed given that the percentage of gmail users far outweighs the percentage of yahoo/aol users, and to be honest I've not bothered to check much more beyond that point. But the core group of crotchety old folks who frequently send round interesting local stories are a decent mix. The last couple of days I see 4 different gmail accounts, one work domain (that I assume is outlook but might be some other provider), the aforementioned aol account, and the burner riseup account that I use when I am feeling crotchety.
> The fact that some gmail users can see emails doesn't mean 100% of gmail users are delivered your email.
Sure, its possible but I've not seen any evidence of it after we made the transition. I've seen email sent from gmail accounts to gmail accounts not get delivered, expecting 100% is never going to be realistic even with the largest mail providers.
Unless and until I start getting in person complaints (which, trust me, I would if there was a problem) or see people mention emails going into spam, I'm going to keep on keeping on.
Setting all up is complex, but there are project that give all the tools in a Docker container. Since I wanted to learn it I did all manually, and did set up postifx, dovecot, opendkim, postfixadmin and even squirrelmail as a "retro" web interface, with a postgresql database to keep the mailboxes and the accounts. It was an interesting experience, plus now I have full control of the mail system, e.g. I can decide to route the mail in whatever directory just by changing the SQL query that decides in which mailbox a message should be delivered.
I personally did do all of that, and my emails didn't get delivered. I wrote about my experience in detail here: https://www.attejuvonen.fi/dont-send-email-from-your-own-ser...
With DMARC you can specify that the recipient sends you a periodc report of the delivery status, this is useful to debug eventual problems. For example GMail and Outlook send this out, and gives you information about the fact that your DMARC is working correctly and SPF and DKIM checks are passed.
Anyway when you manage to get it working, it did work for around 4 years without any issues. I've even reinstalled the mailserver to another machine (keeping the same IP) and I did not have any issue in the migration.
- Better IMAP support: ProtonMail doesn't support IMAP without the ProtonMail bridge. The ProtonMail bridge was very buggy for me, and I have to self-host it anyways. With self-hosted email, I can just use IMAP normally (yay).
- Better email search: With ProtonMail you can search for keywords in the message body but it doesn't work if you have lots of mail. With self-hosting you can search however you like across as much mail as you like.
- Easier backups: On my self-hosted setup I can configure automated periodic backups. I don't think that's possible on ProtonMail.
- Cost: ProtonMail is really expensive :(
The main reason I switched was the IMAP part. I just got tired of the bridge bugs.
Also if you're already paying for IP service, there is no technical reason you should have to pay an email provider. Email standards and software are all open and free.
Yes, you can log the crap out of "$Server.$OtherProvider.$TLD accepted $MessageID for delivery" - but that doesn't stop $OtherProvider from failing to deliver it to the recipient's IN box.
Custom filtering/sorting.
Easy burner addresses.
Custom reader tracking for marketing.
Custom reader db response pattern mailouts/campaigns.
Some of these are Some Assembly Required, but they're hard/impossible/expensive to do with 3rd party mailers.
I can create as many alias, mailboxes I prefer.
Overall the freedom is soothing.
I to have never had emails denied but then again I'm using colocation based hosting, so my IP block is one that's never been used before.
For this reason I can have a different address for each service that I register with, for example facebook@domain, and thus easily block services that start spamming out, and even know if a service leaks my address without authorization.
Yes, I know that GMail has a similar feature (by using address+whatever@gmail.com) but some sites blocks it, and still you don't have that ability to have a completely different address, that if one day you want can become its own mailbox.
Another reason is that commercial email services started blocking the simple SMTP/IMAP authentication with username/password, making it difficult, if not impossible, to configure a server them to send email trough GMail or Outlook, or a service to subscribe trough IMAP subscription to the mailbox to trigger an action when a mail arrives. Proton even decided not to support plain SMTP/IMAP, and you need a bridge to use it!
All my systems, being them servers or services that run on them, are configured to send the email trough my server, and the configuration is super easy. So that, for example, I'm notified if a cron job fails (basically the mail to the local Linux users gets routed to my server) or I have my Home Assistant installation notify me trough email.
Next I manage different domains, and I can have all that domains in a single central mail server. That allows to manage the mail of different systems in a central place, and with a single account. As far as I know this is not possible with commercial services: what you can do is of course to have one system forward the mail to another, but to send the mail as another domain with the same mailbox? With my mailserver I can, I just configure multipe FROM addresses in Thunderbird.
Finally on my server I don't have a limit on the number of accounts/mailbox, so I create an account/mailbox for whatever reason I want in two seconds, for a fixed montly cost (of around 10 euros, since I use the smallest AWS EC2 that they have available)
As you see, there are plenty of reasons to have your own mail service.
And, and let's conclude with: I've lost more mails thanks to antispam in commercial email software such as Outlook or GMail, that on my own server. I don't care about spam, I (if needed) know how to filter it out client-side. I don't want systems to refuse or even hide messages to me, something GMail and Outlook does.
I do not intend to ever give up on this.
I had problems in the past, mainly with Microsoft, but they were not insurmountable.
The worst problems with delivery have been a few years ago, I had less problems recently.
I'm interested to know which one. I'm guessing Gmail.
I don't believe you. Sorry. Unless you never actually use your email address for anything, to have zero spam is impossible.
Unless you mean that you have a 100% success rate in filtering that spam out, and in which case, I'd still not believe you.
the portion of the article that you are quoting seems to mean that his domain/subnets have a perfect record of not sending any spam.
in the weighting system in my head, people who don't meticulously keep track of differences like this in a writeup make me worry that they weren't keeping track when working on the problem itself either.
Preventing spam from being sent from your IP is achievable.