Small mail server best current practices | Hacker News Reader

Small mail server best current practices

bridge.grumpy-troll.org

401 pointscase168 comments

Running an email server became a real pain about 4 years ago. Until then, we could run our own exchange server, control everything and it was awesome. Then came the email security products from the large companies.

It became a self fulfilling prophecy. Our email would trigger a rule in big co's third party security app, which would then report to a centralized rule breaker clearing house automatically. Clients couldn't receive our emails, we would dive in, submit appeal to clearing house, get clear and a week later do it all again.

Everything was configured properly on our end. We passed all the online validation engines. IPs were our own personally owned block and pristine.

It became too much.

Switched to office365 and all problems magically disappeared. Sent emails to same big cos with third party email security and haven't had a single issue.

Is the implication that this is an intentional practice to drive out small players?

I don't think it is an attempt to drive out small players, it's a natural consequnce of domains being almost free to stand up. They just don't provide any level of assurance that the owner isn't a spammer. So the blackhole systems treat them as "probably spammer" and fire up a block at the first whiff of malfeasance.

It's harder to sign up for a Gmail account than it is to register a domain and get some hosting. And Google has their own protections against sending large amounts of mail built-in. The system is as it is now because spammers will abuse it otherwise.

gmail is effectively block by default these days, unless Google has some reason to not block your mail, it will be spamboxed.

True. It's a mistake to do a job search using gmail. Interview invitations and job offers are going to go straight to spam.

Sending email to Google is always a crap shoot and I still find to this day that sometimes when I send email contains logs to my own account using my own account credentials it'll end up being classified as spam.

Right now, Google is refusing to deliver email from logwatch - giving me the "Message rejected. See https://support.google.com/mail/answer/69585 for more information." so those are going to /dev/null with no recourse to fix it because Google literally doesn't like the content of the email.

The fact that I can't hit a button and tell Google that yes indeed I'm not trying to spam myself is ridiculous.

That was the case for me few years ago. But I decided to set up mailserver once more recently and Gmail to my surprise accepted my mails. I configured everything mentioned in article.

That said, my outcoming mail traffic is almost non-existent outside of few test emails, LoL.

I have an old, but frequently used email address that got force-moved from a pass through relay to gmail.

After setting gmail to forward everything, I noticed it was sending 33% of my legitimate email to spam.

They don’t let you disable the spam filter, but you can set up an escoteric filter that prevents it from actually putting stuff in spam.

I honestly don’t understand how people cope with gmail. Fastmail is cheap and orders of magnitude better.

I created my gmail account in 2004 I think. Things were different back then. Now there's just 16 years of momentum preventing me from switching.

I like the idea of hosting my own email but even if I get around to doing that I will still need a backup. Fastmail is probably where I will go.

With Outlook you can't unfortunately do that. Create rules, which would un-spam mail. Because filters aren't applied to spam emails at all.

I disagree.

The big email players are accepting mail from universities and large companies with misconfigured email servers. If you use the same rules that Google and Microsoft use to refuse emails, you will be bouncing emails from valid domains. If you don't apply them, you end up accepting spam. The big players are able to deal with the spam using specialised teams and I suspect advanced algos including machine learning.

IF the big players applied their famous rules to everybody equally, everybody, including universities, big companies etc... Would quickly configure thier mail servers properly, which WOULD reduce spam, possibly eliminate it.

Being able to keep inboxes fairly free of spam in a world full of spammers is what distinguis big email providers and enable them to sell their services.

> The big players are able to deal with the spam using specialised teams and I suspect advanced algos including machine learning.

I remember when I imagined big companies might use their considerable resources to put together something like that. Then I found out it's just a handful of random engineers cobbling together crap and hiding it behind a fancy domain name.

It is probable that Office365's entire spam filtering system is a very large and ugly spreadsheet that gets passed around between a few teams using a ticketing system.

I used to work on the O365 antispam team, and I'm not sure how much I'm allowed to say in public, but I'm pretty sure I can publicly say "a very large and ugly spreadsheet that gets passed around between a few teams using a ticketing system" is not accurate.

Relevant wikipedia page: https://en.wikipedia.org/wiki/Exchange_Online_Protection

Originally, the product started out as an acquisition, and there's still some legacy code that references the old Forefront name. There's about 50 people on the team.

DKIM/DMARC/ARC is worked on by two engineers.

No, I think that there are just too many spammers, so big players have no choice except to make it hard to own your mail server until your server gains positive reputation.

I don't think there are many spammers compared to, say, 2005.

In 2005 you would get 100 spams to an unfiltered, widely published address, these days it is more like 2-5.

The continued lockdowns and protocol changes of gmail and yahoo are a sign of being overstaffed, "feature" oriented and yes, the attempt to shut down competition and the free exchange of mail.

I do not think so, I think the spammers are still out there, armed with cheap cloud email providers, but the "block by default" is putting a nice dent on them.

You're lucky if you only get 2 spams a day. My parents are getting more like 20-50 a day on the email provided by a national ISP.

It's getting impossible to read emails, too many. If it were not for them being used to their email address and me being lazy, I would move them off to gmail.

This is not true. They're not even making an effort and are quite probably actively malicious. There is nothing you can do to build up your reputation if you're a small email server. There are no introspection tools to give you a hint on what's wrong and no one to contact. Besides, what are those fancy ML antispam algorithms for if the only cure for spam is "reputation"? It's clearly an undefined term meant to be exclusionary.

A person from Gmail even posted on HN a while ago, stating they'll look into this and do better. That was about a year ago and the situation is exactly the same or worse.

I don't think it's done intentionally, but I also don't think the big players go out of their way not to harm the small ones. All the big email players would love to take your money to "outsource" your email.

I really think so. They are forcing everybody into their services.

It is, at least if you ask me.

People here are speaking about how gmail will "effectively spambox your mails by default". That's not been my experience (from setting up multiple small customers). Anyway, at least I've never heard of gmail just eating your e-mails. They either get rejected or accepted and put somewhere (maybe the spam folder, but at least somewhere).

Office365 / hosted Exchange / Outlook Protection or whatever is it called these days... they should be routed to /dev/null by everyone.

They just won't track your reputation unless you send them more than 100 mails per day. Does this sound bad? There's more: if they don't have reputation info from you (because they refuse to track it due to the low volume), your mails will go to spam inboxes even when their filters indicate that the message is not spam.

And there's more! Don't dare to ever get a bad reputation (i.e.: a user managing to get hacked and their account used to send a couple hundred spam e-mails before triggering your countermeasures). If this happens you are 100% fucked. Now they will DROP your e-mails. Hear, hear: their servers will accept your mails and just DELETE them. No spam folder, nothing.

You will try every possible thing: set up everything for their feedback loop, sign up in their "Smart Network Data Services" to track your reputation (it will be empty except for that day)... and finally contacting them at their sender support.

Do you want to know what they will reply? That you should be patient and let your reputation build up over time. What a joke! How on earth can your reputation improve when users cannot mark your mails as "not spam" because they (outlook, not the users) are simply DELETING them without a trace?

Oh, there's a way out of all this though: obtain a "Return Path Certification" [1]. That is, pay them an absurd amount of money and your mails are guaranteed to get to the users' inbox unless you are clearly spamming (all of the above assumed you are NOT).

Up to this final point you could think they just do their best and all that I've explained is collateral damage. That last "pay and we let you off the hook" is what clearly signals to me that this is an elaborate scheme to get small players to either pay them anyway or just give up and use a big-provider service.

[1] https://sendersupport.olc.protection.outlook.com/pm/services...

> a user managing to get hacked and their account used to send a couple hundred spam e-mails

For this reason one of the necessities of survival as an email sender is to classify your own outbound messages, to see if recipients might think they are spammy. If they just look spammy but you still want to send them, you need sacrificial IP addresses on separate netblocks that are dedicated to spam, so the reputation of your main IPs isn't polluted.

If there's any possibility that any of your users could get their account hijacked, or there could be malware on any device permitted to send messages, you need outbound classification.

> People here are speaking about how gmail will "effectively spambox your mails by default". That's not been my experience

It is mine. I used to have to take a proactive approach to email - send an email, and if I don't hear back in a timely manner, I check their MX records. If it's Google, I'd contact them out of band - yep... spamholed.

These days, if it's important I'll contact out of band. If not, meh... I'm not going to bother just because of Google Knows Best. I'm over pandering to Google.

Has anyone tried pursuing a defamation lawsuit over systemic false spam labeling?

Edit: or, alternatively, tried fighting this using anti-racketeering laws? The payment aspect sounds like a protection racket scheme to me...

Antitrust, too. Making a market impossible to enter for new entrants is a blatant abuse of market power, and EU antitrust law would probably care about it.

> Oh, there's a way out of all this though: obtain a "Return Path Certification" [1]. That is, pay them an absurd amount of money and your mails are guaranteed to get to the users' inbox unless you are clearly spamming (all of the above assumed you are NOT).

Could you name the cost?

They are a "contact us" kind of organization, but if you look hard enough you'll find prices from a while back [1].

TLDR: Minimum tier was $400 signup + $1,375/year for up to 100,000 mails/year.

[1] https://returnpath.com/wp-content/uploads/2015/06/Return-Pat...

Bonus: They straight up declined my business because it was too marketing-y for them.

What was too marketing for them? A 100% opt-in list that reminded customers when a product was in stock. Fully made in-house, with captcha, privacy policies, opt-out, and with the user typing their email and name, the complete works for a one-time reminder email.

Nope far too marketing related for them.

Your experience tracks with mine as well.

I run VoIP PBXes and they occasionally need to send an email, usually for voicemail to email but occasionally for other alerts.

By default they just send straight to the internet, sending from pbx@pbx.domain.name, forward and reverse DNS mapping back to their own addresses, etc. but no other setup.

This works perfectly fine with Gmail and Gsuite accounts, but Office 365 occasionally decides it hates one of our servers. Even if the client has the sending address whitelisted, it still just gets hard blocked for no apparent reason.

Gmail, I can fire up a telnet session right now and send myself an email from my home IP address just typing raw SMTP commands in to a console. It's going to work, and unless I'm spoofing a real email address it's not even going to end up in spam.

Yep, you can do everything outlined in the article and have an IP that has not send spam in 10+ years and gmail will still spambox every single one of your mails.

However if a domain signs up for Google Apps, magically you get whitelisted, even if you are handling your own email, funny thing that.

I guess Google's customers are just more trustworthy...

> I guess Google's customers are just more trustworthy...

I wish. I've been having issues with being sent to spam when replying to a colleague on the same domain which is on Google. The email never left their servers, everybody was authenticated, and the emails were within an hour of each other, so it's pretty obvious that it's a reply. Still: off to spam it went.

We now have a filter rule to always consider our own domain as not spam, because it's just too unreliable otherwise.

This is a really annoying Gmail+Gsuite bug that has affected us as well. Surprising that it seems to happen this often.

Note that your filter might be breaking an important security feature:

If you get an incoming email “from” yourself and it is not marked as spam, it will be put into your sent label.

At least one person claims to have been incorrectly fired because of this. Their employer found incriminating emails in the person’s sent box, and considered it a closed case.

That's interesting, do you have a link for that case? I don't see why it would get put in the Sent folder, but if it does, that might certainly lead to confusion.

Counterpoint. I run my own mail server, and have done for 10+ years on the same IP.

In that time I've had a single incident (earlier this year) where spammers got it and sent 20k spam mails in less that 24h mails. Luckily I noticed this and stopped it pretty quickly.

I had mails to some destinations bounce for around a week after that.

Beyond that incident, it's very rare to have trouble delivering mail - I think maybe once in the past 5 years there was a company who I had trouble with.

Having said all that, the spam incident this year highlighted how fragile hosting your own mail server can be if things go wrong, and I felt really bad about the spam mails sent from my server - continuing to run it almost feels like a liability, and when time permits I plan to move our mailboxes to O365

Unless you have substantial volume to gmail (hundreds of mails a day), I can virtually guarantee gmail is spam boxing your mails, try mailing an account you have never sent mail before for instance.

They don't refuse delivery, but that is entirely meaningless when nobody ever checks the Spam box anyways (which is also hidden by default)

Our email volume is pretty low - average is probably 5-10 a day.

Hmm, I just tried what you suggested, with 2 different accounts never sent to, and the emails got through fine.

We use SPF and DKIM, but otherwise it's a fairly standard Postfix/Dovecot install (albeit a very old one).

Would you mind sharing where you host your email server?

I have a pet theory that big email platforms care deeply about the IP "neighborhood" the sending email server lives in. So a small email server set up perfectly in Digital Ocean IP space would struggle, while an email server set up OK in (for example) CenturyLink Enterprise IP space would get through.

It's with a European (Austria-based?) VPS provider, Edis (edis.at), and my server is hosted in a UK DC.

I know I've said this before here but.. it goes the other way too.

People make a big deal of SPF/DKIM etc. I think you should, for the sake of best practice. But you can set yourself up with gsuite or Office 365 and pay no attention to these things, and you can fully expect mail to get through to everyone. You can even setup a broken SPF record that disallows office 365 email and for the most part, everyone will still get your email.

That angers me so much. I had same IP address and domain for 15 years. Never a spam was sent from it, since the same address was used for NAT in home network I also blocked outgoing SMTP (forcing to use my SMTP instead) for cases that a PC got a virus and starts sending spam.

Things worked fine, but I think 2 years ago I realized that my e-mail started going to a spam folder. Fortunately adding SPF/DKIM appeared to fix it, but it feels like it is getting harder and harder now to have own mail server.

I would probably gave up, but it's infuriating to me that from courts rulings if your mail is handled by 3rd party, the 4th amendment no longer applies.

Not that I'm expecting it or anything like it, but running my own mail server means I'm the one who gets the subpoena for anything that might be on it. Google/Microsoft giving Third-Party Doctrine-driven open or nearly-open access to every email on their servers for any purpose is not my idea of a good business decision.

At my last job at an email security provider, this was a huge pain in the ass. We would provision new mail servers, and because the new servers' IP reputation to Office365, Google, etc. was poor initially, someone (until it became a shell script) had to spend the better part of a week taking the new server out of rotation, putting it back in, and so forth, so not too many of our client's emails got stuck in the new server's reputation blackhole.

I was there for 10 years, and this was only a problem in the final ~2 years of my employment, which roughly lines up with your 4 years remark.

Yup. As more and more domains centralize email in the handful of mega-corp hosted solutions the hosts have less and less reason to care about accepting mail from outside the walled gardens.

It's not about setting up perfect signed email and all the new techs. It's not about having the same clean IP for a decade. It's just the same old network effect combined with profit motive. The obvious change in the last few years (I've run a mailserver for 9) is managerial not technical.

For this reason I keep running my email server[1] even if it take some time and increase my attack surface.

In my humble opinion, leaving email to mega-corp alone can be a freedom risk, even if a small one.

[1] https://gioorgi.com/2020/mail-server-checks/

A middle-ground that can be worth considering is mailgun.org or something similar - you can host the rest of the stack yourself and just use them for the last hop of outbound SMTP from your own MTA.

You don't have to throw out the baby with the bathwater (:

that is really not a middle ground, you are handing full control of your email over to a third party.

That's not really true. You're using a third party to send out messages, and if you don't like it, you can switch pretty painlessly. You could even use you're own local MTA at the same time, if you really wanted to.

The IMAP/POP3 server is yours, and all the email is stored on something you control.

This is what I do but I use Amazon SES to send out mails from my selfhosted email infra. All emails pass through and I have the satisfaction of knowing I can just switch to another provider if SES shitcans me.

Running an email server has been a pain forEVER. Running one's own personal mail server is a terrible idea unless you are a professional sysadmin with time to kill.

I disagree. The last couple places I've worked at have all had their mail hosted internally, meaning on a dedicated IP from one of the big ISP's on a business plan.

No real issues. Sure, you make it on a spam blacklist once in a while, but at least you know when it's going to be fixed, instead of dealing with zero response (or denial) from a big hosted mail provider.

Besides, not all of "our email" comes out of our office, our CRM and ERP products send mail on our behalf, and they have problems delivering mail more than we do.

There really needs to be

a) a docker container (or other standardized, easy to deploy technology) with a "mail server in a box implementing best practices"

b) a test for the stuff that Docker can't cover like DNS config, ideally with scripts to set it up on the most popular providers and clear copy&paste instructions for the rest.

People shouldn't need to fully understand DKIM, SPF, etc. - they should just be able to click a button, copy&paste generated DNS records, and be up and running.

Otherwise, commercial providers are just the only option that makes operational sense for most cases, _even if you know how to do it right_, because doing everything right manually step by step is still to much work to be worth it in many cases.

None of this seems _hard_, it's just tedious.

Agreed. A few years back when moving my personal domain from a failing provider I considered running my own email server on Digital Ocean; they had some great tutorials on setting up various parts and I followed a HN recommended tutorial. I was already happy setting up LAMP with a nginx reverse proxy, how hard could it be - I got it setup, and realised two things: Keeping everything running without getting compromised was going to take way too long compared to the benefit. Two, Google and MS wouldn't accept my email - not even to myself when I whitelisted it - it's a crap shoot.

Email has serious economies of scale and keeping such a critical system operational, on a small scale, was going to be some deal of stress.

So I just went with my hosting providers Plesk-fronted setup, which matches your para.3 description.

>People shouldn't need to fully understand DKIM, SPF, etc. - they should just be able to click a button, copy&paste generated DNS records, and be up and running.

I oppose this. People should know what they are configuring. This applies to all stuff related to mailservers.

You probably learn more by setting it up the "hard" way (without containers) so that in case of failure you at least have a little insight in how the components work.

There are many more people who want to drive a car, than people who want to learn the minutiae of how it works.

If it breaks, then you get someone else to fix it / get a better product is all they need to know.

Not everything is a car analogy.

But if you want ... it's like having a drivers license.

Or: you only "drive" self-driving cars. In case of emergency you don't know what to do. Perhaps an even better analogy.

I have my own domain, my own bind and my own mail servers. I forward all the mail to google for reading and searching and the main value I get out of it is I actually understand how DNS and SMTP work. Which comes in handy once a quarter at work. I book marked TFA because I never got the newer (!) stuff setup like proper TLS and DKIM. It is very old setup from like 2005 or so and I think is grandfathered into Gmail somehow. At least my newest domain I treated worst than my older ones.

For A) I believe that is what https://mailinabox.email/ is trying to achieve

In search of self hosting email I tried out both MailInABox and Mailcow. MailInABox wanted to control DNS records which was a no-go. The primary driver behind the project seems to have arrogant attitude towards having users do DNS records themselves. Mailcow seemed nice but was guzzling up ram for seemingly no good reason. Finally tried out Mailu a few days ago which seems to work nice, albeit with limited testing.

Mail in a box suggests being the DNS server in an attempt to be more user friendly, there are a lot of DNS records to setup for mail. How ever external DNS is supported (see "System status & DNS" in https://mailinabox.email/guide.html)

The RAM intensive elements of mailcow are search and clamav, you can disable them both if you don't need them

If only it didn't come with so many things (Roundcube, NextCloud) I don't need.

I thought the same thing, but then I ended up using roundcube as my default non-mobile device mail client. And I just ignore the nextcloud bit.

(This is admittedly a very small deployment for two users and 3-4 domains on a $5/month VPS).

A lot of the other stuff I evaluated at the time (3-4 years ago) came with yet more stuff.

Try https://github.com/tomav/docker-mailserver

It’s pretty good, I use it for my own server. It still requires some config, but better than running all the services from scratch.

A very promising project is https://github.com/foxcpp/maddy. It is an all in one server that can both receive and send email.

As mentioned in https://news.ycombinator.com/item?id=23960831 it's good for security to have SMTP server and IMAP server segregated, even separate physical machines, which you'll have a hard time doing when everything runs in the same process. What happens when maddy gets a security hole discovered.

To me maddy would feel like sleeping on live electrical wires, only they are insulated by a thin layer of paper. Hope the isolation does not get scratched anywhere while you sleep. Even a non-dockerized, traditional setup like postfix+dovecot has better isolation.

I believe a server wriiten in a memory safe language greatly reduces the risk of running such setup. To the point where it is an acceptable trade-off to get simpler maintenance.

Depending on how many targeted attacks you get in a day, you may pull apart the monlithic server and run inbound processing, filtering, IMAP access and outbound queueing as separate processes or even on separate systems.

(Probably i should put a better wording on https://foxcpp.dev/maddy/faq/)

You sir, created in me hope for mail server in Rust. But this is just a Go Lang

If I had an infinite amount if time, I would sure write maddy in Rust. But as practice shows, there are only around 24 hours in a day.

That being said, my project greatly benefits from Go concurrency features and simple I/O primitives. Also there are libraries written by emersion for nearly everything.

Oh when will this Rust hype end, November needs to come already.

Thanks for the tip.

> you may pull apart the monlithic server and run inbound processing, filtering, IMAP access and outbound queueing as separate processes or even on separate systems.

Can this be done without forking source code? E.g. just configuring two instances.

Thanks for the link to FAQ. Actually, simple, no bloat’ little dependency software definitely has some appeal.

Yes, this is possible.

This page shows how to configure maddy to only handle SMTP: https://foxcpp.dev/maddy/tutorials/smtp-only/ You can follow it but swap Dovecot for another maddy instance, configured as shown here: https://foxcpp.dev/maddy/tutorials/imap-only/

This works because maddy implements both client and server for Dovecot's SASL delegation protocol.

Alternatively you can configure pass_table to read from shared SQL database. Hm, maddy-tables(5) man page is not rendered on the website but you can look at sql_query module.

even separate physical machines

I think that is overkill for self hosting. Also as the author says below, Go is more memory safe than C/C++ in that there is no use after free and buffers are bounds checked.

Memory safety helps, but isn't the only source of exploitable bugs.

You would not really want so many "private" ports exposed on a public facing machine.

A machine which has IMAP open tends to look like it has some good stuff.

I just forward incoming emails to an internal SMTP server which also serves as IMAP server, so MX records won't simply reveal your IMAP server as well.

You can also just redirect receiving email from a public facing SMTP to an internal SMTP server which also runs IMAP.

This is actually how usual Postfix+Dovecot setup works in practice. Dovecot runs an LMTP server which is SMTP with some minor modificaitons. Postfix forwards messages to Dovecot's LMTP.

I use mailcow and it does both a) and b) very well. You can either selfhost or pay the developer to host it for you: https://github.com/mailcow/mailcow-dockerized

+1 For mailcow !

A simple `docker-compose up -d` does the job. Then you just have to configure your records (SPF, MX, DMARC, ...) and it's all set.

A really nice work from the mailcow's team !

The push for me to use nixos was simple-nixos-mailserver which does exactly this while being fully configurable & extensible.

https://gitlab.com/simple-nixos-mailserver/nixos-mailserver

Doesn't seem to have external user auth mechanism than "declarative user management"?

There is: mailcow https://mailcow.email/

Automatically checks DKIM, SPF and DMARC settings for your domain, comes with docker and can be upgraded by running a single script, and has a webgui and spam protection.

Not a bad list of advice. I'll add a few things for infrastructure:

* Setup ARC (RFC8617) for all mailing lists and forwarding.

* Sign up for the Google & Microsoft postmaster tools. They're not great, but it's better than nothing.

* Implement RFC8058 / RFC2369 unsubscribe mechanisms.

* Use an outbound suppression list to ensure that unsubscribe requests or unwanted mail complaints have lasting effect.

* Article touches on DMARC reporting, for which I suggest Postmark's free monitoring tool at https://dmarc.postmarkapp.com/

* From time to time run your domain name(s) and IP address(es) through a panel of reputation services. Check out any red flags (some of them matter, some don't).

And for content:

* Re-evaluate email templates. Many out there are hot garbage, practically inviting users to hit the junk button, which just hurts sender reputation. Ensure there's always a plaintext part. Don't use tracking/opening pixels. Write validating HTML. Minimise size. Don't bury the unsubscribe link, put it at the top, and make sure it clearly works without hostile barriers like "are you sure" pages or sign-in.

* In the very first paragraph, and without scrolling on a mobile, we must explain, honestly and concisely, why we are contacting someone, and why they might want to read the rest, because we have no entitlement to someone's attention.

* Run all mailers through a spam scoring tool as part of CI/CD, because a high score is a bug.

In addition, Google's sender guidelines are well worth reading, they're more than just self-serving gatekeeping: https://support.google.com/mail/answer/81126?hl=en

The volume of spam is mind boggling. When we first implemented DMARC / DKIM etc our legit emails were 3% or so of all outbound mail per the reporting we got back! We have a somewhat trusted domain.

Some mail delivery systems start to notice that your domain name is being used as part of a lot of spam / bogus emails, so you can have perfect IP history / no spam and STILL start triggering some random filters (no big players but smaller protection product filters).

So the game must be absolutely never ending for everyone and the inbox a valuable target - especially now that unsolicited phone calls really do seem to get ignored these days - I feel like spammers killed the golden goose on phone calls and the telcos let them.

I will say DKIM / DMARC is working well, except google (which we now use for outbound) gives us transient SPF errors even though we are 100% using their IPs. Not sure why that is (ie, SPF failure on an IP that should clear)

For your last point, check https://toolbox.googleapps.com/apps/checkmx/check. Chances are, google doesn't like your DNS server answers to its ANY requests, regardless for them complying to the RFCs.

As someone who works in email security I can tell you that it is a never ending cat and mouse game with a lot of false positives as to who is a mouse. Even our customers that have good domain rep, SPF, DKIM, DAMRC, etc still randomly get blocked by filters and large email providers. Part of the problem I see it those large providers and some of the filtering systems send back NDRs that basically provide no information as to why the message was blocked so trying to get your legitimate mail flow working again once someone has flagged you can take days to months as you randomly adjust all the knobs trying to figure out why they are blocking you. I'm sure they would say providing more informant NDRs would make it easier for the spammers to work around the filters. I don't know what the answer is but the current system sucks, it is a huge pita and a dumpster fire for the most part but it's what we have to work with. I'm sure someone will come up with another scheme like DKIM that will become another check box on the list of things you have to set up and once agian few will understand how to implement correctly.

I think we are just going towards centralized trust proxies.

Ie, if you are paying for google apps, have credit card on file, meet their rate limiting rules for outbound with SPF / Dkim etc then you are probably OK. Some random IP doing direct mail? Much less likely to be OK.

Given govt has done such a bad job in this space, these big corps are essentially picking up the slack / trust that you'd normally say govt was responsible for. Gives them a metric ton of power, and they don't tax so have no money to provide any corresponding service.

Tangentially related but I'm having some issues related to email beeing refused by Gmail.

In my organization we have ~10 linux VM and they are all configured to send email to an exim MTA. For example monitoring stuff. The system mail name is set to vmName.srv.domain.name (domain.name is replaced by our actual domain and this resolve to a local address) so it's quite common to receive mail from root@vmName.srv.domain.name. This cause no issue when it's delivered to the local Cyrus server but when it's transfered to a Gmail address for some reason Gmail reject the email because the headers are bad.

I am considering rewriting the address to user.vmName.srv@domain.name in exim when receiving email from local VMs and I am wondering if this could be a bad idea.

I'm going to redo this mail server from scratch soon because it has been left untouched for something like 6 years. Some software was compiled on it with shared libraries which then were updated but the software was not recompiled. It's a dumpster fire and I'm actually surprised most of our emails seem to go through.

I would also add to the article that's it's a good idea to have some rate limit on outbound email, outbound email spam checks and to check that the user sending email has the right to use that address.

Quick question (as someone who doesn't manage email): why not using an external mail provider (like mailjet) ? Privacy concerns and/or internal mails that shouldn't get leaked ?

That server is at least 10 years old and quite custom. For example when an email is received the address is queried to an ldap and if it's a mailing list the email is sent to all the recipients of that mailing list which can be other mailing lists ! This goes on until an user address is found and then the email is delivered to the cyrus server.

Each user can also have many alternative addresses that map to an unique mailbox

Those mailings lists are used a lot internally and managed with internally developed tools. This make migrating to an external service difficult.

That's the classic example of some critical piece of infrastructure that nobody dare to touch because it works... until it doesn't anymore.

That's why we don't use external services to receive email. I'll consider using mailJet if setting up exim & related stuff correctly to send email in 2020 proves too painful.

Somebody else mentioned you could temporarily sign up for G Suite and use the same domain, and it will magically be whitelisted.

I am assuming you have some budget for this.

Are you using mxtoolbox.com or other third party troubleshooting services?

If the reverse DNS/MX is configured to allow emails from @domain.name, then you have to send email with an address in @domain.name.

@vmName.srv.domain.name is not configured so it goes to spam.

Microsoft (Live/Hotmail/Outlook/...) is the single worst offender against small mail servers. I have run a mail server for the last ten years spam-free and vulnerability-free. It has never been on any spam blacklists, delivery to GMail is just fine. I have implemented everything from SPF to DKIM to correct rDNS.

Yet, my IP address predictably ends up on Microsoft's "blacklist" every two months or so because of "spam behaviour or user complaints". I am rather sure that neither is true. Every time I have to fill in their appeal form, get unlocked immediately but have to do the same dance again two months later.

They have every right to reject a mail server but at least they shouldn't lie about the reason. My server's IP address gets blacklisted again and again even during months I don't send any mail to Microsoft addresses.

Maybe they have a rule that unless you keep sending clean emails to them periodically, you get blacklisted.

With all the fixes and standards just to send a cute cat GIFs to your friends on Gmail is such a pain in the butt for most self hosters. If you're doing it, great! Keep it up! All the better for you. But for me, using a e-mail hosting provider with a good reputation is preferable, they've already done all the hard work (my preference is FastMail, but it's not everyone's cup of tea). I've been in the IT field for almost 20 years, I've done the self hosting e-mail thing. In my early teens and 20s, I didn't have anyone to please or take care of, so keeping up was easy with new e-mail standards as they rolled out, 15-20 years later, I'm divorced, work full time and have two kids, I simply don't have the time to self-host anymore. I'd rather give somebody a few $ a month to take care of that for me.

I've been so irritated with self hosting I've gone the other way: I'm in the process of setting up an email hosting co-op, with the aim of paying for managed service from a small hosting provider for most things but paying no more than I do now. That way it won't only fall to me to deal with deliverability issues etc.

See my sibling comment to yours! I think we are both working towards the same goals. Nice to meet you.

> I'd rather give somebody a few $ a month to take care of that for me

I'll do it though i'm not FastMail!

I'm working on a SES backed email client so SES takes care of deliverability (assuming the user does the basic DKIM, MX, TXT set ups) so it will work out much cheaper to send and receive mails than a FastMail/GSuite type solution.

Plus you'll get unlimited disposable emails for free, a shared inbox (if you want to share an inbox with your family to help stay on top of every one's schedules, for exam), and while the client I have built is pretty rudimentary, i'll be adding bells and whistles to make it better.

Github @ https://github.com/saiorama/ses-email-client Landing page to submit a request @ https://shared-inbox.landen.co/

Note to others - I'm open to collaborating to take back email from the big guys. Hit me up via Github.

This just seems bizarre - why?

If the point is to take back email from the big guys, building on top of AWS SES seems to do exactly the opposite of that.

It's built on a completely proprietary service, which you have very little control over. If you're having issues with it, you're SOL unless you're paying for an AWS Support Plan, which is already well over the cost of Fastmail etc.

And on top of that, now you are stuck on a specific web email client as well, instead of POP, SMTP and IMAP which all of email has been running on for decades.

> why?

because I could! Isn't that the point of tech - to be able to build whatever you want, whenever you want, for any reason you want, in any form or shape you want without asking for permission?

The hexagon nut in your car is built on a technology that you don't know or own. That doesn't mean your ability of tinker with your car is subject to the whims and fancies of a nut manufacturer or the hexagon spanner manufacturer. You just trust that when you need a hexagon nut of a certain spec, you can shop for one. SES is exactly that hexagon nut to me. If you consider how easily an MX record can be created/changed, how am I losing anything by attaching myself to SES's email service?

AWS Support Plan - of all AWS technologies under the AWS sun, SES seems to be one service whose support needs are minimal and outages aren't going to end my productivity. Email is async so that delivery delays by a day or two aren't really an issue (to me). Email is federated so if AWS wants to be taken seriously is needs to interoperate with other systems a high degree of reliability.

a) The web client i am stuck on is mine. I can stay stuck on it or enhance it or shut it down or let it lie for long stretches of time.

b) What S3 provides is even more basic than POP/IMAP/SMTP - it gives you the literal email files which are the bedrock of email to do as you please. I like my chances of building something amazing with the raw data rather than having to have my emails intermediated by a GMail or a FastMail.

> because I could! Isn't that the point of tech - to be able to build whatever you want, whenever you want, for any reason you want, in any form or shape you want without asking for permission?

Well yes, if it makes you happy, absolutely, do it.

But another reason we are all here is to discuss the merits and problems of various technological approaches.

> The hexagon nut in your car is built on a technology that you don't know or own.

This just isn't true – the hexagon nut is completely open. I can go look up the length, shaft diameter, thread pitch, etc. Countless different manufacturers make one to the exact size. I could even make one myself with a lathe if I were so inclined.

SES is proprietary. I could not replace SES with something else.

> If you consider how easily an MX record can be created/changed, how am I losing anything by attaching myself to SES's email service?

Because if you wanted to move away from SES, you would have to completely rebuild how your client receives, stores, and sends email.

If you build your client on POP/IMAP/SMTP, then you can just change your MX settings, and your POP/IMAP/SMTP settings, and you can move all of your email to AWS, or Fastmail, or Gmail, or your own machine, in the time it takes DNS to propagate.

I made Forward Email after running into many issues with self-hosted solutions (and realized thousands of others have the same issue due to the high-demand after launch back in 2017). It's a great alternative to having to set up your own small mail server.

https://forwardemail.net

P.S. I'm coming out with a fully-fledged email service built on top of my R&D with Forward Email.

Any cheap paid services for this? I'm not comfortable with routing my emails through a free third-party.

Then how are you comfortable routing it through a cheap service instead? Asked more precisely: how do you determine reliability from price?

It's more that you tend to get some legal implications from selling a service vs. having some website with no payment on the net.

We do offer paid plans with enhanced features.

And we're completely open-source, store zero logs, and store zero metadata too. All source code is on GitHub as well.

I formerly worked at DuckDuckGo, and take privacy very seriously.

You can delete your account at any time.

Forwardemail.net seems to have paid plans.

Reading through the about and FAQ they also seem to be driven by decent principles.

https://forwardemail.net/en/faq#how-is-it-free

Yep! Both free and paid. We have some pretty serious security principles. I'm the creator, so if you have any questions let me know. All of my work I do is open-source -- and we also have a public Slack channel and Twitter where I post updates. The paid plans are for if you don't want to use free DNS-based forwarding setup, a small fee for unlimited domains/aliases/etc to help me maintain server costs (see the FAQ). I may make Enhanced Protection available for free at some point in the future, but I built it this way at first because the website used to be a simple wiki, and I used the DNS-based setup myself 2 years ago.

Having the top reply to an article about hosting your own e-mail be somebody shilling their service based around the idea of how you should not in fact, run your own mail service is a bit gross.

Will the fully-fledged email service also be open source?

(This is the first time I've become aware of Forward Email despite having run my own forwarding email server for years -- you should do more/better marketing/outreach!)

Yep, everything I create is open-source!

Where are your servers located?

Our existing infrastructure is hosted on dedicated (not shared) hardware in the US, UK, and Europe. Our new infrastructure (which will be running on bare metal servers) will be hosted in the US and Sweden (in underground bunker data centers).

I'd still have to have a mail server somewhere that receives the messages your service forwards to me though? Your service just provides one level of indirection, correct? Edit: and what about outbound email?

We are going to release outbound/inbound SMTP feature soon, so you can use our service outside of Gmail (and others') 'Send Mail As' features. If you sign up (and don't add any domains or anything) I will send you an email when it's live. Or you can follow me on Twitter or subscribe to the GitHub releases. Or I guess you could just email niftylettuce@gmail.com and I will put you on the list to send to, haha.

Mailinabox handles all of that automatically and nicely. You just need to do some manual DNS work and check your IP against blacklists on http://multirbl.valli.org/ - and mostly done.

I think part of the point of the post and the PDF to which it links is that Gmail and the other big operators don't use the blacklists, so for deliverability to the majority of recipients it makes no difference whether you are or are not on those lists.

Example of how the blacklists contain essentially no information: every one of Gmail's outbound VIPs is in a SORBS blacklist. Every one I checked is in at least 20 other blacklists. So if you just don't want to get mail from hundreds of millions of people, subscribe to IP blacklists.

That's not true, unless "VIP" means something that I don't understand.

I use SORBS on a rather busy mail server with thousands of messages sent to and from Gmail daily. Aside from very occasional SORBS rejections, which happen more often with Outlook.com and Yahoo, this simply isn't the case.

See for yourself:

http://multirbl.valli.org/lookup/209.85.128.65.html

Seems to be mostly not listed?

Sure but blacklists are binary. That’s why they don’t work.

It's perfectly fine to check 282 blacklists and treat only N hits as a positive?

"every one of Gmail's outbound VIPs is in a SORBS blacklist. Every one I checked is in at least 20 other blacklists"

I still don't understand your statements. That's simply not true. It's not even close to being true.

Post a Gmail outbound VIP that is not in a SORBS blacklist, then, if you're so sure.

Whatever the opposite of a disclaimer is: I was the tech lead of gmail delivery SRE for four years, so it's likely that I know more about this topic than anyone on this board.

Thank you! I recently managed to get a static IP and was looking forward to set my own mail server to be independent of the big providers. Now I have been searching for something exactly like this, a recent, concise overview of the hoops I have to jump through to be allowed to send to the big guys.

I wrote one as well @ https://medium.com/@ssr233/get-disposable-emails-for-your-do...

I'm also trying to build my own email client for SES @ https://github.com/saiorama/ses-email-client. Happy to collaborate in case you want to work on this issue.

Point 14 is a very good one. If any account gets compromised, you might as well request a new IP as your previous IP will be in the dog house for a long time.

Almost more important than monitoring failed logons, monitor successful logons, for instance by country. If it is a small mail server, chances are your users are in a handful of countries. If you suddenly see a successful logon from vietnam or brazil, you probably want to be notified ASAP.

Addendum: if you host a mailing list server such as Mailman, you'll need to do one of (1) don't change the content or DKIM-signed headers at all (e.g. by adding Subject tags or unsubscribe instructions); (2) rewrite the From: header so that you don't look like you are impersonating the individual senders; (3) use ARC [https://en.wikipedia.org/wiki/Authenticated_Received_Chain] signatures. I don't know if Gmail and the other big providers actually honor ARC headers yet, but if they ever do, that would seem to be the cleanest solution if you need to add unsubscribe instructions or similar banners.

I set up a personal mail server (SMTP+IMAP, no webmail) a couple months ago for personal/family use —- no sense in paying about $60 per user per year for GSuite or 365.

(Microsoft actually offer a custom domain email solution for families that doesn’t charge per user provided you host your domain with GoDaddy, but it has some limitations, eg with respect to aliases, so that wasn’t quite right for me.)

I did so with some trepidation, having been told by multiple people that I shouldn’t. Maybe I got lucky and had an IP with good reputation, but I was able to send mail to Gmail, Outlook, Yahoo and specific email lists after a few rounds of configuration.

The only problem I had was with Apple’s iCloud email, but engaging with ProofPoint soon fixed that.

Overall it wasn’t too bad an experience. I’d class myself as beginner level in that I’ve done this for myself ages ago (but never in a professional capacity).

Just leaving this anecdote out here as a small counterpoint to all the horror stories out there. If you’ve got an IP address with a poor reputation I can imagine it’d be a much more difficult exercise.

Would you mind sharing if you're using a cloud provider or something hosted from home/co-located?

It's a small droplet on Digital Ocean.

Not sure if I want to carry on host other people's email anymore, it's become a huge pain. It's bad enough that Google and MS frequently refuse to deliver mail; without any apparent reason or any humans to discuss the problem with. But now that zimbra is dead (no new open source versions), email hosting is pretty much back to where it was in the 00s.

For a long time, the high-risk crap-spewing networks tended to be the same bad actors (eg: OVH, Digital Ocean, Psychz).

This year though, most of my dodgy SMTP traffic is coming 3 new problem children - Gmail(spam), Amazon(spam, other) and Microsoft Azure(other).

"Other" here is anything besides spam, like dictionary attacks, transactionless connections or service probes.

I went through the journey of configuring Postfix for a small side project I was working on.

The idea was to accept emails sent to a unique, per-user address and forward all attachments to the user’s cloud storage account.

I looked into using mail services like SES and Mailgun, but given the average expected email size and the fact that I would primarily receiving mail, the additional cost didn’t make sense. Along the way, I ended up learning quite a bit about Postfix filters, virtual domains, SPF, and DKIM.

I also learned how quickly spammers can detect open relays. Apparently, while copy-pasting a config from somewhere, I had a slightly incorrect 172.x subnet listed as part of “mynetworks”. As a result, a spammer with a machine belonging to this subnet was able to use my server as an open relay. Long story short, it took me quite a while to root cause the issue and get my domain off the blacklists!

Make sure you add yourself to dnswl.org if you're running a mailsever. Lots of people who use Spamassassian will check it (by default, it's part of the base install) and if you put yourself in there you will score yourself a few negative figures to help get your mail delivered. Of course DMARC, DKIM and SPF are key as well. I've run my own mailserver for the last 20 years and properly signing messages and being sure not to accept/forward spam (i use rspamd) has helped me be able to have a stable personal server all those years. The only places I have trouble with are Microsoft, because it seems if you don't send a mail every 24 hours your reputation gets "reset" and it can be hard to deliver into them.

Ugh.

I have been running my own UNIX mailserver since 1997. I have consistently and loudly evangelized this practice, especially here on HN, and pushed back against the notion that it was too difficult or too time-consuming to implement and maintain one's own mailserver.

But ...

Having just recently made the major transition from relying solely on clean IP space and proper DNS records (which, in the last 18 months, has become increasingly untenable[1]) I have changed my mind.

I will continue to run my own mailserver, for activist, ideological reasons, but I must now agree that it is too difficult and complicated - and fragile.

In no particular order, some of the things that I find overly complex and disturbing are:

- The DKIM implementations are very, very over-engineered. I understand, in principle, why we want DKIM to be a pluggable component that can be used as a general building block in various implementations, blah blah blah, but there is no reason that DKIM can't just be a feature, with a corresponding blob of config lines, in sendmail or OpenSMTPd or whatever. Having to pkg-install, and track, and maintain, whatever DKIM implementation you choose, along with all of its various dependencies, etc., is a big truckload of complexity and fragility that I just can't see ever appreciating. And the irony ? A popular DKIM implementation is to use the bundled DKIM functionality built into rspamd (!) ... so it ended up being just a feature anyway.

- The whole SSL component ... I appreciate this, I understand this, and for many use-cases I just don't care. I don't currently need encrypted email and if someone makes a decision to disallow sending to addresses/domains that don't support it, fine. Of course, if all of gmail decides not to, now I have another giant truckload of complexity and fragility that I need to implement and maintain. My mailserver should be an extremely tight, stripped down host with as few packages and daemons possible[2]. Instead, I am now installing a big list of packages just so that letsencrypt can fire up a temporary web server to clicky click make my certificates[3].

- Now my mailserver is running a database (Redis) which is required to run rspamd, which is required to implement DKIM, without which I cannot send mail to yahoo.com addresses. True story.

So much complexity and fragility ...

[1] About 18 months ago yahoo.com stopped, with no errors or bounces, delivering mail from an 11 year clean IP with proper DNS/MX entries. I implemented DKIM and the problem is solved.

[2] My mailserver is a FreeBSD jail and previously ran only sendmail - and nothing else. Now I am running OpenSMTPd, rspamd, redis ... and once in a while I am running a webserver to communicate with letsencrypt.

[3] Speaking of letsencrypt ... requesting, generating and implementing SSL certs (even for use on the web) is simply creating, and trading, blocks of plain old ASCII text. I haven't looked into this, but is there an "expert mode" somewhere in letsencrypt where I don't have to run webserver instances and ... I can just run openssl command lines and cut and paste blobs of ASCII ?

Fellow long-time self-hoster. I hear you on a lot of this.

Regarding letsencrypt, you can use DNS challenges. See https://letsencrypt.org/docs/challenge-types/ .

I've been self-hosting email for 25 years also.

I think of it as a process of punctuated equilibrium.

Every several years (or ten), it requires a flurry of activity to update to new preferred daemons, or new hardware.

Between these brief periods, everything is stable and solid. I don't have to think about it at all. In 25 years, I've rebuilt things 5 or 6 times. After the first few times, it has been more of a chore than an adventure, but it's still worth it to me, for now.

One development that might change my mind about that: Ubiquitous and non-vendor-dependent MFA. If email was not the security mechanism of last resort (or the skeleton key), then I might consider outsourcing it.

...

Re: Rspamd: It works well, but I did not enjoy the configuration effort. And yes, I never wanted Redis on my mailserver, but there it is -- along with an astonishing amount of RAM usage, which offends me in theory even if it has no practical impact. On the positive side, after the initial few hours of effort to get things set up, I have spent zero hours maintaining any of it, so I can't reasonably complain.

Re: letsencrypt: I use dehydrated for certificate renewal, and I run a temporary webserver as you describe. Not a full dedicated webserver package, just python (or ruby) which is already installed. It's launched from the same script that automates the license renewal. Not elegant, but adequate. You could probably get away with netcat.

Of course you can validate by DNS instead of HTTP. But that's not always convenient either.

"Every several years (or ten), it requires a flurry of activity to update to new preferred daemons, or new hardware."

I think you are correct and this is a decent description of how this has worked.

I don't begrudge the every 5-10 year burst of activity/implementation. What I begrudge is the increased complexity. We're both talking about Redis as a hard requirement for rspamd which is the typical recipe for implementing DKIM which is required for email. That's a lot of moving parts.

I've been running amavisd-new for ~10 years (it runs spamassassing and clamav) and have been DKIM signing with it the whole time. Never had the need to switch to rspamd. You can also sign emails with OpenDKIM.

I've also been self-hosting email for 25 years. I see these exact patterns you describe.

Currently I'm using postfix, dovecot, and rspamd, with several users using pine and mutt locally. Every time I do a major server upgrade, I re-evaluate the configuration.

My mail server also runs a web server for some static sites, so the letsencrypt part was simple.

There are much simpler ways to implement DKIM than you have done, not sure why you have chosen such a massively complex one, for instance

http://dkimproxy.sourceforge.net/

I think the reason rspamd+DKIM is so popular is that you're going to run a spam daemon anyway so ...

> you're going to run a spam daemon anyway

I don't.

AFAIK dkimproxy is no longer maintained. That is the main reason I dropped my plan using dkimproxy with OpenSMTPD for a simple send-only mailsetup. The rspamd+redis setup felt way to overkill for a tiny vps.

Postfix+opendkim+opendmarc works fine and it is very simple to setup as well.

..and even if you follow all the best practices you'll just randomly end up blacklisted for what I can only imagine is misguided users clicking on spam.

If that happens on the wrong email provider you're out of luck. One of the mail servers I have a hand in has been blocked by one of those difficult providers for the past 2 months and we sent them 4 mails/week before the block on average :)

> [3] Speaking of letsencrypt ... requesting, generating and implementing SSL certs (even for use on the web) is simply creating, and trading, blocks of plain old ASCII text. I haven't looked into this, but is there an "expert mode" somewhere in letsencrypt where I don't have to run webserver instances and ... I can just run openssl command lines and cut and paste blobs of ASCII ?

Let's Encrypt provides an ACME service. So, you will want to (run something that can) speak ACME, a somewhat RESTful HTTP API documented in RFC 8555 - https://tools.ietf.org/html/rfc8555. You can use a web service to do this part, but then you're relying on some random web service and you have to manually do a bunch of steps to renew at least every 90 days which I'm going to assume you agree sucks.

There are lots of people who have built such tools that can look after this for you automatically, perhaps you would like https://acme.sh/ which as its name might suggest is a shell script.

Let's Encrypt offers three of the (increasingly misnamed) Ten Blessed Methods, you can prove control over a DNS name by doing any of these three things:

1. Running an HTTP server on port 80 which can answer certain well-known requests in a specific way acknowledging your ACME keys. This server needn't have any privileges, acme.sh can help you spin one up that will basically say "Yes, rsync is allowed this certificate" for any request. Except instead of "rsync" it's a key only you have, so your server answering this way never helps any bad guys so long as you keep your ACME keys secure. [You may not be aware you have "ACME keys" the software you've used probably just minted some and squirreled them away in a dot directory]. With that server in place, your requests for a certificate for that name will always work, because the key matches, Let's Encrypt will call your server, make up a random nonce and ask if that's authorised and for who, your HTTP server will always answer "rsync is authorised" so it works, tidy.

2. ALPN on port 443. A TLS server answering port 443 can accept a special ACME-only ALPN protocol choice from the Let's Encrypt servers and use that to prove control. You almost certainly don't want this unless you implement web servers (which run on port 443 with TLS already)

3. DNS queries for a magic name that can't be a hostname. The queries ask for _acme-challenge which has an underscore in it. Underscores are allowed in DNS but can't be hostnames, which is fine because this isn't a hostname. Your ACME client will need to write to DNS, it can either have a way to write to your actual DNS, or you can use a CNAME to redirect the names you need to another DNS server and give the ACME client control over that. [Edited to fix DNS name mistake]

You can insulate the ACME client from the server completely with either the trick I described in (1) or using DNS like (3) and using a Certificate Signing Request, CSR, which is one of the many ASCII text blobs you can get from OpenSSL. If you're comfortable re-using a private key for longer periods than the 90-days assumed in Let's Encrypt (a 2048-bit RSA key seems safe to me for the usual lifetime of a mailserver) you can even re-use the same CSR again and again to request each new certificate without sending any data from the mail server to the system calling Let's Encrypt.

You will need to arrange to get each new certificate installed (the private keys don't change in this scenario). But certificates are public documents, if it came to it the mail server can (this is very silly but it would actually work) just wait until 24h after the certificate should have been issued and then go get it from https://crt.sh/ -- In reality you can probably find some sensible way to get this public information back to the mail server each time the certificate is renewed.

Can you elaborate on your problem with Yahoo? I ran into that a while back, where a yahoo email address couldn't send to me... strange since DKIM validates for the domain.

There are all valid points and I question my decision to run my mailserver, but in the spirit of the Internet I continue to do so.

I meant the opposite - I could not email yahoo.

> Now my mailserver is running a database (Redis) which is required to run rspamd, which is required to implement DKIM, without which I cannot send mail to yahoo.com addresses. True story.

I have DKIM set up just fine without running Redis or rspamd so this part is purely down to your choices.

> Speaking of letsencrypt

Let's Encrypt is a website/service. There are many clients and you can write your own. At some point Let's encrypt will need to verify that you control the domain so you will need to be able to either host something via HTTP or have sufficient dynamic control over the DNS server. Neither DNS nor HTTP needs to be hosted on the same host as your mail server though.

Gmail still sends good emails to spambox even today... For example, a realestate agent sent documents with really personal info in them and it still got qualified as spam

Without more information it is hard to say if that is a mistake.

Sometimes when businesses send sensitive documents they use a third party "secure" service, if the realtor's domain isn't configured correctly (common) to allow that third party to masquerade, it would correctly be flagged as spam.

Plus I get actual spam (and even generated blackmail) with personal information in it (inc. old passwords), farmed from one of a dozen service break-ins over the last 20+ years (thanks Adobe, amongst others).

If the user wanted the email, its a mistake if it went into the spam box, period.

Trust me when i say you can have a 100% flawless email setup in every technical respect (the full works from this article and more) and still end up spam boxed by Gmail approximately 100% of the time.

Conversely, people that Gmail feels are important wont get spamboxed for minor technical reasons either...

Email / domain reputation is very complex. Their domain may have been flagged due to excessive marketing emails, so no matter what they send they are heavily weighted towards spam.

I worked at a SaaS company that had about 50 sales reps. They were sending emails day and night, eventually getting the main corporate domain flagged. The product's transactional emails (signup, confirmation, etc.) would be flagged as spam, until we moved them to their own domain.

Very interesting article, but hosting my own email is a fantasy for me that I will probably never spend the time setting up and maintaining. ProtonMail does everything I need in an email service, with FastMail and Hey (which I only tried for a week) being excellent runners-up.

Sorry for going off topic but everyone should host their own web site(s) and stop putting their content on other people's platforms.

I personally use zoho.com(free plan) for email to and from my domain. What is the use of self-hosting a email server over using Zoho or Gmail?

I like hosting mine for a few reasons:

- It's a hobby and I enjoy it

- It like fiddling around with advanced features, like wildcard addresses, server side filtering (via dovecot-sieve), and other fun stuff. For instance, a friend once was mulling on facebook on detecting tone of email and rejecting ones that were too mean. I made a simple python script that attempted this and had her try it out. It was funny. (See previous point about hobby)

- I figure that being on your own server you're at least slightly less likely to be a victim of a big internal platform hack like twitter's recent one, or state-run ones.

- It feels slightly badass.

I don't recommend it to people who don't want to do it for fun.

Sovereign ( https://github.com/sovereign/sovereign ) takes away most of the work of setting up an email server. The project needs volunteers!

TL;DR: Cool that this thread exists because people will mention (please do!):

* Docker-based e-mail setup, with comments about their experience, like https://github.com/tomav/docker-mailserver (which incidentally IIRC was recently looking for a new maintainer).

* ...ideally that would have one host running SMTP facing the big bad public Internet and therefore minimalistic, and a separate more isolated host that runs the IMAP server. (My current setup does that with, looking for hints but perhaps I have more to share than to learn this time.) -- https://foxcpp.dev/maddy/ mentioned in this conversation may simplify things but with IMAP and SMTP inside one process you'll get a hard time doing this separation, not a good point for security.

* perhaps advanced mail routing like user-selectable option to automatically create mailboxes, per mailing-list and envelope address, although technically we could start another conversation about this. (My current setup does that, looking for hints but perhaps I have more to share than to learn this time. One pain point is many senders generate uninformative List-Id or even change them at each message, even Mozilla for example, making things messy.)

My current setup runs Postfix for SMTP on one host, doing all the rejection of spam and forwards approved messages via LMTP to another machine holds the private mailboxes contents to be served via IMAP by Dovecot. Plus custom recipient delimiter instead of the traditional "+". And all point above except being Dockerized.

I wonder if there's an ansible playbook implementing all that?

Sovereign does most of it:

github.com/sovereign/sovereign

I was an early contributor to Sovereign and used it for many years, but I found that we kept adding features to it and it got too large for my liking. Now I use docker-mailserver:

https://github.com/tomav/docker-mailserver

I looked at Sovereign a while back, and came to the same conclusion, and ended up going with mail-in-a-box, which is possibly a little too far the other way, but at least very low maintenance.

I'm not brave enough to run my own mail server, it's just such a critical service to me.

Eh, I didn't read it...so my thoughts. From experience the prob with DNSSEC is latency in verifying PKI of the record. Usually what happens is the timeout per resolver has to be greater than 15 seconds under no load in a private network which is very inconvenient in comparsion to plain DNS that has a default of 5 seconds.

I’m pretty technical but the one thing I’ve sworn off is DIY email servers.

It’s not worth the pain.

Running a personal mail server in these times is a little bit complicated but well within the capabilities of anyone who manages servers for a living or even advanced hobbyists. That doesn't mean it is a sensible thing for most people to spend their time on. The alternatives are both very good and very affordable. I do it because this isn't any harder than programming or administering other server technologies and the only way to keep skills current is to actively work on real systems.

Many things on that list are not necessary to get emails accepted by the big players. There are many email servers with bad or outdated configurations that still work. To keep emails flowing to outlook/hotmail/live etc you need to sign up to SNDS and their junk mail reporting program and if you haven't sent emails there regularly enough to maintain a reputation expect to be blocked for no reason. If that happens there are ways to escalate the issue.

The basics you must do is secure your email server. Don't relay email. Have all senders authenticated over secure connection. Web email forms are a bad idea. You don't want to have to recover from being on everyone's block list. If you do any sort of mass mailing you would probably be better off moving it to a specialised email marketing platform. I handle some application related emails myself and you have to be prepared to check delivery problems and follow up. The ones I do are B2B, opt-in, relatively low volume, small number of business email systems who can whitelist and are expecting emails which is just about manageable.

The bare minimum to be a good email sender is to have your mail name, hostname and reverse ip all match up and to advertise SPF. I would recommend also signing with DKIM, advertising a DMARC record as these are very widely checked and not that hard. They generally will give a slight negative spam score although the downside is you get the occasional business that doesn't know how to configure their mail system to work with third party filter (eg mimecast). They may reject emails according to your dmarc policy for correctly recognising that your email has been intercepted and tampered with but usually those sort of mistakes get discovered and fixed. If you use an intermediary like this to validate your emails you have to turn off validation of these things on your own server.

It probably won't be long until MTA-STS is as widely supported by big emailers and used as a reputation signal so it doesn't hurt to set up a sensible cipher suite and a certificate on your server even if you don't care if emails are sent over tls or not. These addons are mainly a hassle because they all rely on dealing with lots of separate services - mail servers, certificate authorities, web servers, dns servers but this is because smtp didn't anticipate the abuses of the modern world.

I have dnssec and DANE as well but it is just for my own amusement like trying to collect all the moons in Mario Odyssey. No big email providers even look at it as far as I know.

Reply on news.ycombinator.com