Deploy a Gmail-like email server in 30 (ish) minutes | Hacker News Reader

Deploy a Gmail-like email server in 30 (ish) minutes

andycallaghan.com

192 pointsacallaghan179 comments

It's not the initial setup. It's the maintenance over the years that really makes you question the universe, life and the decision to host your own mail. When you can't send that important mail because $big_provider is blocking you. When someone decides to run a persistent brute force attack from a botnet, eating up 100% of your CPU and you have no meaningful ways to block it. When you need to explain to people why they can't send you that 100 MB video attachment which they sent to other people just fine but only your address is bouncing and why don't you fix your email already. When you need to research, understand and implement standard X pushed by $big_provider because otherwise things will stop working and you have a ton of better things to do. When you get a random alert that email volume is too large and you panic because someone hacked your server and is probably sending spam but realize it was just triggered by a huge kernel patch series sent by someone on a mailing list. When a zero-day CVE for your mail software just hit the top of the HN and the fix is not in your distro yet and you scramble to find a workaround. When a bunch of weird log lines appear in your mail logs and you don't understand where they are coming from and they seem benign but can't lose the feeling that someone is trying something malicious. When you constantly fear that you'll lose that good IP and domain reputation and one day wake up with half of the internet blocking you.

As mentioned last week here in a previous “run your own email” story, your setup can also get rejected for not sending enough email, meaning your volume can’t establish good reputation.

...or for choosing the wrong VPS provider. Op talks about Digital Ocean. But, just like Linode or EC2, their IP blocks are inevitably on some undisclosed blocklist that livemail, yahoo, gmail randomly use.

Digital Ocean sends so much bad traffic in the form of bots trying to run cars numbers we’ve had their entire AS blocked for several years.

> bots trying to run cars numbers

I'm having trouble parsing this. Are you talking about VINs?

My best guess is it was a typo for (credit/debit) card numbers.

Yep sorry. Autocorrect.

I was going to say, IP blocks for DO make it one of the last places I would use for serving mail I actually wanted to be deliverable.

Yeah, my solution for low-volume self-hosted email is to relay my outgoing SMTP traffic through Amazon SES. I get good delivery to all the big players, bit still control all the parts of the email stack that I care about. (Plus at low volume, SES is basically free...)

Your solution makes a lot of sense if sending emails through AWS is acceptable, but I'm not sure we can say it's self-hosted.

When you send an e-mail, as soon as it leaves your mail server it's out of your control and you have no say in where it goes before it ends up at the recipient. There's no such thing as self hosted sent e-mail.

Sending an email through AWS has strong implications regarding privacy though.

The odds of the emails going through AWS, GCE, or Azure infrastructure in one form or another is probably pretty damn high even if you host your own SMTP service.

Yes but that's the recipient choice. I'm not responsible if they use American cloud providers for their emails, but I am for the ones I send.

Sure, then you can use whatever SMTP option that is best for you. Some people are fine with using American cloud providers and some are not. If they are technical enough to host their own email server, they are more than likely to understand what choices they are making.

My main point was that sending email using AWS SES is not self hosting. Like hosting a website on S3 + CloudFront is not.

I believe the correct way to write it is self-hosting.

That is _your_ definition of self-hosting. Do you rely on an entity for your internet connection? At some point, you _will_ need to rely on someone else's infrastructure.

If you run your own incoming mail, nobody but you has a copy of your entire mail archive and you are self-hosting that data. It can't be handed over to the government, it can't be sold, it can't be analyzed by gmail, etc. That's where the big win is. If you send e-mail using SES, that's one untrusted hop out in front of an unknown number of untrusted hops that you can't opt into or out of - there's little practical difference. The privacy win come from hosting your own data, not your own smtp server with a carefully curated reputation that you spend hundreds of hours a year working on.

Chances are you’re already compromised eitherways , most mailservers interact with you over STARTTLS, which means any middleman/isp can strip your tls encryption on your emails, midway while you’re reading them over imap. (ISPs worldwide have been caught doing this before)

Most providers do not mandate a strong SSL only imap system.

Along with other similar caveats, do not depend on your mail system to keep your emails private.

If you want privacy, encrypt your emails yourself.

Also most people who selfhost their email usually do not encrypt data at rest. As its not the default norm. As such anyone having access to your disk can compromise you at anytime too (especially likely if you’re using a vps).

So, unless youre encrypting your emails yourself E2E. Assume you have no privacy. If you’re doing E2E, aws cannot decrypt your mails anyways.

There is a large difference between "them" having to m2m smtp sessions to intercept emails versus directly sending the emails through them.

Besides, metadata is probably already more significant than the message contents and encrypyion solves nilch there.

they m2m the imap sessions using the method i described not smtp.

Do you have any more details on implementing this? Sounds like something a lot of people would be interested in

I have set up the same thing for myself and have been using it for several years now, so i’ll join in.

Aws ses has this offer where for a few thousand emails per month, email sending is free.

The steps are this:

1- Signup for aws ses, once you do that they’ll put you in a sandbox environment

2- After that they’ll ask you a few questions on why you need it, just tell them its because you’re a growing startup who expects to send thousands of emails per month, (make sure to say this, they don’t crosscheck later, if you dont say something along the lines of this, they usually reject your application to avoid having to serve small customers who might not scale their business later. )

3- After you’re approved, they provide you with a mail relay api key, just take that api key and attach it to your postfix or other smtpd installation

I use docker-mailserver[0] which packages everything I need for my mailserver into a small container and was good to go, it consumes minimal resources too.

For me, i just had to add the ses relay api key to the config file of my docker-mailserver install and it was all setup.

However you can do the same with any provider that gives you an option to act as your email relay, I remember both aws ses and sendgrid provide this service, but I’m sure there are more niche businesses providing this too.

[0](https://github.com/docker-mailserver/docker-mailserver)

>2- After that they’ll ask you a few questions on why you need it, just tell them its because you’re a growing startup who expects to send thousands of emails per month, (make sure to say this, they don’t crosscheck later, if you dont say something along the lines of this, they usually reject your application to avoid having to serve small customers who might not scale their business later. )

I have the same setup as you, relaying outbound mails through SES. I told exactly how I was going to use it and was accepted promptly. Maybe I just got lucky.

As others have mentioned, setting up SES is straightforward (as much as anything AWS) and for the rest of my setup I just use mailu.io containers. The config for setting up the relay is here: https://mailu.io/1.8/configuration.html?highlight=relay#mail...

I have had very few troubles sending outbound email directly, however there was one email provider that always rejected me because they were blocking all of DigitalOcean's IP space. This provider was quite niche, but it still bothered me.

My solution was to set up SMTP relaying based on the recipient domain. So nearly all my email can still be sent direct, but I have a list of domains that get routed through mailgun.com (or you could use SES or whatever).

More info here: https://github.com/docker-mailserver/docker-mailserver/issue...

> there was one email provider that always rejected me because they were blocking all of DigitalOcean's IP space.

This would be me. DO traffic is overwhelmingly hostile traffic. It's like Psychz or MCColo got massively scaled up.

OVH is a close 2nd. Amazon was a solid third, beginning in 2019. Not sure if they still are.

You work at Mega? Nice to meet you!

Is there anything I can do to get whitelisted? How can I contact you?

Not a huge deal if not, I've implemented the workaround already. But to be whitelisted after a chance meeting on HN would be a nice way to finish this story.

I've also blocked tens of thousands of ips from DO, and AWS on several web sites. Mostly for attempts at logging into wordpress admin accounts, some were spammers I'm guessing were using vpns that sometimes go through them. Those and tens of thousands from brasil, and several other countries.

There was a time when I looked at sending in reports -

and a time when I asked someone in the wp plugin directory who had a detector-like plugin to have it spit out a chunk of fields that would be ready to fill in the amazon complaint form and to do a cidr lookup to port over to iptables.. but that never got made.

This was all made worse when maxmind went registration needed and ruined the most effective security plugin for wordpress I'd been depending on for years.

I've noticed an increase in the microsoft ips I'm blocking these days to.

for now I don't mind doing an ip lookup when I can block 64,000 ips or more at a time I find it's a solid win.

My "favorite" story is a certain not to be named French university which was polite enough to inform me that they block all email from .org domains.

I always wondered why there isn’t a config option in MTAs that tries to route an email over different other MTA‘s submission port and tries until delivered. I mean a automated setup not static rules you mention. This would make self hosted setups so much easier. The chance that someone blocked your cloud server‘s IP address 4 years ago and never bothered removing it is high.

Do you run into any issues with your outgoing mail being marked as spam by the recipient? Particularly with GMail’s aggressive filtering?

I imagine 99% of outgoing Amazon SES traffic is transactional or marketing email for various online services. I worry this could make my personal emails look more like spam to the big providers. Or maybe it works out fine.

FWIW running a service who's job is to send email, I have found that GMail is one of the best providers to send to. They strongly value domain reputation and are popular enough that it was quick to build up a basic reputation. Now my messages are rarely flagged as spam even coming from a Digital Ocean IP address. I just rely on DKIM.

This is in contrast to Microsoft which seems to rely mostly on IP reputation and trying to send from a new server is incredibly difficult as they (reasonably) treat unknown Digital Ocean IPs as likely spam but (unreasonably) don't allow a good domain reputation to override that.

This is a bandaid that avoids solving the harder problem of trust/spam. It is such design patterns that make a fundamentally open/federated protocol more centralized, exacerbating the problem.

Personally, I think the use of proof-of-work like methods can mitigate the problem by a large extent, making it computationally expensive to spam users. This was one of the original goals of what has now become the "blockchain" revolution. Is anyone aware of any projects that are still implementing similar (open) systems?

I am happy to be proven wrong here (not an expert) but IMHO there is not much hope or solving the open decentralised communication problem with email at all. It seems that something like Matrix.org presents much more promise in this area. I also host my own Matrix server, but sadly not everyone I need to communicate with uses Matrix....

i get sweaty hands from reading this

Less FUD, please, don't discourage people just because you couldn't do it :-)

EDIT:

> When someone decides to run a persistent brute force attack from a botnet, eating up 100% of your CPU and you have no meaningful ways to block it.

postscreen? http://www.postfix.org/POSTSCREEN_README.html

BTW, there is soo much FUD in your comment, check http://www.postfix.org/ before claiming "someone will hack your email"

""" First of all, thank you for your interest in the Postfix project.

What is Postfix? It is Wietse Venema's mail server that started life at IBM research as an alternative to the widely-used Sendmail program. Now at Google, Wietse continues to support Postfix.

Postfix attempts to be fast, easy to administer, and secure. """

All of these don't seem that out the ordinary

If that was the case, we wouldn't even plug anything on the internet, ffs.

It sounds less like FUD than a change in perspective.

When I was in my twenties, I would have empathized with your point. I used to host my own web servers, but back then, my main priorities were curiosity, privacy and independence.

Not just that, I opened accounts for friends and family.

A decade later, I made all my hosting someone elses‘ problem, because I had different priorities.

There‘s nothing like the sound of a friend shouting in your ear because he trusted you with his mail address and he‘s running into weird errors. Or trying to get an important email delivered after a 10h crunch shift when you just want to bring your kids to bed instead.

I‘m thankful for all those learnings, but nowadays, I‘m old enough to just want mail to frickin work, that‘s why Google does it for me on a custom domain.

As I got further into my thirties, I became much more aware of the concept of opportunity cost: by deciding to do one thing, I'm by definition deciding to not do others. Running my own mail server is one task that has not made the cut for being more worthwhile than other priorities in my life.

Sad to see all the negativism. To anyone considering it, just run your own email infrastructure and don't let the naysayers put you down. Email is far too critical to let any megacorporation own you on it.

I spend approximately no time at all in maintaining my email infrastructure which I set up around ~10 years ago.

Of all the things I self-host and self-manage, my email server and related parts is the one which requires the least attention and ongoing work, by far. Set up postfix, it'll take some work initially, then it'll chug along forever.

I the webmail client experience better?

I was thinking of stopping to use gmail and hotmail, and run my own webmail client on a droplet. I don't like the idea of all my documents being tracked. Is there anything that competes with them that I could deploy.

Not really. Gmail is still one of the best webmail experiences IMO, so you have to prefer clients like Thunderbird if you want to go the self-hosted route. Some people really like RoundCube, but I only use it when it’s absolutely necessary. I did get some novelty out of using Squirrelmail as a teenager.

Roundcube has a built-in editor for Sieve scripts.

Sieve is pretty cool; it runs user-defined scripts server-side, on delivery to addressee's mailbox. The scripts are in the Sieve language, which is just for mail filtering. But it's a bit abstruse - it may be rudimentary, but most users don't want to tangle with a language at all.

Anyhow, the Roundcube UI includes natively a Sieve script editor made up of drop-downs, which makes it much clearer what filter-steps you're asking for. I'm rather minimalist; I prefeer Squirrelmail, if I have to use webmail. But this feature of Roundcube is really good.

Interesting. I did not know it was (mail) server side! I assumed that, like many PHP applications, those filters were just ran using a cron job that calls a PHP script periodically.

It runs server-side, on the delivery server. And the scripts are stored on the delivery server too.

This means that you don't have to synchronize your filters between mobile client A, laptop client B, webmail client C. Thy're all on the delivery server (which is usually the same as your IMAP server). The filtering happens before you log in to email, and before your client knows there is an email.

Sieve is a specification, not a product (and not some specific PHP script). It's part of the Dovecot package in the Debian distro. It's client-server, only because there is a protocol for a client to upload/update scripts. So there's a wire protocol for management. But it has nothing to do with retrieving mail. Sieve on the server only affects what happens on delivery.

I think Sieve is totally the sanest way to do per-user mail fitering.

Try https://www.rainloop.net/ it remembers the previous Gmail webmail interface. Pretty nice IMO.

OK I run several mail servers and I do have problems but of another kind that you describe. But this one is a bit ridiculous:

> When you need to explain to people why they can't send you that 100 MB video attachment which they sent to other people just fine but only your address is bouncing and why don't you fix your email already.

The maximum attachment size for Gmail is still conservative 25 MB and they basically dictate what is currently to be expected in terms of attachments going through.

https://support.google.com/a/answer/9050120

According to this page the incoming limit "depends on several factors" and can be as high as 150MB.

OK, so I tried to test it in practice, starting from 120 MB down to 25 MB. The largest that went through had 35 MB; the larger ones bounced with:

> Remote Server returned '552-5.2.3 Your message exceeded Google's message size limits. Please visit 552-5.2.3 https://support.google.com/mail/?p=MaxSizeError to view our size 552 5.2.3 guidelines.

That makes sense, because the attachment is going to be base64 encoded within the email and thus the size is inflated by 33%. This results in the size being roughly 47 MB in size.

You can easily search around that the Gmail limit for personal users is 50MiB, recently enlarged from 25 MiB. As the other poster says, even between Gmail accounts I still have trouble sending files larger than 30MiB.

For Apple’s iCloud mail it’s even just 20 MB.

These concerns are common across virtually any internet facing service.

Yes, if you choose to run a service it will need to be maintained, and occasional issues will come up.

No, email is an especially brutal service. I had taken this path for 10+ years.

If my custom media server or private photo site setup fails, it is not a big deal. But if I can’t login to a shopping site or my family can’t checkin to a flight because the two-factor auth email disappeared in to thin air, I am the “horrible IT person” who spoiled Christmas - end of story.

This 100%, e-mail is too critical and is not core to what my skills are, so I outsource it, even if I think it would be fun setting up initially.

What makes it particularly brutal, to me, is that the failures are typically silent. If GMail starts spamboxing my emails, I don't get feedback about that. If I'm not getting emails, I don't get feedback about that.

You can script checking those, but you'll have to re-implement it for each provider. And there are some that you can't; if I'm applying for a job, they're not going to give me an email account to test whether my stuff is deliverable.

HTTP services are absurdly easy to monitor in comparison.

I’ve tried testing gmail and it’s very difficult to even get consistent results

In my experience, the brutality grows rapidly with the number of users. If it's just for you, and you have at least one alternative account, then it's not brutal at all.

A mailserver administrator is a sysadmin. Being a sysadmin is on the face of it unrewarding - nobody pats you on the back when everything is working normally. They only call you when there's a problem (and it's usually urgent, and sometimes critical, and they'd like to know who to blame). So if you run a mailserver with users, it starts to become a people job, and things begin to matter.

People certainly rely on email to a greater extent than other services - even mobile. I can order goods online without surrendering my mobile number, but I always have to provide an email address before I can complete the order.

But I've still enjoyed administering mailservers. Perhaps I liked the slight paranoia induced by trying to reconcile the boss's demands for functionality with his demands for security.

Seriously?

If other email providers are blocking you, smarthost through an email provider.

If you're getting brute forced, learn how to set up and run blocklistd or fail2ban.

Not getting 100 meg attachments is an issue that other email providers have, not people who run their own servers. If your server doesn't have any free disk space, that's on you. If it does, then set confMAX_MESSAGE_SIZE to whatever you want.

If by "standard X" you're talking about SPF or DKIM, there are lots of tutorials.

If your email software is vulnerable because of issues with your distro, you're doing things wrong.

The point is if you can't, don't. If you don't want to think about issues like these, then you shouldn't be running servers, anyway, so you're definitely not the target audience.

If you can, then these things aren't issues.

I get it, if you're genuinely interested in hosting email, it's fairly easy... but those who aren't will lose interest after a couple of major setbacks and that's that. E-mail is, whether you like it or not, pretty critical, so it's not the best place for hobbyists to start.

Isn't the big/only issue regarding getting into inbox? I would imagine if warming / domain warming would be ridiculously difficult in modern times.

That's true, but to do it securely isn't as simple as one would like to believe. Setting up "DKIM/SPF/etc." simply provides auth/verify security on send, but I'm more specifically referring to the receiver side security.

If you're really properly securing your mail server, it would likely be isolated behind a firewall and only have a LAN ip of some kind and utilize UUCP for transport to another LAN machine that does not have WAN access, and then, only allow POP3/IMAP access to machines in the LAN or connected to the LAN via VPN tunnel. Finally, you would want to setup a backup system of some kind for this machine to periodically backup via rsync when the inotify/fswatch file modification triggers.

Next, you'd have a separate SMTP machine. For things like critical deliverability, you can't rely on SMTP to 'retry' albeit it's how they are supposed to act, so it would make sense to have multiple SMTP machines across multiple different backbones in different physical locations with backup power and the like with different MX priorities set.

The initial configuration and running a mail server are incredibly easy.

It's running it securely that increases the difficulty on order of magnitude (because you essentially have to setup a proper security protocol across multiple machines (a network) - defense in depth).

That said, it's easily doable if you're already running complex infrastructure. Hopefully, you're getting paid for your time and costs for doing so.

If not, then I hope you need to rely upon the protection of needing a home-invasion warrant vs a simple-subpoena since a machine at your home can't really just be 'subpoenad' while a machine at some datacenter business can. This of course assumes you're even running the machine at home because if you're doing all this on some VM the value of doing so diminishes ever so quickly.

EDIT: Just to be clear, you can't simply rely on fail2ban and some other on-machine script / snort / daemon / kernel feature to protect you. There are bugs in software/systems and 0days are very real (as well as the market places for them).

> If other email providers are blocking you, smarthost through an email provider.

Can someone please explain what this means?

Your SMTP server uses an authenticated relay through (let's say) sendgrid that you pay for. Sendgrid delivers your mail using their reputation.

It means no longer hosting your own server for sending email, but rather use eg smtp.gmail.com to send outgoing mail. It would mean allowing Gmail.com to send email from your domain via SPF, thus allowing all with a Gmail account to spoof your from domain/header.

Ed: as sibling correctly notes, "smart hosting" specifically referes to setting your smtp server to relay via another (eg: Gmail) - eg exim or postfix allow setting a "smart host" so that rather than looking up the receiving smtp server, all mail is shunted to the smarthost to figure out.

Added: you might get good delivery to Gmail using Gmail smarthost (requiring you to have a Gmail account) - but you might need to use outlook.com as a smarthost to get good delivery to o365 accounts... Etc.

Google requires that you verify ownership of a domain before you send emails as it with Gmail.

Ah, indeed it does.

They also require a paid account for this now. Though, I have one grandfathered in from like 15 years ago.

It means that "hosting" your own email really means hosting your inbox, and only your inbox. Because "the bigs" have basically monopolized sending email, you will tire of working through everyone blocking email from your 2-bit server. You will cave, and eventually use one of those "bigs" to do the sending half of "hosting your own email." Which means they will "see" and index all your email anyway, which WAS THE POINT most of us were TRYING TO PREVENT IN THE FIRST PLACE. I gave up about 8 years ago, and gave everything to Apple.

> fail2ban

As a sidenote: The two RCEs this year were enough for me to judge the quality of this software.

If a whois entry of the attacker's IP/domain can RCE your intrusion blocking software, I mean...really?

> If you're getting brute forced, learn how to set up and run blocklistd or fail2ban.

What about updating OS/packages/CVE when on holiday? Note that many CVEs are usually sent only to top-tier providers.

Well, then I must do something wrong. I run my personal email server based on Debian stable on my own SBC hardware in a colo for a couple of years now . Manual maintenance is just upgrade to the next stable Debian version every 3 years or so. I had zero of the problems you describe.

Yeah, me too. My life was basically set up and forget. Except for that time I upgraded my instance and the upgrade came with an unexpected and unwanted IP change. Unfortunately it seemed like the new IP has been used for spam and I had to spend some time clearing my IP status. But eventually everyone understood and my IP is clear and shiny now.

100% this. I did it for years. I think I’d rather stab myself than do it again.

Unfortunate side effect of avoiding self-hosted email is that email is becoming centralized service.

if you’ve ever tried running your own email service you’d realize it might as well be already

Just stick with Gmail then @avian, it’s not for everyone. I think that you might be constantly think of worst cases, where as most of the time it’ll just be fine? And even I it isn’t, then I’m learning more about the web and email. When did people stop doing something because it was challenging?

True user-controlled e-mail, the kind without third parties, contemplated in the original e-mail RFCs, is a "two-sided equation". Both sides need to run their own SMTP.^1

1. I have tested both sides running their own SMTP on small, i.e., less than 100, peer-to-peer overlay network and it works. "OTT email". Interesting question for the reader is how does spam enter this system. If the spammer is not a peer on the network, then they cannot send mail to the other peers.

If only one sides runs their own SMTP, that only solves one side of the equation. Almost every HN discussion of user-controlled e-mail focuses only on users controlling one side, while ignoring the other and leaving it to third parties. That can still have benefits such as not storing mail with a third party, but obviously only focusing on one side, e.g., the receiving side, will fall short of true "user-controlled e-mail".

User-controlled e-mail is a solvable problem for most users, i.e., those whose contacts in a given context, e.g., personal, school, or work, are under 100. User-controlled e-mail is an unsolvable problem for people who want to send e-mail to 100s of people, e.g., people they do not know, or people who want to recieve e-mail from any random person/organisation, treating their address like a phone number on a bathroom wall. Maybe, for most users, that is actually a good thing. (Given the attitudes toward "spam", it appears most users generally do not appreciate unsolicited mail.) There could be separate systems for unsolicited mail. We already have such systems in place.

> It's not the initial setup. It's the maintenance over the years that really makes you question the universe,

Initial setup of my mail server and related bits took a few days. Ongoing maintenance over the years? None, basically.

> It was easier than I thought to create a mail server that works as well as Gmail’s

No it isn't and no you didn't.

The article doesn't even cover basic stuff like email rules and spam filtering (incl. tuning and spam learning). It doesn't "look after itself" like the author wanted (article doesn't mention any update strategy). The author acknowledges that email servers are "open to attack" but this setup doesn't seem to include any security improvements over traditional setups. In fact, maintaining this looks harder due to the amount of custom scripts and lack of good documentation.

And of course it doesn't cover any of the things that actually make Gmail special like labels, having a consistent set of apps for web and mobile, push notifications (esp. on iOS), really good spam filtering, really good search (incl. OCR for attachments), high availability, image proxying, smart suggestions, datacenter security, Google doing code and infrastructure audits all the time, using reproducible builds, ...

It's great that the author is experimenting and learning, but if I had any private data hosted by the author, I would be worried now.

Gmails spam filtering isn't exactly a high bar

Based on my experience running mail servers in the past (both personal and corporate), I'd say you're wrong.

I second this.

Gmail spam filtering is top notch. I just stopped to care to obfuscate or hide my email adress (which I use since the beta invitation program of gmail) and I can count the spam I actually read in a year with one hand.

"I can count the spam I actually read in a year with one hand."

This is partly because Gmail is good at classifying emails as spam/ham.

But it's partly because it's more tolerant of false positives (ham sent to the spam folder) than you or I would be if we were tweaking our own spam filter.

I occasionally check my spam folder, and there are usually some mailing list emails that I don't care about, but which I did actually subscribe to, and would have wanted to reach my inbox.

> Gmail is good at classifying emails as spam/ham.

I wish they'd apply that discrimination to their SMTP output.

Gmail's spam filtering has a high false positive rate.

It classifies Stripe's and PayPal's important security emails as spam; I posted previously on HN:

https://news.ycombinator.com/item?id=19536465

It's easy to bring down the number of false negatives if you allow the number of false positives to be arbitrarily large.

On my GSuite business email, I've had > 50 incoming business-relevant emails this year that were incorrectly classified as spam. My personal self-hosted email server [1] lets through a bit more spam than Gmail, but it also doesn't suffer this big false-positive rate.

[1]: https://nh2.me/recent/Running-your-own-mailserver.pdf

Based on my experience running mail servers for a long time and today, I'd say OP is right.

gmail spam filtering is terrible. On my various gmail accounts, I both get spam and the good email goes to the spam folder. And there's nothing you can do, you can mark it not-spam a thousand times and it's still a crapshoot.

I don't have any of those problems with my self-hosted email.

Seriously? I actually think gmail's spam filtering is brilliant - I probably average less than a single spam email a year that it doesn't catch.

Contrast that with every corporate email spam filter I've ever been subject to, which vary from "shit" to "OK", and Gmail is completely in another league.

My problem with Gmail is the false positives. (Or is it negatives?) They routinely send too much to the spam box and others tell me they have the same experience.

The worst is when they take email from one Google hosted domain and send it to spam in another Google hosted domain, even though the email didn't leave their network at all.

Still, I agree that the overall level is pretty good and hard to duplicate.

> even though the email didn't leave their network at all.

FYI gmail treats all of its children equally. Mail from one Google user to another is subject to the exact same treatment as mail received via SMTP (and, indeed, Gmail sends traffic to itself over SMTP). If you study the headers of messages in Gmail, you can form a picture of how they allocate and use the virtual IPs.

I get them every day.

https://imgur.com/a/wXCocLd

Have not had to deal with spam on my personal Gmail address in the 10 years I've been using it, and I'm having the same experience running a big Workspace organization. Their spam/fishing detection is making my job a lot easier.

Gmails spam filtering is still the best I've seen from the major e-mail providers, so I disagree with your assessment.

I also have serious doubts about Google's spam fighting. While they catch a lot of spam in the spam folder, they are simultaneously overzealous, catching normal emails that I receive and read regularly, and underprepared, as if putting myusername@aol.com and sending the email to Gmail servers isn't totally obvious spam.

I second that, my worst problem is spam filtering. The rest I have set up, except DKIM and DMARC which are not worth bothering with.

For me personally, one of the most effective means of knocking out the first 95% of spam was using the S25R regex methodology [1] created by Asami Hideo which seems to keep the load on SpamAssassin and ClamAV really low. I've had to adjust the regex rules over the years a little bit but it's really low maintenance for my setup. There are also lists of IP addresses and networks you can block that are known to be malicious which also reduces the load and log volume. [2]

[1] - http://www.gabacho-net.jp/en/anti-spam/anti-spam-system.html [No HTTPS, Sorry]

[2] - https://github.com/firehol/blocklist-ipsets.git

Thanks a lot for this ! I’ll try them out too

In my experience spamassassin works wonderfully. There are some few false negatives (1-2 mails per week), but I did not have a single false positive which is very important for me. For example Google is much worse in that regard, which forces me to check spam every few days to ensure no legitimate mail ends up in spam, so it's like no spam filtering at all, I have to read it all anyway.

> if I had any private data hosted by the author, I would be worried now.

Merry Christmas to you as well.

Such negativity for just showing something I knocked up in half an hour. - something that I thought might be helpful, with experiences on how to make it more Gmail like.

Attacking the writing is fine, but insinuating my custody of private data is at question is pretty shitty

> Such negativity for just showing something I knocked up in half an hour. - something that I thought might be helpful, with experiences on how to make it more Gmail like.

GP's feedback is direct but quite right imo. I trust the author had only best intentions in mind but "Knocking something out in half an hour" and sharing, but good privacy and security engineering requires probably much more time. Quite frankly, the wording of the article can be insulting even for folks that are working on that problem professionally for several years.

Were it presented differently, it would get different feedback I'm sure. More like "hey HN, i made the first three steps what would be next?" -- i.e. efforts towards trying to understand the problem better.

It's not negativity. You wrote an article showing you clearly don't understand at all what are the stakes or what you are doing.

What you did is a basic setup which was covered in O'Reilly's TCP/IP book back in 1996. World has changed since.

Please learn from the community here.

Docker, cloud volumes, SpamAssassin, Dovecot, ClamAV, fail2ban, DKIM, DMARC. Ask what these are to someone in 1996 and see what you’d get back. The article covered setting all of these up

However my main objection to the OC was attacks on my professionalism. Unless you’re going to defend that, I don’t really care

I think the author and submitter got exactly what they asked for by posting something to HN that is by their own admission low-effort as it took only 30 minutes to knock it out. Many of the commenters call this out, since in their opinion the content does not hold up to what the headline promises.

Also, one aspect of professionalism is also to be thankful for the feedback rather than trying to interpret it as attacks.

I am the author.

> if I had any private data hosted by the author, I would be worried now.

Is an attack on me personally, nothing about the article. The article also took much longer than 30 minutes to 'knock out' - more like 3 hours all in all

I didn't mean to insult you. I think it's great if you're experimenting and I fully support that. It's just that the headline set high expectations and the article reads like this is being used in production, which I would strongly advise against.

> that actually make Gmail special like labels

I hate labels.

At $WORK we use Gmail and I get a lot of automated stuff (cron, etc). I want these types of message to go into folders. I don't want it in my "all" / archive area because they just clutter up searching for other things.

Perhaps labels work for other people / general public, but for me 'traditional' folders is how things work best.

> I hate labels.

Serioulys! gmail labels are a very poor mis-implementation of folders that just make a mess of sorting email.

> basic stuff like email rules

I mean gmail has the most limited frustrating filtering (lack thereof) rules of any email system I've used. Any self-hosted solution will be infinitely better.

> Gmail special like labels

How is that special?

Also, gmail spam filtering is not very good. You know how every business has that "check your spam folder" bit? Because gmail is so terrible about it. It is easy to do much better with a self-hosted solution, put an end to the false positives of gmail.

Seems that you like gmail, but in my experience it's one of the worst mainstream email implementations ever. Doing better is a trivial bar.

The nicest email stack is: postfix, dovecot, rspamd and rainloop.

EDIT: go check it out :-) https://www.rainloop.net/

EDIT 2: I don't understand why other comments are so agressive against the author for sharing how he runs his own mail server, I'm not sure if it comes from one's frustration, failures, unreasonable expectations about email, but I noticed that everything related to servers or email receives this hate (here on HN, eh?). Come on, let's start a new year where we appreciate someone sharing their experience in running a mail server :-)

Happy Holidays!

If youre not on the cloud (and even there on AWS and maybe Google Cloud…Azure is ok in a pinch… and iCloud everything if you also overlap with the Mac crowd) are you even an engineer, seems to be one popular strain of HN thought, which comes out particularly aggressively against e-mail servers because they are arguably the worst type of server to run on your own.

Yes, and doesn't makes sense, I don't know, I run my own email server for more than 10 years and my experience has been "setup and forget".

I don't understand why so much frustration coming against owning your own stuff.

I think it's because many of us here also did run our own email servers at some point in time, until realization came how hard it is to implement and support all features one gets instantly and effortlessly with gmail and such

"I don't understand why other comments are so agressive against the author for sharing how he runs his own mail server"

The author has been running his own mail server for less than half a week.

There's no suggestion in the post that his setup is robust or 'Gmail-like', as claimed in the title.

The email inbox with Archive/ All mail does work like gmail - I don’t use the UI. The filters are something I’m looking into

Here is a true statement: "I speak English and have black hair, like Keanu Reeves. I'm looking into growing a beard."

If, based on the above, I were to tell people I looked like Keanu Reeves, would you consider that a reasonable claim?

BTW I'm not deriding your efforts. I'm just saying there's a big gap between 'setting up my first email server with webmail and IMAP access' and 'setting up something with the features and reliability of Gmail'.

Thanks and cheers for being nice! Happy holidays to you too - I’m not really that surprised how negative the comments are here, including attacks on me personally it feels like.

If they don’t like it, stay with Gmail, I don’t care. I would just rather live in a world where the internet isn’t controlled by 2 or 3 big companies. Hacking a server for email and making it work like gmail was the aim, and I did it in less than an hour. Some people on here are pissed that I didn’t consider every eventuality, and filtering, and spam and this and that. Fine, but attacks on me as a person reflect more on who you are as a person.

If you don’t like how I wrote or setup the server, do one and make one yourself - or just stay with Gmail

dang must have the day off or something. The number of rulebreaking "shallow dismissals" in the comments is staggering. Hope you don't let it get to you! This community often thinks it's the center of the internet in my experience, and by proxy any mistake that happens is some sort of crime against the internet that you should pay for.

Cheers, I haven’t - I’m old & ugly enough to read them and let it not affect me. A few have genuinely good points, and I’ve replied to those. Some are just annoyed i didn’t setup a 6 node Kube cluster with 24/7 AI intrusion bot detection with a brand new UI with OCR text detection for image uploads.

Experimenting on the internet is what i did, and it’s what I’ll continue to do, despite what other dullards might say is a waste of time

I recently got into running my own mail server on my NixOS instance using[0]. The server has a total of 1.5 GB RAM and 10 GB of disk space, but it was sufficient to get 10/10 on mail tester[1]. Here's my 12 line mailserver config[2]. It was quite liberating once everything was set up, because then you know you are in full control of your communications.

It was more annoying to set up DNS than the mailserver itself, is there a good way to automate that as well?

[0] https://gitlab.com/simple-nixos-mailserver/nixos-mailserver

[1] https://www.mail-tester.com/

[2] https://github.com/siraben/dotfiles/blob/master/server/mails...

Terraform has modules for many popular DNS providers. But that's another tool with its own state to maintain. I've used terraform for both Route53 and Cloudflare.

siraben.dev doesn't seem to be registered anywhere so I don't know if there's one for your provider.

It’s registered via Google DNS. Looks like terraform can manage the records for it. Thanks!

According to many of the other posts here, running your own email server does not put you in"full control of your communications".

The reason that motivated me to run a mailserver is best described as an artisanal choice, not a practical one[0]. After all, I'm running this on a constrained system and certainly don't have ambitious to scale this to corporate scales.

Like some other commentors here the point is mostly to learn and have something semi-useful at the same time (I've had some pleasant exchanges over my own email already.)

[0] https://utcc.utoronto.ca/~cks/space/blog/sysadmin/EmailServe...

Once your outbound mail leaves your server, it's destiny is out of your control. And you don't have full transparency into the processes your inbound mail has been subjected to. That's in the nature of email. It's orthogonal to running your own server.

> The blog article is the setup to make Docker Mailserver act like a Gmail server.

I'm not sure what a Gmail server is. I was expecting this to include a web ui, admin ui, and the things that actually make Gmail hard to move away from. The docker-mailserver container doesn't seem to include something like that or am I just not seeing it?

The killer feature for Gmail has always been the spam protection and the fact that the emails I sent actually get delivered.

Does it have labels and full-text search, at least?

edit: does the described setup include a UI at all?

This is nice, but even though I've administered email servers for a quarter of a century, I haven't got the foggiest clue what makes an email server "Gmail-like". What does "Gmail server" mean?

I would think, if anything, that what Gmail has that typical email servers do not is somewhat decent webmail, but that can't be it because webmail isn't even mentioned.

Or is this another one of those instances where people use "Linux" to refer to all things Unix? I genuinely would like to know.

Seems like a way for the author to get clicks.

I was referring to Archive/All mail working, instead of just deleting all email, I should have been clearer I think - I don’t use the web UI so don't need a full replacement for that, even though it might be helpful

In my experience [1] running a small private mail server is very much doable, and a good learning experience.

[1] https://jschumacher.info/2021/05/running-a-private-mail-serv...

I think those projects:

- https://mailinabox.email/

- https://github.com/modoboa/modoboa

are better replacement. They are battery included with a webUI

There is also mailcow.

I've been using mailinabox for years now, and it is really good in the sense that it gets out of my way.

I've included it in my ansible setup, so the basis, distro, os updates, firewalls, backups are cositent with my other servers.

That took some effort: mailinabox is opnionated (and that is good. It is the main reason it works well and is secure), which can be a bit confronting if your opninions are very different.

So many negative comments, sheesh.

Many of us run our own small email servers quite successfully, even in 2021. Every time there's a post about it on HN, all these commenters come forward to say it's a fools errand, that it's nearly impossible, nobody should try it, anybody who says it's a good idea is a lying idiot, etc.

Sure, it's not for everyone and there are pitfalls that require effort and sometimes creative solutions to overcome. We should celebrate these projects like we do with other similarly challenging projects that get posted.

You can't replicate Gmail, but with Mailcow I've gotta say the whole process is pretty seamless. You can throw it onto a 5 euro VPS at Contabo, run docker-compose up and be done. Just regularly run the update and backup scripts to make sure you're up to date but that's it, really.

Exchange ActiveSync, multi domain + multi aliases with catchalls, (temporary) aliases, mail delivery rules, TLS requirements, you name it, all configurable in the web UI. There's even a built in DNS checking tool to verify that all the necessary records are set up right.

I concur, a very well crafted and maintained project.

I’ll take a look into this ta - the only thing really lacking in this setup is a UI for webmail or admin

I'm using mailcow for year and a half now, moved to it from Kolab. It's really great and painless to manage it. The only thing missing for me is the LDAP auth, something I got used to over the years with Kolab. The OP's solutions has built-in LDAP auth, so I'll give it a try.

If you just want your own address, iCloud+ now supports custom domains. You might already be subscribed to it and not know it. This also includes private relay and email hiding. It might be the easiest way to move your email out of gmail.

https://support.apple.com/guide/icloud/add-a-custom-domain-m...

What would be the benefit of moving from one mega billion dollar company’s server to another?

Important note - custom domains for iCloud is a very limited feature: - you can't send mails from more than 3 addresses per domain - it doesn't support catch-all

Since I am in a time of moving to other city to study on university, I decided to abandon my mail server and migrate to iCloud... so now I am moving every of my service@domain.tld to prefix+service@domain.tld (tagging system that doesn't parse properly on some sites). It's no fun, but at least I'll take off my head caring whether my server is on fire, as it's now Apple's issue.

I had to manage email infrastructure for years as part of my job and I really don't see how running your own email server can be a good idea for anyone. Setting it up superficially might be a quick and easy task but maintaining it stable takes hell of a lot of effort. I seriously cringe every time I see this type of guide and articles, it just makes me think that people who write them have zero experience running a mail server and have no idea what it takes to set up one that is secure and stable.

For majority of people best middle ground is to buy a cheap domain and a cheap cPanel/web hosting and just use that to host emails. You'll be done in 5min, it will cost you a cup of coffee and you won't have the headache maintaining anything other than passwords.

I'm using mailu.io with docker and haven't had any trouble for two years now. Cost me 1-2d in total.

No need for negativity. It takes very little effort to maintain a highly functioning self-hosted email infrastructure. I don't get these posts saying it's some impossible effort. Been doing it for a decade+, mostly not-doing since there's nothing to do, it just works perfectly.

AFAIK Digital Ocean blocks outbound connections to port 25. Has the author actually tried this setup?

Source: https://docs.digitalocean.com/support/why-is-smtp-blocked/

It works perfectly yes. Maybe it’s not fully blocked, or DO restrict the amount of traffic over :25

I run a production email server on Digital Ocean and have not experienced this issue.

An issue I have experienced is that one email provider (who provides a white label service so that small regional ISPs can include a free email account to their customers) has blocked anything coming from DO's IP block. Ultimately my solution is to route those emails (and only those emails) through mailgun.com. The other 99.9% of my outbound email gets delivered directly to the final email server with no issues.

> one email provider has blocked anything coming from DO's IP block.

Me. It is my consistent experience that traffic from DO's netblock rarely (actually never IRL) brings good tidings.

> Even on accounts where SMTP is available

I don't know what those accounts are but you shouldn't make statements that are only partially true.

You know, it's possible to build a house by yourself in about a day or two with no knowledge of carpentry. But I wouldn't want to live in it :)

incoming email is easy. Outgoing email is a whole different ball game. I tried setting up an SMTP server on digital ocean and found it to be impossible due to the fact that all Digital Ocean IP ranges are on various blacklists. I moved the server to AWS and was able to eventually get a running SMTP server, but it requires additional steps to gain the trust of AWS and outlook.com and other providers. It SHOULD be possible to set up an email server if you never never never never never send marketing email. If you DO send marketing email, then your stuff should be sent straight to the garbage bin and you should be on a blacklist. That's the whole point of spam.

I'll be interested in the one month update post.

What's a good way to monitor a self-hosted mail server? I can easily set up uptimerobot.com or similar and get alerted if my website fails, whether it's a DNS, IP, firewall, nginx, TLS^, application, or database issue. Is there a way to check my mail server and get alerted if it is not accepting emails for some reason?

^: uptimerobot.com specifically doesn't warn you if your site works but is using an expired certificate, be careful there

I use and recommend the prometheus blackbox_exporter. You can configure a TCP connectivity check with TLS validation and an SMTP "expect" transcript. If you ran this in the cloud from multiple probing regions/clouds you'd have a monitoring scheme on par with what Google uses to monitor Gmail.

Good article - seems comparing with Gmail upset some peeps but well done for having a go and trying to not just using off the shelf saas for every little things. Progress should have made it easier to host stuff ourselves not harder right?

Thanks! It works well for me, so people just are so negative - I didn’t really expect to upset their day so much… by ‘Gmail like’ I just meant Archive/All Mail working, not just deleting email

This feature is not exclusive to gmail, I have it set up with all my email providers.

It wasn’t the default for docker-mail server & had to work out from the wiki how to configure it to work like this

For docker and mail, also check out "mailcow dockerized". Lovely stack of software, been using it for 5 years with no problems.

This absolutely nothing like gmail. The great parts of gmail can’t be replicated at home anymore, unfortunately.

> The great parts of gmail can’t be replicated at home anymore, unfortunately.

Curious, what are those in your opinion?

Well interface is an easy one. All the available guis for email work like steaming garbage. Another few are mail deliverability SSO, and security. Back in my day, deliverability wasn’t an issue, security wasn’t an ever looming specter, and feature parity was mostly there.

I ended up throwing in the towel with Hey mail, and have really found love for email again.

Personally I find the Gmail UI to be unusable for serious work. The threading is awful (based on subject instead of headers and collapses everything into a linear sequence) quickly going through an inbox is painful because the split view is incredibly basic and the way it tracks read-status in threads is far too unreliable to find what you need to see. Plus every time you open a long thread you need to scroll forever to get to the new messages. It makes me understand why most of my coworkers don't read their email.

Let's say I'm tired of self-hosting my email (for all the reasons previously mentioned by others). What's a good option of privacy-conscious provider I can move my domains to?

I think ProtonMail (https://protonmail.com/) is the only one that comes close to being "privacy conscious," mainly because it's their entire focus. Any other non-major, non-free provider might be an alternative but they're probably not as focused on being proactive about the privacy element?

Now you have your own mail server. Great! But if you don't know how it works or if you don't have something that will help you maintain it, sooner or later it will break.

Isn’t that the same with all things tech? Why try anything new at all without first getting a degree in it?

I might get things wrong, so be it - I’ll use it to learn and be better next time.

You can deploy an email server almost instantly by getting a cheap vps with cPanel on it -- with everything you needed already configured, including spam filtering, security, etc.

Uhm, what about MailCow[0]? found it quite a while ago. Didn’t test it so far.

[0]: https://mailcow.email/

It opens with some fair points why somebody might not, but the main reason —perhaps on par with the constant security headaches— isn't there. You will never block spam as effectively as Google, Microsoft, FastMail, etc.

They see [for lack of a better word] infinite times more spam and ham than you'll ever be able to train your little Spam Assassin database, and millions of users to sort through it.

Email without spam control is not a pleasant experience.

This is a valid point - The setup I used integrates SpamAssassin for the basic setup which would get a fair number. There’s a way to learn from listening to when you tag a message as spam that I didn’t cover in my article. But yeah, you’d never do as well as spam protection as the big companies, but is that really such a deal breaker? Maybe if more people like us ran their own infra for email then we’d have better and stronger tools for spam protection?

> You will never block spam as effectively as Google, Microsoft, FastMail, etc.

Well I certainly do, more effectively. I don't do anything special, it's just that gmail isn't that good.

gmail gets a ton of false positives and to add insult to injury they also let spam through occasionally.

My self-hosted infrastructure lets spam through at about the same rate as gmail, but my false-positive rate is orders of magnitude superior.

Mail servers are all pretty much blacklisted by IP unless you work some kind of deal with someone somewhere.

Not really. By definition, a blacklist is a list of things to block, and things not found on the list get a pass. Most use IP address or domain. Unless someone has reported something, they don't get added. The only real exception I can think of is "residential IP addresses, " as some blacklists will try to keep up with multiple ISPs' residential assignments to block them. Even then, they still have to he added to the list to block. Most have forms easily accessible to request review or unblock. Even Spamhaus has a very easy process, and I find them to be rather more aggressive then the others.

Zoho has a 1 dollar a year plan for using with your own domain.

Zoho Mail has a free email hosting plan for up to 5 users (web and mobile/desktop app only, with no IMAP, POP, or ActiveSync access), but its paid plan (with IMAP, etc.) actually costs $1/user/month billed annually, which is $12/user/year. It's still less expensive than Google or Microsoft.

https://www.zoho.com/mail/zohomail-pricing.html

define gmail-like :)

Reply on news.ycombinator.com