Personal Mail Server on OpenBSD (2019) | Hacker News Reader

Personal Mail Server on OpenBSD (2019)

nicolascarpi.github.io

Love the shoutout to openbsd.amsterdam. It is pretty impressive how they're running a hosting company while committed to the OpenBSD stack[1].

[1]: https://openbsd.amsterdam/setup.html

Its quite a nice service. Host my own mail server quite happily on it. I don't have any anti incoming spam setup which is getting a little annoying though.

Greylisting helps a lot.

I ran OpenBSD on buyvm.net no issues. Never tried sending email from it so don't know what their IP reputation is like.

I love BuyVM. But they host almost everything (as long it's not illegal).

Being a neighbor to Tor exit nodes makes mail servers complicated.

You can use the blacklist check tool at https://mxtoolbox.com/SuperTool.aspx to check your IP's reputation.

Bad actors frequently use BuyVM's services, also known as FranTech and Ponynet. Their popularity with Tor operators probably doesn't help how their network traffic is perceived either. As a result, organizations may handle traffic from their ASNs unfavorably.

I don't have a personal opinion about the company; however, email is important enough that it's worth considering edge cases and future scenarios.

I run my own email server and get nearly zero spam due to the simple reverse DNS check which catch most of them, rspamd catches the rest. I have about 50 000 emails in daily with about 500 delivered and in those 500 about 20 which are spam.

Also, to remove the problem of false positive, refuse spam at the SMTP connection level and do not use a spam folder. This way users will get an MTA email in case of false positive.

> get nearly zero spam due to the simple reverse DNS check which catch most of them

On my Fastmail account, 99% of spam comes from a gmail address. I imagine there's a lot more that are filtered upstream with a similar DNS check, but my point is I receive most spam from legitimate addresses on the world's largest email service.

I feel like 5-10 years ago most of the spam I got was from Yahoo accounts, and sometime in the past few years that's switched over to being mostly Gmail. I used to think that Google was doing something to make it harder for spammers to mass register accounts, but if that's was ever the case, it doesn't seem to be anymore.

As more mail servers use domain reputation to classify (accept/reject) incoming deliveries (helped with dmarc-like policies), messages from gmail.com will get a harder time getting delivered to mail servers they also try to deliver spam to.

This provides natural pressure for gmail to reduce their outgoing spam rate. I'm sure they do try to get outgoing spam down. Spammers are just a resourceful bunch...

Any large free email provider will have these problems. It's a reason to use your own domain for email, with your own reputation, instead of sharing your reputation with the whole world, including spammers.

That's weird, I get zero spam from gmail and my domain is from 2000 so my email addresses are everywhere with catchall on the domain.

I am not doubting you, but did you check that the email was actually coming from google servers?

This is exactly what I do, and for exactly the same reasons.

I don't do any kind of content based filtering in part because some of my users do anti-spam, anti-phishing work, so of course they need to be able to talk about spam and phishing, and they need to forward along spam / phishing, without worrying about filters.

Also, every email, accepted or not, has specific reasons for what happens to them, not a vague set of rules that nobody knows like Gmail has.

I run email for myself with OpenBSD. The only spam mitigation is spamd(8) in blacklist mode, using nixspam. DKIM isn't checked for inbound mail. Outbound email is signed using the opensmtpd-filter-dkimsign package. There's some spam that makes it through, but not enough to take additional measures.

I also skipped using IMAP or POP3. There's a mail server with global IP addresses, that forwards inbound mail to my local workstation over WireGuard. My email clients read mail directly from /var/mail. Remote email access is via ssh terminal sessions. Not for everyone, but that's what I do.

Do you run a ssh terminal on phone to read email? I suppose it takes a few steps to attach a file from the phone.

I don't. A laptop is used for email when needed.

Should probably have a [2019] tag, as things do change through time.

I always enjoy the self-hosting explanations. Starting with mail is an interesting choice though. It's relatable to most people, but also very complex compared with a tougher DNS setup, DKIM, SPF, all that stuff.

I'm not sure what the right approach is to maintain good security, and then open up the right ports for simple services.

I do wish this article had talked about SPF.

SPF has challenges with shared infrastructure - if you are sending from a large service and using SPF then anyone else on that service and spoof you unless the service has outbound controls to restrict which addresses you can send from.

Fastmail had to implement this a few years ago ourselves, after 20 years of allowing whatever, we had to start by auto-whitelisting all the addresses people were sending from for a while, then slowly start introducing a requirement to prove control of the sending address to add new sending addresses over time! Obviously hosting your domain with us gets you auto-approved for any address on that domain, but otherwise you either need to confirm that you can receive email at an address to send from it now.

But SPF by itself is pretty flawed. I'm keen to write more about DKIM2 when it gets chartered at IETF (hopefully) and we can post more public documents, but it should supersede SPF/DKIM for most uses.

The ISPMail tutorial from workaround.org is the gold standard for "host your own email" and has been for years.

I set up an email server quite similar to this a long while ago like a year or so and it gets so much spam it's ridiculous haha. Really should have something in there like amavisd/spamassassain and scanning with clamd at the very least, because that's potentially sending attachment malware to your mail client of choice - unless you're happy with 100000 notifications with subject headers like 'BOOM OF SALES' or 'knock down trees with your hugec0ck'

I run a personal mail server on OpenBSD and I love it.

The one big problem I've run into is sending emails to mail servers running the Proof Point blocklist. They have my IP blocked, and there seems to be no way whatsoever to get it unblocked.

Maybe you need to have an enterprise account with them for them to even listen to you.

I, too, have issues specifically with recipients sitting behind Proofpoint setups. My IP isn't blocked per se, it's just not "trusted" because I don't send enough, so it's permanently stuck in "new untrusted sender" purgatory. I can't even return responses to e-mails that were sent to me from behind Proofpoint. At this point I consider Proofpoint a completely counterproductive piece of garbage product.

Why not use SendGrid/Postmark or some other delivery service and forget about using a random untrusted IP.

I prefer as few middlemen as possible for my personal communications, and not having to spend a hundred bucks a month on it.

Where did you get the idea of $100/mo? SendGrid has 100 emails per day free tier and PostMark has 100 free emails per month and basic plan is $15/mo.

If it doesn’t work without a middleman then it doesn’t make sense to run your own service. Any spammer can rent a server the way you can.

That's their cost for a dedicated "clean" transit. At $15/mo you're just sharing tenancy with everyone else, using their SMTP setup together with everyone else. I'm running my own setup for a handful of domains, my own configurations, signing procedures, filters, features, aliases and catchalls etc. Incidentally almost all spam I receive comes in via Gmail, Yahoo and Hotmail.

What matters is how inbox providers look at the source IP. At least delivery services have higher chance by filtering obvious outgoing spam than a rogue server IP.

My problem isn't the network I'm originating from. It's a) that Proofpoint doesn't track "state" (message-IDs) between outbound and returning e-mail, and b) that I don't send regularly enough to be a "familiar".

Would love to get Google and all the big companies' fingers out of my life, but spam is overwhelming today.

Would love to see a robust tutorial to show us how to really do spam protection right.

I wonder if anyone has tried training an LLM on known spam and measured it's performance? Such an LLM would ideally be run local to the mail server for maximum privacy.

I don’t know why that would be necessary. The vast majority of the spam I get is obviously spam from the subject line alone.

Ignoring e-mail content and throwing Naive-Bayes on the header alone is pretty much hove we got amazing spam filters about 15 years ago. All of course using a millionth or less of the resources a large language model would use.

Sir! The willies you just gave me have no compare.

What if said AI gains sentience, but trained on that data?!

rspamd has had an option for it for a while, but the older markov chain based filters tend to work well enough.

I'm on Debian stable, not OpenBSD, but SpamAssassin + razor + pyzor works really well. Roughly 1 spam per month, and 1-2 false positives a year. This is for an email address that has been used and openly spread widely for 25+ years.

The real work is making sure that outbound mail gets delivered, but even that is just making sure you have a clean IP and setting up reverse DNS + DMARC/SPF/DKIM...

Nice never heard of those until now. Link for anyone here cause it's kinda hard to google razor email filter for some reason. What does that setup have over amavisd?

https://notes.sagredo.eu/en/qmail-notes-185/razor2-pyzor-spa...

I investigated further and these don't really seem to be incredibly active projects, you sure this is the best solution?

https://github.com/toddr/Razor2-Client-Agent

https://github.com/SpamExperts/pyzor/tree/release-1-0-0

> you sure this is the best solution?

No, but I've been using it without issues for close to 25 years.

Love hearing when old code does just fine after that much years to be honest. Very impressive

To send an outbound email today you must have a special skill set and years of carefully built reputation. I just outsource it to smtp2go.

I understand and respect this opinion, but it is clearly not true that you need "years of carefully built reputation" as per my own write up in this thread and plenty of others here and elsewhere. Still, I do respect and understand that e-mail is a particularly nasty hole to dive into with potentially serious consequences so I do not look down on those that bow out and go for alternative solutions.

I've been running a private mail server since the early 00s, spam protection has actually improved drastically in the past ten years or so. For the most parts, SPF and dkim make it very easy for servers to identify scam, for everything else rspamd and clamd seem to take care of the rest.

If you don't want to run a completely custom setup, there's projects like mailcow out there that can do the heavy lifting for you.

I really don't see a quantitative or qualitative difference between the gmail experience and mine, with the caveat that my setup doesn't label ham from other private mail servers as spam (arguably a good thing)

The big thing is that you're presumably already established, which means your IP/ASN is clean and "warm".

I self hosted for several years and gave up because even with a clean ASN, I simply wasn't sending enough emails to keep my reputation score high enough, and so deliverability into the big players (Microsoft in particular) was very spotty.

Email isn't that hard it's just laborious to administrate.

> I simply wasn't sending enough emails to keep my reputation score high enough

I’ve used a smaller hosting company for over 25 years run by a competent admin and it’s now dying a slow death I believe exactly because of this reputation problem from infrequent outbound emails from my domain.

I don’t know what to do tbh because putting my fate in big tech seems super dangerous.

Anyway, everyone is worried about spam but the real problem is sending and having people at outlook.com and gmail.com actually receive your emails!

I've long been convinced that Big Tech wants email to go away because it's neither fashionable nor particularly profitable. Gmail was famously somebody's "10% project", after all, and not a real product initiative.

Now that the era of free money appears to be over I'd not be surprised if I was reading a blog post about an "incredible journey" at Gmail within the decade.

While I think that everyone hosting their own email is the ideal, it's not really feasible on today's Internet. I content myself with fastmail. They're big enough I'm not worried about them dying any time soon.

I tried hosting my own email server again earlier in the year. I’d forgotten the process so when googling around I found numerous YouTube videos of spammers doing this themselves …

Get a clean IP and start long form email threads between this new domain and personal Gmail / outlook accounts: checking ‘this is not spam’, and coherent responses.

They also mention getting DKIm and SPF working.

The need for separate caldav , and all the major cloud providers blocking port 25 bummed me out.

Even more amusing is when half your customers are in Gmail, the other half in Exchange, and Gmail and Exchange are having some snit so the emails ain't happening. You call up Microsoft and they want you to reboot (??) or login to some windows account (??), and good luck getting someone from Google on the line. Fear not, for outsourced email saves money, and increases productivity, or anyways something like that, and if you have sufficient faith those big old corporations will fix things, eventually, maybe.

I, for one, welcome our new AOL overlords.

I have no explanation for it, but I also run a tiny mail server and I'm always fascinated that despite extremely low volume I still manage to get through without being flagged or blocked.

Best I can guess is that my host's netblock just happens to be sparkling clean, but it sounds like even that may not be enough anymore

Similar story here; my only guess (which I don't want to verify in case it jinxes it) is that I've been on the same name and netblock for an extremely long time (~20 years) and so I'm grandfathered in to a lot of undocumented IP rules at the big houses. Long may it continue.

Hm, perhaps they do have a longer timeline, I've had my domain and one of the IPs for over 10 years

You don’t have to choose between big tech and self hosting though. There’s thousands of medium sized, sustainable businesses that host your email for money and provide human support on top.

One of the good ones would be Fastmail but there’s many more.

I dunno if there's thousands. Maybe if you include ISPs! There's certainly quite a few though.

(and thanks for the Fastmail plug)

As someone who managed email (postfix / dovecot) for the Engineering program at a university for over a decade - I would highly recommend that people not do this. At least not for email that's important to you. It's just too easy for something to go wrong and you won't notice until it's too late.

While I'm every bit qualified to run my own email service, I don't. I pay protonmail to do that for me these days and save myself a lot of time, effort and stress.

Same, I'm qualified to do it but turning it over to Microsoft for 8 bucks/mth for wife and I was no brainer. We get 50GB Mailboxes and working ActiveSync. Web Interface is much improved as well.

Any post about self-hosting email inevitably gets posts from people who think it's their place to tell others what not to do.

Replace "email" in what you wrote with "web". Is it just too easy for something to go wrong? Sure, for certain kinds of people. Everyone should just stop hosting servers altogether, if we're worried about things possibly going wrong.

People here, generally, aren't technically illiterate. We don't need you to tell us that because you're not comfortable doing something, we shouldn't.

Okay dude.. I'm offering feedback based on actual experience I have. You're free to ignore it if you want, but don't attack me for my opinions.

It's obvious when your website goes down. It's not obvious that emails aren't being delivered and you've missed something important that you'll never see.

I'm not attacking you. I just see too many people say to not do something, and think that's a silly thing to say to a technical crowd.

Also, I don't agree that it's not obvious when something goes down. You don't know how I or my users check my email. If anything, I hear from people much more quickly when there's an issue with email than web!

You can check your email, everything will show up fine (or at least what you think is everything).. that doesn't mean there's not some problem with email delivery that you don't know about.

To reply to something you said in your first comment, as someone with over a decade of experience it is my place to share my thoughts about this subject. If you want to ignore my thoughts go ahead, I could care less.. but I don't appreciate how you're acting like I'm doing something wrong.

That's called a lively discussion. What's your problem with that? People here are not technically illiterate, but we aren't smart either, are we? Most of the articles found here talk about issues found while working on or with XYZ. Why do you think that is? Because the people here make mistakes, too, or make wrong decisions. And hosting your own email requires lots of knowledge about server and email security, so a voice emphasizing that is just as much a contribution as a voice emphasizing that you can host your own email.

I’ve been running my own mail server since forever on a VPS (so, stable IP address, which helps). Still need a third-party spam filter as the primary MX to avoid the deluge, and my mail doesn’t always land reliably. But it’s kind of a point of pride or stubbornness.

What we need to have is a P2P email system with maximum privacy so that it will be maintained collectively and we don't need to use the BigTech systems.

I used to run my own mail server until dealing with spam became unmanageable, even with spam mitigation tools in place.

Maybe spamd is sufficient, but I ended up switching to gmail which was initially great but has dropped off in efficacy over the years.

I also understand sending email can be difficult with strict SPF rules in place causing many email providers to reject legitimate emails from smaller email servers.

Aren't there third party, for-hire, spam and malware filtering services? Have you tried them?

There are. So many of them have pricing models that become unaffordable quickly for personal users. I hosted email for 6 people in my immediate family and per-user pricing added up quickly. I used SpamHero for quite a while and it did a pretty good job, and $10/mo for a whole domain was worth it to me.

I've had quite fine experience with the FOSS solutions, not alot gets through and never seen a false positive which I can't even say for gmail

openbsd literally had the morris worm vulnerability up til very recently. not sure this is a great idea. the way it is (still) programmed is just asking for it.

https://blog.qualys.com/vulnerabilities-threat-research/2020...

The exploit was fixed in February 2020 almost 5 years ago. For how long after a security patch will a piece of software still be a bad idea?

Went through the same journey in November and now handle both inbound and outbound e-mail on my Hetzner box (should get a secondary SMTP going at some point for redundancy though). I have delivery to both Google and Microsoft (known to be among the worst of the actors) confirmed working for about a month.

My simpler (?) setup based on the same logical flow as Nico's:

1. Check assigned IP from provider (Hetzner in my case) for issues on black lists (MxToolbox worked great for me).

2. Set up reverse DNS with your provider.

3. Install OpenBSD: confirm default, confirm default, ..., enter hostname, enter username, enter password, confirm default, confirm default, ..., select mirror, confirm default, confirm default, ...

4. Use ssh-copy-id(1) to authorise key logins for the user you set up during the installation.

5. Set up DNS records for both the hostname and SPF (confirm propagation/settings with say MxToolbox, it will be helpful at pretty much every step, so I will stop repeating it now)

6. Enable httpd(8) with `rcctl enable httpd && rcctl start httpd` and set up acme-client(1) with the examples from: https://man.openbsd.org/acme-client

7. Enable and configure spamd(8) (note that I disable greylisting) and uncomment a few lines in `/etc/pf.conf` and reload your PF rules:

    > echo spamd_flags=-b >> /etc/rc.conf.local
    > rcctl start spamd
    > vi /etc/pf.conf
    > pfctl -f /etc/pf.conf

8. Configure and enable (`rcctl enable smtpd`) OpenSMTPD, which is about as easy as it gets (I am being more explicit about the hostname than I need to, but it is necessary as the box has multiple hostnames and the MX one is not the primary):

    pki $HOSTNAME cert "/etc/ssl/$HOSTNAME.fullchain.pem"
    pki $HOSTNAME key "/etc/ssl/private/$HOSTNAME.key"
    
    table aliases file:/etc/mail/aliases
    table domains file:/etc/mail/domains
    table secrets file:/etc/mail/secrets
    table virtuals file:/etc/mail/virtuals
    
    filter "rdns" phase connect match !rdns \
            disconnect "550 Reverse DNS lookup failed"
    filter "fcrdns" phase connect match !fcrdns \
            disconnect "550 Forward-confirmed reverse DNS failed"
    
    listen on all tls hostname $HOSTNAME pki $HOSTNAME \
            filter { "rdns" "fcrdns" }
    listen on all smtps port smtps hostname $HOSTNAME \
            pki $HOSTNAME auth <secrets> mask-src
    
    action "local" mbox alias <aliases>
    action "relay" relay tls helo $HOSTNAME
    action "virtual" mbox virtual <virtuals>

9. Fill in `/etc/mail/domains` and `/etc/mail/virtuals` with the domains and virtual inboxes you want to handle.

10. `rcctl start smtpd`

That is it. DKIM is annoying, as it requires a package from ports and all we just did above was with the OpenBSD base system, but it turns out delivery works just fine without it for a small family server (even for Google and M$). I think there is an argument for DKIM (although it adds next to nothing over SPF) to be in base OpenSMTPD as I believe all the complicated code is already in base, but I am not intimately familiar with the OpenSMTPD code base and trust that it will happen if the e-mail climate becomes even more oppressive.

I am cheating somewhat here as I am not doing local delivery/retrieval but relaying to an external SMTP server as I have yet to find a more minimal solution for POP3 than Dovecot that I feel comfortable hosting (pop3d gets close (https://github.com/snimmagadda/pop3d/), but I would need a code audit and I have lacked the time).

This is all from memory (apart from the nearly default `/etc/mail/smtpd.conf`) and I of course take no responsibility for anyone copying and pasting blindly without thinking (this is OpenBSD after all: Use your head).

A final word, if you ever have trouble, use MxToolbox or similar and they will lead you in the right direction if you have misunderstood the documentation or in other ways messed up the configuration.

Trick is to just use delivery service like SendGrid or PostMark and as long as your domain isn't brand new and the content doesn't smell phishy, you should have a decent chance. Don't try with a random IP that server vendors give you for $5/mo, which can never look too clean.

Look, I get it that many people have been burnt. In particular for marketing e-mails or services that rely on hundreds of e-mails an hour. But "Don't try with a random IP that server vendors give you for $5/mo, which can never look too clean." is far too much of a general statement as I know a fair share of people that have done exactly that and are getting their e-mails delivered and not silently dropped by lying SMTPDs from the big players.

Admittedly, my "random IP" (and box) with Hetzner comes with a cost of seven or so times per month compared to what you write. But I find it tiresome that we pretty much end up with two camps in every single e-mail thread. One that claims that self-hosting e-mail is dead and impossible in year $X and one that claims that it is not. Stating either of these absolutes is unhelpful as e-mail in year $X requires a lot of nuance.

I should end my comment by stating that I am not attempting to call you out as the worst offender and I appreciate you mentioning delivery services. Until Google started enforcing SPF or DKIM I had a nice working setup where I used a well-established, non-profit SMTP as my outgoing relay as I was scared to bits after reading so many comments about e-mail horrors on HN.

What’s the reject rate?

OpenBSD is unusable as a server. No checksumming filesystem. No even TRIM for SSD.

Which BSD flavor would you suggest for server work?

I love OpenBSD, have been a user (including professionally) for over 2 decades, and appreciate their stubbornness when it comes to security (including sacrificing performance), but for work for anything other than a layer 3/4 firewall I'd use FreeBSD (FreeBSD has an older, threaded version of PF, though). It's got ZFS for storage, a much more robust threading system (meaning modern multi-core processors will be better taken advantage of) and generally has broader support for hardware.

Pre-2005 running FreeBSD was a nice little "secret" that allowed you to run a rock-solid OS without drama or headaches. However, professionally nowadays I've accepted the fact that linux "won" and I don't want to deal with the headaches of finding people that can admin something niche, on top of so much tech tooling being designed on and around linux. Most of my work is done supporting docker containers in some way, so why fight even if it's possible to run docker on FreeBSD...

No docker is deal breaking these days but podman may be a better option for FreeBSD.

Depending on your SSD TRIM may not actually be required anymore. If you're running on a VM it's even less important as the disc is likely to be virtual anyway.

That being said the filesystem certainly is the weakest part of the OpenBSD and given the uptake in filesystem designs with ZFS, Btrfs and Bcachefs it is interesting to see that OpenBSD is left behind.

For a personal mail server, or even small business, it's not really an issue, you're likely not going to have terabytes of email.

It’s not about having tons of files but compression, snapshot and ease of management to cut out any size from a pool.

It has certainly worked very well as a server OS for me the past 20+ years. I agree that FFS on the whole is a very dated and sluggish file system, though anecdotally I've not once suffered a loss of data with it despite several power outages and sudden hardware deaths. It may have mattered that I rarely ever had "softdeps" enabled on my file systems.

Reply on news.ycombinator.com