Hacker News Reader

I'm surprise how long qmail lasted but it definitely has issues (some of them with patches, the patches definitely have much more flimsy security and realistically you can't run it without many feature patches these days)

I actually wrote a replacement for vpopmail called 'rmail' that stored the virtual user databases in cdb. I never liked requiring a central DB like MySQL for your mail receipt to work in real time in an otherwise relatively decentralised server protocol (despite literally working at MySQL for 9 years) and liked the idea of a file better. Powered 1000+ users for many many years. Now replaced with postfix for about 5 years :)

Still, it definitely has a surprisingly solid place in my historic heart.

Postfix has always (or, at least, as long as I can remember) used something very similar to that idea for all of its various user, alias, virtual, etc. tables. It's mostly transparent to the user, but when you run postmap, you're creating a key/value database (for those who don't know, cdb was djb's efficient key/value database implementation way back when) of the things in the map file.

It can even use cdb as a backend, but realistically for the kinds of map tables one works with in a mail server, any of them are Good Enough(tm) because the entire data set will always fit in memory (even if you have an absurdly large mail server with 100,000 mail users on a single system, it'll still fit in RAM).

We used to get a lot of people wanting MySQL support in our mail stack, which I actively have always discouraged. It's (orders of magnitude) slower, harder to maintain, and provides no benefit for the vast majority of deployments. For whatever reason (maybe just that fewer people want to host their own mail beyond sending mail from apps and notifications from the system itself), we get a lot fewer requests for it lately. Maybe I finally convinced everyone it was a bad idea.

I'm responsible for several mail servers and all are them are running postfix with the user data stored in SQL (including my own personal mail server). The primary reason is to simplify administration (management of user accounts and such, using various scripts and webapps).

There is an argument to be made for some sort of central backend store when you have multiple mail servers for the same domain(s) and users. SQL isn't terrible in that context (I would likely prefer LDAP for that, but SQL works, too).

Most of our users have one server, and lots of tutorials and stuff are about one mail server. It just makes no sense to add that huge amount of extra complexity for that simple use case.

The way we do it is the primary store is in SQL but I sync that data out to a file on demand

Yep, qmail was awesome back in the day -- especially all the virtual hosting stuff. It was used extensively at Yahoo! for quite a while with great success. I imagine they've probably long since moved on, though.

WRT OpenSMTPD: I have used it and do use it on several machines, although just for locally generated mail (e.g. system e-mails to root) that I want delivered elsewhere. It's lightweight, easy as hell to configure, and just works(TM). Unless you're running a large mail environment or need lots of integrations with other software (e.g. SpamAssassin, OpenDKIM, etc.) it'll probably serve your needs quite well.

As a hardcore qmail (and qmail-ldap) user back in the days, I can confirm that this is all accurate and correct.

... and yet writes netqmail entirely out of history.

* http://netqmail.org.

From the page you've linked: "Last modified: Wed Feb 2 23:37:31 EST 2005"

What aspect of the existence of netqmail disputes what I wrote?

The onus is upon you to explain where you have accounted for the existence of netqmail in your narrative, from which it is otherwise conspicuous by its entire absence.

You clearly went to its WWW site, but didn't even look at its changelog. Instead, you took the last modification date of the opening page of the WWW site itself and bizarrely tried to base some sort of argument, it is unclear what, upon that.

When it comes to discussion of qmail and patches, netqmail is a fairly important part of the history. Hint: One of its co-authors literally wrote the book on qmail. (And indeed updated that book to describe netqmail.)

20071130 version: netqmail 1.06

2007.

FWIW, I still use qmail. I wouldn't recommend it to someone who is setting up a new mail server -- then again, I generally wouldn't recommend that people set up mail servers -- but I've copied my setup from one system to another over the years and it's still working just fine.

I really hate that the general wisdom is, "don't run your own mail". Email is such a beautiful thing; anybody can talk directly to (almost) anybody else anywhere in the world, and it's been consistent and (mostly) compatible practically forever (much of what we call email was defined before it was even called "the internet").

And, that negative general wisdom is all because of abuse. Even if you figure out how to solve incoming spam, you might still have trouble sending mail to others because of their abuse prevention practices.

But, it's really not as bad as the general wisdom would indicate. Even with the particularly severe challenges we deal with on behalf of our customers (we mostly interact with them while they're initially setting up a new server on an IP of indeterminate history), we almost always get things to a state where delivery is reliable. DKIM and SPF, while a little complicated, seem to be helping a lot...the big providers are seemingly relying less and less on IP address in determining spamminess and more on history of the sender.

Anyway, I really hate that email is heavily consolidated into the hands of a very few very large corporations who have little respect for privacy.

We still have some old-timers using QMail, and we keep supporting it in a sort of "we won't break it" way because we like our old-timers and they mostly take care of themselves and sometimes even submit patches. There's nothing really wrong with QMail. It's just quite dated in a number of areas. So, I don't mind supporting it. It's not bad software, it's just old software.

> the big providers are seemingly relying less and less on IP address in determining spamminess...

It's not the big providers that you need to worry about. Setup SPF and most of them are fine. Some will look at your DKIM record too. The problem is every mid-level smtp service that has been configured by a varying number of people with varying levels of policies and ideas around how to mitigate spam. If your system triggers their spam detection then sometimes you get an email saying it was returned to sender and marked as spam. Sometimes, it will quietly go into their spam folder. I only ever got DMARC reports from Google.

One needs an IP registered to their own AS number owned by a real company to reliably deliver email to the entire web. You can't run an MTA on a VPS or most home internet connections without spending a lot of time monitoring and troubleshooting delivery. Using a 3rd party service for sending while hosting the MX at home is still the most secure and reliable setup. What would be really useful is a "bounce_forwarding_mta" where bounced emails would always attempt to redeliver through the 2nd MTA. Then you could send everything through your own MTA on your home internet connection and if one bounces then the system automatically attempts to send it through whatever smtp SaaS the user configures.

Things on the internet usually suck because of corporate policy, not usually because of the technology.

Yeah, I agree with most of what you say, though I disagree about needing your own AS; we've never had one and almost none of our clients have one.

Nonetheless, there's a lot of pressure toward using the big providers, and that's dangerous, IMHO. Also, the mail client landscape isn't great (lots of people are working on it, including my company, but a great Open Source webmail client comparable to GMail does not yet exist...and, the mobile client situation is probably even worse than the state of webmail).

Nonetheless, we have never outsourced our mail, and we deliver several thousand emails a day (forum and tracker notifications, mostly, but also expiration notices and billing failure notices) to thousands of different domains from our own MTAs. We are on static IPs in a colocation provider on our own physical server. We lived on the same IP for about five years and then moved to a new one a couple of years ago (same colocation). We only had one deliverability problem on the fresh IP, which was with Hotmail/Outlook, who seem to block every IP by default and you have to ask to be unblocked. But, once we jumped through their hoops we've never had deliverability problems since.

We may be dealing with less troublesome recipients. Our users are mostly Linux/UNIX users at small-ish companies. They probably care about people being able to send them mail reliably more than maybe bigger companies with layers of security theater to deal with.

I like the idea of a bounce forwarding MTA. That's actually a pretty brilliant approach to the problem; having something like Sendgrid or Mailjet or whatever as the backup would be a pretty neat thing (and for such a low volume of mail it'd be cheap or free for the small companies we deal with). Briefly googling didn't really turn up anything about that approach to bounces (googling anything related to bounces is tough because it can be about backscatter, my mail bouncing, my mail server bouncing incoming mail, delivery of bounces, etc.).

It's already possible to detect bounces in Postfix, but I dunno how one would turn the bounce message (which are infinitely varied in their contents) into a forward for the original email. Maybe just instant rejections would be handleable in such a case. That'd be a reasonable start on the problem.

And, in the case of email, corporate policy sucks because abuse sucks (and people are dumb about security practices). Because somebody is gonna click on that virus attachment if it finds its way through, IT is expected to act like the TSA and engage in all manner of security theater to prevent it, at any cost.

> there's a lot of pressure toward using the big providers, and that's dangerous

This is true and you mention this in other posts. The resounding answer to "how do I run my own mail server" is "don't" and that creates a dangerous environment where we abandon a previously working and useful decentralized part of the core internet because corporations do not accept our individual participation.

This is a form of censorship and I really don't know the best way to approach it.

If a message is signed with DKIM there is a statistically insignificant probability that it is not from the asserted sender. Every MX should block based on content and behavior after initially accepting based on sender validation (SPF/DKIM). It seems like we need an email advocacy group to fight the long fight and tighten the screws on individual companies with overly protective policies like comcast and att. Perhaps even working at a federal/eu policy level would make sense.

How do you deal with the spam issue?

SpamAssassin, greylisting with postgrey, DKIM/SPF checking in SpamAssassin (though DKIM is still configured to be somewhat forgiving).

My email address has not changed in 12+ years and is widely available on the internet, and I get maybe a half dozen spam messages that make it to my inbox each day. That's more than I'd like, and slightly more than my GMail address receives. But, it's tolerable. I could probably tweak it a bit more to prevent more spam, but I don't really have time to micromanage it...I just let the filters do their thing. If I enforced DKIM I'd probably successfully block nearly everything, but I'd get a lot of false positives, too.

A surprisingly large amount (like 5%) of the spam I get that successfully makes it through is not mass-email. It's people who have acquired a list of Y Combinator company founders and are sending out semi-personalized mail. A surprising amount of my spam mail name drops Y Combinator (and tries to imply they're already working with other YC companies). Which is really the weirdest damned thing, but it happens a couple times a week. But, I don't know that any spam filter could stop that. I get the same kind of thing (not YC related, usually, but things like recruiter emails) on my GMail account which is what I use in some developer communities.

For sending or receiving?

How do you avoid receiving spam messages?

I can't claim to be an expert in this field because I gave up years ago. Back when I was actively trying to prevent spam from entering my mail servers I would use:

* Fail if no SPF record

* Fail if DKIM record present in NS & not-present in message

* blacklists (DNSBL)

* heuristics (SpamAssassin)

Back then, not everyone published both SPF and DKIM. I would only fail a message if it did have a DKIM record and the message did not have the matching DKIM header. If I recall correctly, I failed everything that didn't publish an SPF record.

Blacklists are tricky because they are outside of your control. You are relying on a community of folks to tell you what is best. They are still extremely useful but use them with caution.

Heuristics need to plug in to your mail clients' "Mark as Spam" feature. There has to be a feedback mechanism.

There are a number of other configuration settings that control what can and cannot get through like requiring reverse DNS to accurately resolve sender ip = domain name.

It is a wicked problem and a game of whack-a-mole. I found it better to pay dedicated experts to handle it.

I recently received via email an update for a patch I wrote for qmail over a decade ago. I'd forgotten I'd even written the patch and was shocked anyone was even still using qmail.

+1 for OpenSMTPD.