Announcing Notqmail
schmonz.com
schmonz.com
- the config system mostly used the name of a directory where lots of software would use a filename (qmail.ini => /var/qmail/conf), the name of a file where other software would use a variable name, and the contents of the file as the value of the variable. This is really, really easy to read and write without writing a parsing library.
- separate executables do separate jobs, with well-documented interfaces between them. Oh, and if a part didn't need root, it didn't run as root, and if a part didn't need to be the same user as another, it didn't share users. While it was easiest to replace something at the edge of the graph, it was entirely possible to replace middle jobs as well.
- qmail was the original design home of Maildir. I approve of various Maildir++ implementations (nesting folders is useful) and you could add support for them by replacing your delivery agent.
- qmail expected to be controlled by daemontools, a supervision system that has inspired/been blatantly copied and improved upon as runit, perp, s6, and nosh. (I have a suspicion that nosh is what most critics of systemd is really looking for.)
- fast.
- nearly bug-free.
What qmail gave up in exchange for all this: modern features and required antifeatures... which, it appears, notqmail is going to implement. Huzzah!
this djb-ism has seriously affected a lot of the ways i unix, to date. envdir is such a great idea.
There were a lot of great things about qmail. I really liked the envdir and how secure it was. But eventually I found the attitude behind it very tiring and switched to Postfix, which is a really great program.
Any reason why you don't (can't?)?
qmail had a great impact on system design for many years and especially in the mail delivery area. Things like Maildir redefined how even MUAs store and use email. djbdns also provided a breath of fresh air at the time it was released, worked much faster and much more secure than something like bind.
DJB also had many other (likely even more impactful) roles in the software industry, leading the fight against the encryption software export restrictions of the 90s.
FYI, "I’ve learned more C, reduced build-time complexity, ..." probably isn't confidence-inspiring. I work with C a great deal and have for a very long time, and I though I'm very comfortable with C, I treat it with fear and respect -- I have found far too many security vulnerabilities just from staring at C source code to do anything else. Now, I too have "learned more C" over the years, but I wouldn't write a sentence like that as part of justification for some project written in C -- I would fear that others would wonder if I don't just know enough C to be dangerous. Perhaps one should not write "I've learned more C" as part of anything other than justification for moving to Rust.
Postfix is not alone in this; no other MTA/MDA can do what qmail can do.
However, the exhaustive list of things other MTAs can do that qmail can't would be huge (starting with backscatter-proof bounces for example)
qmail was limited back in the 90ies and it's horribly behind the times now.
I'm tempted to start auto-allowing new aliases and just having a list of names that can't be used (postmaster, webmaster) etc, that some services use for domain ownership verification.
As an aside I do hope qmail offers similar limiting abilities. It wouldn't be great if a user could make themselves receive webmaster@domain.tld emails!
If we were starting from scratch, I’d be first to say let’s pick something safer than C. But we’re starting from where DJB left off, so there’s not much left to decide about language. Our roadmap aims to provide mostly Unix-process extension points such that new code can be written in any language.
Postfix is great, and I certainly hold Viktor in high regard (haven’t met Wietse). It’s just that some of us really like qmail. :-)
If you also care about performance, I'd be skeptical of such claims.
For most sites, performance is a total non-issue for providing the feature-set that qmail provided.
And if you are a site that processes enough email for performance to start becoming an issue, you're probably better off with a custom solution that's tailored to your specific issues
These days the project is stalled, I think most of the development moved over to a nodejs based project.
Anyway the project was pretty simple to work with, and scaled wonderfully:
If you're building a competitor for Gmail or Hotmail, then maybe that performance difference will matter. For almost everyone else it's largely irrelevant.
EDIT: I'm confused by the downvotes for this. If someone has practical experience that suggests that performance is a real concern for this type of use, I'd love to hear about it. For my part I built and ran such a system for a couple of years, and spent a lot of time looking at the performance constraints. CPU usage just was not a factor even on ~20 year old (by now) hardware, running on 4-5 machines where the beefiest one had either 2 or 4 cores. All our optimization efforts consistently went on trading increased CPU usage for reducing IO use, because that is where the bottlenecks were. That's been reinforced with other mail systems I've run over the years, including large scale mail processing with PHP (not my choice) - every such system I've seen have been IO constrained rather than CPU constrained.
So, yeah, qmail is fine as a starting base, but there are important things that a modern MTA/MSA needs to have:
- MSAs need to support user authentication in
SUBMIT (no open relays)
- MTAs and MSAs need to support TLS
- MTAs should -really, need to- support DANE
for authenticating SMTP servers
I'm not sure that portability is such a problem that you can't use Rust. Rust runs on all modern OSes, which is all you need to support in a new MTA/MSA. And Rust is fairly easy to get into, actually.
For mandatory user-facing TLS and AUTH on port 587, and opportunistic server-to-server TLS on port 25, I've written https://schmonz.com/qmail/acceptutils. For SMTP recipient checks, I've written https://schmonz.com/qmail/rejectutils. It's too soon to say how notqmail will solve these problems, but they're solvable and we intend to solve them. For DANE... definitely not there yet.
qmail has had 1 CVE in 20 years, a local DoS. [1]
I trust djb to write correct code more than almost any other human on the planet. That includes all the non-crypto code found in qmail, daemontools, dnscache, ucspi-tcp, etc. If you’ve read it, you know how almost supernaturally careful and minimal it is.
Your points about mere mortals writing C are well taken though!
[0] https://www.cvedetails.com/vulnerability-list.php?vendor_id=...
[1] https://www.cvedetails.com/vulnerability-list.php?vendor_id=...
I find it hard to pin a "postfixadmin" CVE on postfix, are we going to blame vmailmgr bugs on qmail? :-) The BSDDB one I'm on the fence about. "postfix_groups.pl" is marked as disputed. Another seems to be clearly a Debian package issue.
CVE-2011-0411 is interesting because, IIRC, qmail had no SSL/TLS abilities, and the hacks you had to put into place to enable it were abominations and likely opened up issues of their own.
I’m not sure this qualitatively changes the argument. CVE-2011-1720 and CVE-2008-2936 are examples of postfix vulns no one has found, or reasonably expects to find, in djb code.
CVE-2011-1720 is related to SMTP authentication which qmail doesn't support out-of-the box. So. Yes you're right: You don't find that vulnerability in qmail because qmail doesn't have that feature.
Can you run an SMTP server without authentication these days? Not if you have users using you as a relay. So I guess you have to custom patch qmail and now you're back to not running djb code in the first place.
CVE-2008-2936 I grant you. Though once we are at doing hardlinks to symlinks, we might hit other edge-cases too, some might even affect qmail in some way.
Because if you disable a feature you don't use, you won't be affected by the vulnerability.
If you use smtp authentication, you're not running djb's code and all advantages of djb's code are moot. If you don't use smtp authentication then that CVE is inconsequential and you can't count it as a black mark against Postfix with regards to the security of your specific setup.
I disagree. It is a fair mark against postfix if (as I claim) most users really don't need it.
Here's my reasoning to that point:
I think most email end-users either use their ISPs mail settings (who know who they are and don't need authentication), or webmail (which doesn't use SMTP for relaying anyway), or outlook (which doesn't need SMTP at all). Fastmail is becoming more popular and it doesn't need SMTP either. The end-users who remain are either business users (who could use a VPN which is better than SMTP AUTH in every way), and nutters with vanity domains.
If that's contended, and note I do believe there are probably end users who are actually still manually entering SMTP AUTH settings into email clients, but I would struggle to believe they're the majority and would need evidence.
Meanwhile on the enterprise, pretty much every machine in every enterprise environment I've ever worked in has been set up for local relay. Again, if you think they're all using SMTP AUTH I'd need to see evidence.
> If you use smtp authentication, you're not running djb's code and all advantages of djb's code are moot
Do you actually believe that if you apt-get install postfix that you don't have any of the advantages of postfix's development processes?
I'm not sure I can engage with that. It's like saying running linux invalidates "all advantages of djb's code" simply because you're (sometimes) not running his code (e.g. when you're in a syscall). Madness runs down that way.
For what it's worth, most of the qmail smtpd patches I've seen to add SMTP AUTH (including the one I wrote nearly twenty years ago) are sensible, integrate well with the rest of qmail. You might not get djb's warrantee, but I can't believe you gain nothing by starting from a solid foundation.
if they don't need it, they won't be affected by that vulnerability because it won't be active.
>Meanwhile on the enterprise, pretty much every machine in every enterprise environment I've ever worked in has been set up for local relay.
In every enterprise setup I have seen, local relay was still protected using SMTP-AUTH for accounting purposes.
Plus in case of end-users, many ISPs still provide SMTP for mail delivery, all of whom need to support SMTP auth. That includes Fastmail and Gmail.
>but I can't believe you gain nothing by starting from a solid foundation.
Of course you do. But saying "Postfix is worse because it had a security vulnerability in a component that qmail doesn't even support" is unfair to postfix.
If you are running qmail with a custom patch then that custom patch will for sure have been exposed to less scrutiny than Postfix proper. It's not valid to extend the scrutiny given to vanilla qmail to all of the custom patches.
Ok, well that's your opinion, and hopefully now you at least know why I have mine.
I can't imagine forgiving Microsoft for all those AD/SMB and explorer CVE just because Linux doesn't support those things, but it's not really that important to me.
> In every enterprise setup I have seen, local relay was still protected using SMTP-AUTH for accounting purposes.
I have no idea what you mean by "accounting purposes" -- it sounds like you're talking about users relaying and not machines relaying.
There is no way you'd get a SOC2 (or SSAE/SOC1) if you used SMTP auth to protect local relays, so perhaps we are talking about different things when we say "enterprise".
> If you are running qmail with a custom patch then that custom patch will for sure have been exposed to less scrutiny than Postfix proper.
I'm still not sure you can have it both ways. Why is it when Debian patches postfix that it's still (in your mind) "postfix proper" but when I download a well-distributed and discussed patch from a mailing list, and read it myself, that you somehow think I'm at higher risk?
there is such a thing as machine- or even application-specific accounts that can be terminated or their credentials rotated.
Even better; you could use something like Vault (https://www.vaultproject.io/) to manage such application-specific credentials.
All of this is much better than relying on IP addresses never changing and never being reused.
>Why is it when Debian patches postfix that it's still (in your mind) "postfix proper" but when I download a well-distributed and discussed patch from a mailing list, and read it myself, that you somehow think I'm at higher risk?
I never said it was postfix proper. It really isn't.
Debian is known to f'up the security of its packages. But it's my hope that distributors have a better equipped security team and bigger exposure compared to some guy putting a patch on some website.
Debian also has a proper security update process that makes sure that patches go out to users quickly and painlessly. And because Debian is big enough, it's usually included in embargoed security vulnerability reports, so usually a patch will be out and sometimes even deployed by the time the vulnerability is known.
But the relative security of Debian-maintained postfix vs. Debian-maintained custom-patched qmail is the same as postfix proper vs custom-patched qmail.
However, vanilla whatever, in my opinion is superior to custom-patched whatever. And while postfix is usable vanilla, qmail isn't. It was bordering unusable back in the 90ies and it definitely is unusable nowadays.
qmail doesn't support TLS, it doesn't support spf, much less domainkeys or dmarc. It doesn't (as we're talking about) support SMTP auth, it doesn't support ipv6. It doesn't support any other common anti spam measures like greylisting or rbl checks, nor does it support a plugin system that would allow for easy plugging them in.
Heck, it doesn't even support rejecting mail at SMTP time so it's one of the worst offenders for backscatter which is one of the big annoyances for mail admins.
At this point, an unpatched qmail is as useful in fulfilling MTA roles as a closed port 25 is (which would totally be safer than both qmail and postfix)
Hence my assertion that everybody is running a patched qmail and none of these patches ever enjoyed the same scrutiny as even the security abominations that are sendmail and exim.
And worse: None of these patches were ever reviewed for side-effects of arbitrarily combining them.
Nearly every custom patch was made against a vanilla qmail because there is no central project to manage these patches. So every site runs a different version of qmail, none of them vetted at all for security.
This is a mess and we should really discourage this rather than dreaming of long-gone times where vanilla qmail at least was useful to a minor degree.
Partially-related update: I just looked at the qmail code: This thing is written in K&R style C and sometimes uses single-character variable and function names. I would argue nobody but djb can actually safely make changes to this code-base. This is all from a bygone age. Let it rest.
You've given some excellent examples of how qmail got this way, and what notqmail needs to change to be viable. I have my own running-in-production solutions to most of them -- for instance, https://schmonz.com/qmail/rejectutils for SMTP recipient rejection and https://schmonz.com/qmail/acceptutils for user-facing AUTH and TLS.
These may or may not become part of notqmail. But we believe that together we can carefully and safely evolve notqmail to meet modern needs.
But being selective about what features to add is itself a valid strategy for writing secure software — perhaps even the most effective one!
I can put a machine on the net with a closed port 25. That's the most minimal implementation of an MTA and it's entirely secure. Guaranteed.
The thing is that qmail was severely lacking features back in the 90ies and it has received zero feature additions by its original author since then, so it's safe to assume that it's now lacking even more features to the point where anybody running qmail these days is nearly guaranteed to be running franken-qmail full of third-party patches which are surely less well maintained than proper Postfix.
That was my point.
qmail was good when you could use it in its default configuration. Unfortunately, very few sites could (much less still can).
Which brings us to the question of who you trust more: The maintainers of a full-featured MTA who have maintained their software for 20 years or some person's patches to scratch their itch before moving on and never touching it again?
Rust is not a magical language. It's just got the right features to be a C replacement, which is mostly: a simple type system (much simpler than C++'s or Java's) that has the right extensibility (traits, no inheritance) and can have an ABI that programmers can reason about due to monomorphization (no v-table dispatch).
All the borrow checker stuff amounts to: just enough pain that to avoid it you must design better APIs.
For a project like notqmail, I would worry about portability. qmail runs on a lot of platforms and getting Rust bootstrapped is a bear.
I wanted to like qmail, it was definitely an improvement over the complicated disaster that is Sendmail, however, it always seemed to require patches to get it to do things that Postfix could already do out of the box. In addition I didn't like the dependency on the inetd replacement thing (daemontools?). It just felt so un-ergonomic.
In addition, the Maildir mailbox format (while very cool in its own right) usually required third-party software to be patched manually, for example WU-IMAP.
A day or two before the September 11 attacks, the UNIX time_t value rolled over to 1 billion, and went from 9 digits long to 10 digits long. The moment this happened, a comparison function in the WU-IMAP maildir patches failed, and our IMAP server started sorting and displaying all our emails out of order.
After scratching our heads and poring over the source code for a couple of hours, a co-worker and I found the problem and submitted a patch to the maintainer.
Funnily enough, the imap-maildir page still references this event even today (it probably hasn't been updated in years):
http://www.davideous.com/imap-maildir/
Under the section "10-digit unix date rollover problem". But it drive home to me as a fairly young sysadmin at the time of the dangers (at least for the inexperienced) of using third-party patches just in order to enable a non-standard mailbox format (eg. Maildir).
> Did your installation stop sorting messages correctly on 9 Sep 2001 01:46:40 GMT? Find the fix to the 10-digit unix date rollover problem in the important updates section below.
Reading that I imagined someone running those patches on their mail server for two decades before suddenly realizing one day in 2019 that for the past 18 years their messages have been ordered incorrectly, ha ha.
It's possible to package qmail in such a way that it's trivial to install and run, supporting many modern features by default. I've done it. Here's a demo: https://youtu.be/Vq6vu9T3vow
But that required a lot of decision-making and a lot of effort by the packager. With notqmail, we hope to make packaging much much easier.
Could someone explain why one might want to use qmail (or this, its newest incarnation) over postfix or opensmtpd? Would I gain anything by shifting to qmail or opensmtpd?
As far as opensmtpd is concerned, it is production ready and has been used in high volume environments sending mails to millions of recipients daily, handling multi-million queues.
The project was started in 2008 so when it comes to being run in production environments, I think we can claim that it's ready :-)
Now as for the what you would gain, I won't do proselytism, we have published a couple presentations on our website which describe how it works and some of the technical details behind it. You will also find lots of technical details on my blog where I post mostly about OpenSMTPD internals.
If you have a running Postfix, I suggest you don't shift but keep your working setup. If you intend to setup a new MX, you should definitely give a try and see how it goes for you.
Vpopmail was great, too for managing users, hope it'll still work.