Mail-in-a-box: easy to set up modern SMTP/SMTPS server stack
github.com
github.com
They're such a pain to use that since becoming the sysadmin in charge of our work email servers, I gave up running personal ones as well and just pay FastMail to deal with it for me - ain't nobody got time for that.
I often dream of taking a sabbatical and writing modern, user-friendly SMPT and IMAP[0] servers. If only so I could use them at work on my return and save myself a lot of time and stress.
[0] Possibly even with Exchange ActiveSync support - Microsoft licenses the protocol, but I've no idea what it costs http://www.microsoft.com/en-us/legal/intellectualproperty/IP...
I've been running it for a few years now, admittedly with a very basic setup. My config is about ten lines long, written in five minutes or so. I haven't really needed to touch it since.
The setup is documented here, and pretty nice https://github.com/skx/ms-lite/
If you don't care about local access (IMAP & POP is good enough, no mutt to the mail spool for you) Dovecot is a huge improvement on everything else. You configure it to listen directly for LMTP and use MySQL for user information and Maildir for data it's almost as easy as running a random php+mysql webapp. Even serverside filtering with sieve "just works".
The inbound SMTP/Spam filtering stack is still a PITA but that's because of security issues (spam).
DBmail (dbmail.org) is one rework that seemed sane but stores all messages in a RDBMS and I didn't want to deal with scaling it at work (little ISP, ~10k mail users, ~1.5 million messages per day including spam) but it was fine for personal use.
I've been playing with homegrown POP & SMTP servers that use a s3 compatible datastore as the backend but that's a side project. People go crazy over email so I really want a simple to operate, sane, zero point of failure mail system...
The clever thing is, it can recover from a split brain scenario completely, safely, and without any losses. If your two servers can't see each other for a few hours and you make conflicting changes on both of them, then it apparently is able to recover completely and entirely automatically when the connection comes back up.
http://wiki2.dovecot.org/Tools/Dsync
You could have one remote server in a DC, and the other locally in your office. Point your mail clients at the one in the DC, but then configure your office router to intercept connections to the DC server and re-route them to the local office server. When you're in the office, you hit the local office server. When you're outside of the office, you hit the DC server.
[edit] Any chance your can elaborate on your S3 test setup? I've considered something similar. I would be interested if there was any mail server software that already does this. The alternative would be to use an S3 based filesystem as your store, but this doesn't seem very efficient. You'd definitely want local caching of messages in this setup.
Basically when I started out learning Go I saw a presentation by bradfitz where he made a POP3 server that used twitter for the backend and thought hey, access S3 instead. He had a smtp library on github so I hacked up something that accepts all mail and stores it in a S3 bucket too. I haven't hacked in things like user auth so it's all very proof of concept/useless at this point.
I'll have to take another look at DSync. The SSH batch job per user scared me off last time. I might be able to use it for HA/Failover.
The main problem was, that you can do everything in X different ways and have to think careful which way you want to go.
In addition i am running a very simple private setup for years now without any problems.
Its the reason I got into Puppet and Ansible - vainly hoping all those barbaric problems could be encapsulated.
Postfix is actually a beautiful piece of software once you spend some time with it and get to know it. Like most things, once you've done it a few times (and written everything down!) it isn't so bad :)
LDAP on the other hand... that's something that _always_ defeats me.
Couldn't agree more.
I've also learned a ton from the source code. If anyone has even a passing interest in security, i'd highly recommend browsing the postfix source code.
The architecture is, at best, an anachronism.
Designs of much more elegance have been explored (qmail). But as a matter of fact, neither postfix, nor qmail, nor exim, nor (god forbid) sendmail belong into our day and age anymore (I have run 3 of them at scale).
A modern MTA is way overdue. Please keep writing MTAs kids!
In modern languages. Until you get one right. Thanks!
I did have some exposure to qmail before that, and it did feel like it was simpler in some good ways -- but then it became a sort of abandonware, sadly.
Any thoughts on Lamson ? (http://lamsonproject.org/)
[edit: I did recently switch a few vps' over to nullmailer rather than use exim4 configured for smarthost delivery -- once I figured out how to get tls working thanks to this blogpost http://metz.gehn.net/2012/11/nullmailer-with-starttls/)]
I can't see why in the world anyone would advocate throwing all of that in the bin and adopting something new, flashy and unproven that happens to be written in go/rust/node/other language du jour? I mean why? So we can write configs in JSON and get a 'free batteries included' web management console for monitoring queue lengths?
Yes. Exactly that.
Postfix requires the user to understand pretty much all of its delicate (and often very much ass-backwards) abstractions at once, to accomplish even the most trivial tasks.
Aliases vs virt aliases, transports, the six dozen magic interactions of $mydestination, $myorigin etc. with one another and everything else, ... don't even get me started...
Nothing is easy in postfix. Everything is deeply entangled, counter-intuitive and error-prone. And there is absolutely no reason it has to be that way.
See what elasticsearch did for FTS. This is how software is designed today.
I'm not bashing Postfix, it's still a workhorse. But let's not be sentimental about software. Being better than sendmail simply is no longer the big deal that it was in 1998.
Today I expect my MTA to be all that postfix is, and easy to use.
Things I didn't like with ES include:
- Awful documentation (even the newly released guide) and too much marketing speak. I constantly felt like I was reading a brochure rather than a man page.
- I found it opaque and difficult to get clear, concise info on how it's put together and what it's doing under the hood.
- Marvel was nice but it's a shame they charge for it (it's not _that_ nice)
- The install routine (in an effort to be super simple I guess) just felt wrong and needlessly over simplified (I didn't like it making decisions without telling me what it was doing).
And more...
I will concede that working with ES's config files is a lot nicer than dealing with Solr's XML though! And in case you're in any doubt, I am by no means an enormous Solr fan either.
Anyway looks like we'll just have to agree to disagree on this one :)
Edit: also, I am occasionally prone to a bit of sentimentality for good, older software (it took me far too long to give up on apache and embrace nginx for example) - so there's definitely a little bias on my part!
Edit 2: If you could get Salvatore Sanfilippo and Igor Sysoev to team up and build a modern MTA, that would interest me :D
I'll probably come across as arrogant now (sorry), but your critique of ES sounds almost ironic to me.
It's especially curious as you do claim experience with SOLR. Have you ever scaled it beyond a single core?
I'll go on a limb and claim that most of what you perceive as opaque in ES is likely magic that actually works. The rarest kind.
Yes, Marvel is not optional to run a meaningfully sized cluster. But where is Marvel for SOLR again? Where is automatic clustering and balancing? Oh, it doesn't exist?
Sure ES isn't perfect. But it's one generation ahead of SOLR. And Postfix is one generation behind SOLR. That's all I'm saying.
What i was trying to dig at was more about the code than the user facing side of things.
It beautifully modular. It clearly written - that's a crazy hard achievement for software as committed to security as postfix.
It's dead easy to hack on the code - you could arrive and even without looking at the first class documentation you would be able to tackle implementing a change. The source is that thoughtfully laid out.
The result is that now when you are configuring SMTP you have to also configure DNS. That means more things that can go wrong, and more things to check as you are setting things up.
It also means you may need to pay a fee for a domain name. This is because we all submit to the notion of an ICANN root and commercial registrars selling (renting) names that cost nothing to create. Thus email is not solely under your control. You generally have to play the ICANN DNS game, only because your email recipients are playing. Nothing stops anyone from running their own root though. And this is what is done with private DNS inside organizations.
And then, as if that DNS complication was not already enough to take control of email away from you, you have various schemes trying to prevent spam that discriminate for or against mail you send based on IP address and domain name.
Can you operate email without DNS? Technically yes. There was a time before DNS, and email worked just fine. Practically speaking, today you need DNS, whether it's under ICANN's root or your own.
All this hassle steers you to just accept third party email hosting. Profiting from this arrangement has become a career for many a man. And with "the cloud" many are hoping to cash in yet again, as organizations who once ran controlled own email feel pressured to let a cloud computing vendor control it for them.
The fact that all this third party control makes warrantless search and surveillance so easy is but one side effect. Centralising hundreds and thousands of accounts in third parties make the spammer's job easier, too. If you think about it, there are many unwanted side effects of centralizing email. When every sender and recipient are connected directly to each other via a network, why would you want to prevent them from sending messages to each other directly?
With the constant connectivity and bandwidth we have today in many places, the centralisation and outsourcing of email is baffling to me... it is nonsensical... until you remember how much of a PITA it is setting up email.:)
It's no wonder we let third parties handle it. Is this PITA by design? Who cares? Let's just fix it. More of these projects should exist. Or made public (I imagine many of these are personal setups now being released for public use). I have my own that uses qmail.
If you want to use names instead of numbers, then you can do that. You and your recipient must use the same DNS root.
I'm not sure why you would have to remember IP addresses. We routinely "dial" telephone numbers by selecting from a list of contacts. IP addresses are approximately the same length as telephone numbers. The folk wisdom is that people can remember about 7 digits. But even if you disagree on all of this, what does that have to do with letting someone else control our email? The issue here is control, not whether we use names or numbers or something else when we enter the address of the recipient.
The nonportability of IP addresses is a problem in its own right, but I don't see the relevance here. Again, you are trying to engage me in a debate over domain names versus IP numbers. Perhaps that is an interesting issue, but here I am interested only in the issue of control over email (and because email and DNS have been coupled together, DNS). And that is what the OP is interested in as well.
I said that email and DNS are closely linked and this makes email more challenging for any user to control. 1. Because it complicates the setup and 2. because DNS as we currently accept it is controlled by third parties.
You are trying to suggest that I am advocating against having email addresses that use names instead of numbers. I am not.
If email is linked to DNS, and someone else controls DNS, then you cannot control email.
If you disagree with the preceding statement then please explain.
What was a lot more difficult was setting up the actual mailserver itself. Even a simple, two-mailbox-operation was an exercise in frustration when it came to trying to get mail working on a little VPS of mine. Shit, you have to make the sendmail config. How balls-out insane is that?
More recently, there's little working tutorials to get yourself a working dovecot/postfix server, which are relatively easy to understand (thanks, digitalocean!) but I just checked out the first one I found on my google search and it's 2,800 words long. 20 pages if you were to print it out dead-tree style. I can give you a tutorial on DNS and MX records in much less time than it would take to go through setting up any MTA on linux, and that's the trouble.
But I am actually referring to something different: hosting your own mailserver.
So when I say "set up DNS" I mean set up a DNS server, not simply an MX record. This allows you to create your own domain names and hence email addresses. As I said above, these email addresses are valid so long as you and the recpipient use the same DNS root (e.g., ICANN's root in the case of the public internet).
As bad as things are in terms of the relative difficulty of setup, I think there are defenders of the status quo for email and I imagine this explains how I could be downvoted for my comment.
Don't get me wrong, I love email. It is the reliance on others to handle 100% of it that troubles me. It is purely a control issue.
Edit: it's not so much that it's about configuring Application X, but that there are so many moving parts, from local aliases to firewall fun to DNS entries. Then there's the fun of 'deliverability', greylisting and so forth.
My setup, with some documentation, is here https://github.com/skx/ms-lite/ and evolved from a commercial service http://book.mail-scanning.com/
I've used Haraka standalone for a couple of custom-jobs, but I've never used it at any significant volume. That said qpsmtpd was a pleasure to use, deploy, and develop against, so I'd expect to have a similar feeling with it.
On a serious note - you'll find the Haraka developers more friendly and active than any other MTA out there. Just join the #haraka IRC channel on freenode and we're happy to help anyone.
It's one thing to easily obtain my own personal mail server, it's another entirely to understand and be able to maintain/improve it.
- DKIM signing on outgoing messages (opendkim).
- The machine acts as its own DNS server and is automatically configured for SPF and DKIM (nsd).
Are you speaking of people who failed to disable relaying ?
You had to jump through certain hoops to ensure that your mail arrived. DKIM/SPF are two such hoops (or related techs, anyway; both have detailed Wikipedia entries). Maybe it's not so much of an issue these days.
Players like MailGun, SendGrid and Postmark provide such features as part of a EaaS product.
https://github.com/JoshData/mailinabox/commit/ec11241a7a619a...
* Test/alert if your PTR mismatches your HELO name
* If your IP is on any of the major RBLs
* Point admins toward AOL/Yahoo/ReturnPath whitelists and feedback loops.
Frankly I've had mixed results over several years of hosting my own personal email server, even with a valid PTR, SPF, and DKIM. Most SMB and personal email systems I interact with are fine, as aside from Bayesian filtering those tend to rely heavily on blacklists which I am responsible enough to keep myself off of—easy given that I'm the server's only user. And I've never had an issue sending to Gmail users.
Microsoft is the real f-up in this regard, originally with Hotmail and now with the updated Outlook.com. Even though I'm on no blacklists and pass both their SPF and DKIM checks:
Authentication-Results: hotmail.com; spf=pass (sender IP is XXX.XXX.XXX.XXX) smtp.mailfrom=redacted@example.com; dkim=pass header.d=example.com; x-hmca=pass header.id=redacted@example.com
X-SID-PRA: redacted@example.com
X-AUTH-Result: PASS
X-SID-Result: PASS
If you visit their support page you're recommended to join their partner Return Path's Sender Score Certified Email program to stop junk filtering of your legitimate messages. That's great, if you're a medium-sized organization that doesn't mind paying their certification fee; but Return Path won't even allow an individual to register (I've tried). I'm familiar with Hanlon's razor, "never attribute to malice that which is adequately explained by stupidity", but this at least feels like a racket.
Unfortunately I don't know what else to do, other than continue to advise everyone I know not to use Outlook.com and friends.
And to finally answer your question, if this VM runs its own DNS then it probably takes care of SPF and DKIM adequately, but you'll likely have to contact your hosting provider for the PTR record, which is at least as important. But even following those best practices you can still have delivery issues such as I have described above, leading to the unfortunate reality that if you really need a message to be delivered you're probably better off sending it through Gmail.
i don't know if they still do that, but it was far easier to convince everyone i knew at the time that hotmail was garbage, rather than making any headway with their mail admins.
https://lists.exim.org/lurker/message/20070614.130838.42d1bd...
I haven't done this for many years though as it was such a terrible hack. I do wonder if it still works, or if the threshold has changed though.
I didn't catch it for a while because so much of the email world simply assumes ipv4, but traffic happily exits my ipv6 interface when the destination supports it.
Not a problem with a good ISP but many retail ISPs don't have a process for maintaining rDNS to arbitrary blocks held by subscribers.
Personally I have a VPS that I relay mail through.
Another downside is you lose logs of and insight in the mailq and the recipient smtp server responses.
By the way, I've used Postfix and it took me like 10 minutes to get it running. I wouldn't call it user friendly, but far from being complex.
We are of course CANSPAM compliant, use good email headers, unsub etc. etc.
However I do not regret it at all. I had Linux experience and it took me one day and a half work to get it all working very nicely.
I am happy with my config: Postifx, Dovecot, RoundCube, SpamAssassin, ClamAV
Server supports unlimited domains and user accounts - SSL is required for all connections, I only allow IMAP and I have configured it with two valid free StartSSL certificates:
One under mail.domain.com (for email clients)
One under webmail.domain.com (for the webmail)
The thing that scared me the most was outgoing emails being dropped - however to date I have delivered fine to all main email provider - followed a few simple rules:
Ensure you have both SPF and TXT correct DNS records
Ensure you have IPv6 configured properly (Google was rejecting due to this)
Set your reverse DNS
Set your machine hostname etc. in postfixThe guy behind it, Josh Tauberer, also wrote GovTrack, which was (and still is) important for anyone who builds things with Federal government data.
The last thing the internet needs is more clueless users with dns servers.
Mail-in-a-Box uses 'nsd', which is a non-recursive DNS server. I don't think it can be used to relay DNS queries in a DDOS attack. But if you meant something else, please let me know and/or file a ticket (I don't think I'll be looking here for replies).
I've been doing this for several months with no issues, and it's pleasant getting out of google's targeted profiles.
The internet is too darn centeralized, people can't just put services out there.
1. ISPs providing asymmetric service (and it's been getting worse, it used to be 10:1 dl:ul, but when I upgraded speed last year it grew to 20:1).
2. Why do we have a plethora of centralized services, when ideally a smart firmware in your home router can handle pages like about.me or simple blogs.
The whole process keeps getting more complicated as the internet grows more hostile and end-user requirements increase.
This project sounds interesting and much necessary, but the real problem to me is the economy of scale. If 10 Unix guys sit together and set up 2 servers for 20-30 personal mailboxes, the time and money might be worth it. Setting up all those services, filtering and redundancy by yourself for just yourself – and monitoring them continuously – is wasteful and painful, even if you like to tinker with Unix systems.
There used to be a few geek-oriented ISPs around that offered SMTP/DNS backup, spam filtering and similar services so you could off-load some effort, but I guess they've fallen prey to low-margin virtual server business.
If ignoring email coming from a major provider isn't an option (it isn't.), then what are your options?
GMail sends from a farm, it's random where the exit ends up being.
I only have that problem from GMail, Hotmail and Yahoo don't seem to have this problem.
If you're going to implement greylisting yourself it makes more sent to hash the sender+recipient pair, allowing IP addresses of the senders to change - which is what will happen with many of the bigger mail-hosts, gmail, hotmail, etc.
I do have a question:
Does hosting your own email truly isolate you from government surveillance?
If a million people use this to host their own email on AWS, what's to keep the NSA from ordering Amazon to tap into your data?
You almost have to host it on your own hardware in the garage in order to add a significant layer of isolation. Even with that in place there are ways to access your communications through your ISP. Harder, not impossible.
What's reality?
Not by itself, no; the government can still get to you by getting access to the machine you host it on, or the switch the mail flows through on its way in and out from the outside world. Hosting your own just reduces the number of third parties the government can try to shake down to get access to your data.
In other words, what's the difference between buying email service from GoDaddy and renting a VPS from them to host it yourself?
Not being critical at all, just trying to understand if there's something fundamentally different about Mail in a Box that makes it more government-secure than other approaches. The description seems to list privacy from government snooping as the primary motivator. Is it truly effective at that? If so, why and how?
As I understand things the very idea of being secure from snooping is an illusion so long as your email is on a machine that you don't physically control in your home or office. And, even then, the best you can do is encrypt all traffic to make it more difficult --not impossible-- to capture your data.
Am I wrong?
In order to escape mass surveilance this is most likely enough. As all you have to do is to be different.
If you think you are actually "target" by a goverment youv'e got bigger problems. And you should probably stay away from email alltogether.
The irony of Docker IMHO so far is that it almost makes installation that needs configuring with user-specific values harder, unless they know how to build docker instances themselves.
1) You have to be your own 24-hour support, or accept that you will lose or bounce mail. You can be stuck with realising that it's failed while you're on holiday.
2) You have to be your own antispam. I've had my own email domain for fifteen years; it's on basically every spam list out there. Gmail is very good at this. At the moment I rely on my hosting provider who filters out most but not all of it.
3) You can have tedious issues delivering outbound mail, even if you're not using SPF.
In my opinion the right level to run email is "small cooperative" (10-100 users), where cooperatives share software, experience and maybe even staff but are administratively and physically separate.
Great job. Mail configuration is pretty hard in the number of steps and amount of configuration that needs to be done to get it right.
Also, even if you both run your own MTAs STARTTLS does not help much except to reduce the scope of the passive-only surveillance dragnet, because the configuration does not require signed certificates.
Finally, if both parties are technical enough to run their own email servers, um, why not just use GPG?
From a technical point of view, I put this squarely in "fun, but a boondoggle" territory.
At this point the only people I know who could or would use Pond are people who can compose crypto primitives on their own anyway.
Also if we are going to go all Pond and do messaging from the ground up, we need to acknowledge that the global passive adversary is real.
EDIT: after replying to rickisen below, I'm starting to see your point. Maybe we need Pond or bitmessage or a resurrected mixmaster if real people think TLS solves traffic analysis.
Where are you hosting your email. Are you keeping it on say an AWS instance? OK flag that, all the plaintext would be there, and also, duh, it's still obvious that it's your communication because you're connecting to it, and maybe even paying for it.
OK, you host it at home on your DSL line? OK, so plaintext is harder, but oh wait. Yeah, your IP.
TOR hidden service or something? Good luck getting SMTP to work over tor.
So tell me again how TLS is going to help with metadata.
Exim and Postfix (and hopefully others to come) are both working on adding DNSSEC and DANE support right now. Which means in the not too distant future it will be possible to publish cryptographically signed data in the DNS that your MX has TLS support, and what the fingerprints are.
> Finally, if both parties are technical enough to run their own email servers, um, why not just use GPG?
OpenPGP encrypted email leaks all sorts of information/meta data.
GPG gives you actual privacy but minimal resistance against traffic analysis. All this other stuff gives you NO end to end privacy, AND minimal resistance against traffic analysis.
Messaging is a weird special case and GPG is behind the times (because people are coming to realise that repudiability and FS are important), but it's still a heck of a lot better than even working TLS slapped on SMTP (which doesn't even currently exist).
The combination of Mail-in-a-box and Mailpile seems pretty exciting.
Since running your own mail server is opening a new vector for attack, I'd be curious how Mail-in-a-box is going to handle security updates.
One thing to watch out for: Yahoo is a complete bastard to deliver to. One day they'll decide to block you (if they don't immediately) and the only way out is to fill in a web form which either allows you 6 months of delivery if you're lucky or blocks you with no hope of resolution for 6 months.
Total assholes they are.
You basically follow a web installation wizard, create a domain and then you can use Roundcube (included) or any IMAP client. Couldn't be easier.
Just my two cents.
Edit: Looks like my idea is far from original<https://news.ycombinator.com/item?id=7634677 >
Because you haven't done it yet. :)
Hours in the day, unfortunately.
This looks way more interesting to me: https://github.com/al3x/sovereign
I gave up and went to fastmail. Setting it all up wasn't a problem. Maintaining it was a nightmare.
+1 to what codexon said about the rate limit, but this seems absolutely amazing
Good luck with your project. I know I will use it.
In short, Gmail provides more value to me than being a simple SMTP server, so I'm willing to pay with my advertising eyes.