A Survival Guide for the Small Mail Server
spamhaus.org
spamhaus.org
I ran a mail server for almost five years, and 9/10 of the issues I had to resolve were various anti-spam blacklists or individual hosts blocking our emails and then having customers complaining because we never emailed them.
There was never any reason given for the blocks, and our logs didn't show any outgoing spam. However all of these spam blacklists would accept a "donation" if you didn't want to wait the 24-72 hours for them to unblock you. Essentially it is a shakedown.
I've just given up self-hosting and I recommend others do the same. The issue isn't the spam, it is how much time you'll waste fighting the anti-spam blacklists and large email hosts. A job better left to e.g. SendGrid or SES.
PS - We sent tens of thousands of emails a day.
The subject is self-hosting a _small_ email server. When you get into 10K/day range, you are bound to have false positive spam reports - both automatic and from pissed users - and this in turn can lead to RBL listings.
I've been running a mail server since 2004 and not once had I have any issues with RBLs. It's a low-volume server though, perhaps several thousand emails per month.
The occasional marketing type bulk email is always sent through an external company. This means I never have to worry about getting blacklisted by those messages.
... but for which there was clearly no valid reason for the mails being sent any more. As in, "the domain expired five years ago and is now a block hole", or "AT&T retired that domain a decade ago and migrated all its users elsewhere", or "that former email provider has had its domain parked on some Russian server for years".
If you're not periodically revalidating your email lists, checking for retired domains, and/or processing bounces properly, you are sending spam, I can guarantee it. Quite possibly not spam that anyone particularly cares about, but it's creating traffic for somebody's email system (starting with your own).
The other blacklist suggests that it's our commercial site that's hosted with a provider who uses linode, and the linode CIDR block our company website is on has been 'banned with no recourse to remove' due to the huge amounts of spam it has sent. For that list, it apparently doesn't matter that our mail comes from google, not linode. When I get some free time, I'll try to move that website.
What do you do when the vigilantes screw you?
It's really not very clear with that site, and ironic that a list apparently named after people who aren't compliant with docs has such poor presentation of information. The documentation is clear... eventually, if you pick the right path. I'll take this on the chin as my misreading, but unclear documentation isn't helpful (particularly if you're juggling a few other fires at the same time...)
Edit: Yes, I was correct. gmail does come with 'postmaster' and 'abuse' built in: https://support.google.com/a/answer/33389?hl=en
Edit 2: I forgot I removed the line where I mentioned that I thought postmaster was baked in :)
Edit 3: It looks like the domain I'm using with gmail has a screwed up abuse and postmaster. It's a domain alias, one of two. I made new mail groups for abuse/postmaster (the goog still monitors them, re: the link above) and noticed that only one domain alias was covered by them, when all other mail groups listed both. Sending test mails confirmed that setup. After a lengthy goog support chat, it seems that the way to fix it may be to remove and re-add the faulty domain alias.
I both love and hate these long chains of cause-and-effect that we get in IT...
I now pay some company to deliver my personal mail, which is just dumb, and clearly isn't caused by them being better about spam (as I sent no spam): it is caused by a militarized anti-spam-crazed group of people who have managed to build a system that essentially requires an oligopoly for email to function. A lot of these people don't even consider it a problem to use "report as spam" as a ways to punish companies for things that aren't even sending spam (which is something I only bring up as it demonstrates the "militant" comment).
The reality is that an open internet where people can talk to each other without going through third parties requires a world in which you, heaven forbid, receive some spam. It is actually a similar construction as the one for why we should have network neutrality: large internet carriers in the world of email should, if you really believe in open access, not be allowed to say "these people can't send mail because they are too small and not paying someone". We all need to accept at least a little spam to guarantee everyone can send mail.
If this means that you have to get better at dealing with spam, as some of the heavy-handed techniques people have put in place for dealing with spam are no longer possible, well: so be it; and if that means you need to turn off the notifications for new email on your phone to keep from getting angry when you receive some spam (which tends to come up as a reason to hate spam), I hope people realize that they caused their own reason to be upset, and that getting notifications for anything is probably unhealthy, at least for you, anyway.
Sadly, the large companies who have slowly come to "own email" have no incentive to break this cycle, whether in action or in words, as the current state of affairs is what is giving them their power: whether they build solutions that can solve spam in a way that is both fair and federated, or they just try to educate users about how to better deal with spam and decrease the stress in their lives (including telling people to stop letting email interrupt them), it will just harm them. This doesn't sound like something the private sector can solve.
I get a fair many spams that have "unsubscribe" links, or even that look like newsletters (for companies or organizations of varying legitimacy). I'm always pleased when they come through one of the major mailing list providers that provides a separate "report abuse" link, with which I can report that no, I don't just want to unsubscribe, I want to report that I never subscribed in the first place so that the list itself gets terminated.
If it's possible for someone to be subscribed to your "newsletter" without having explicitly consented to doing so (and in the process proving ownership of the subscribing email address), you are sending spam.
A problem with this attitude is that most people have no idea whether or not they consented. Perhaps they consented in 2003 but don't remember. Perhaps they consented yesterday but want to revoke consent because they're too busy right now to read your newsletter. There's a reason why Gmail silently converts your spam report into an unsubscribe if the correct headers are set. Too many PEBCAK false positives.
If anyone indicates in any way, shape, or form, whatsoever, that they don't want your crap, stop sending it to them.
Email is all but useless these days. Postal mail's hardly any better.
If it's obviously spam, the sender's IP needs to be blacklisted, his hosting account terminated, etc. in order to protect other people.
If the recipient just changed his mind about whether he wants to receive a newsletter that he explicitly signed up for less than a week ago, there is no need to punish the sender. The sender just needs to be notified that the recipient unsubscribed.
I don't know about you, but if all the social networks, instant messengers, and "we're gonna replace email" startups in the world went dark for 24 hours, I probably wouldn't even notice. They're all but useless to me, after all. But if my email went down for 24 hours, I'd definitely make a big fuss about it. Ditto if someone makes the wrong decision about which senders to block.
Very many senders stretch the boundaries of what they send.
Many major email providers do provide some form of full-loop feedback to bulk senders, and aren't unreasonable in their policies. The expectation that all email senders be perfect isn't reasonable (and generally isn't applied), but reasonable diligent effort really must be followed, and far too often isn't.
The model I've strongly endorsed for a long time is to simply apply a mail acceptance criteria that's scaled to the level of non-problematic mail originating from a sender. If most of the mail meets acceptability tests, most of it is accepted. If most of the mail doesn't meet acceptability tests, then most of it's rejected. If the problem's modest, the rejections are temporary (e.g., re-try in a bit). If it's severe, the rejections might be made permanent.
Poorly-behaved senders (those trying to re-transmit the same message before a reasonable retry interval transpires) are penalized harshly (all connection attempts are refused for some period).
RFC 2821 suggests an initial retry delay of 30 minutes, and up to 4-5 days for delivery attempts.
A typical exim4 retry configuration "specifies retries every 15 minutes for 2 hours, then increasing retry intervals, starting at 1 hour and increasing each time by a factor of 1.5, up to 16 hours, then retries every 6 hours until 4 days have passed since the first failed delivery."
Arranging with specific high-interest peers for expedited delivery, especially from individually trusted senders, might also be a useful thing.
Note that email is not guaranteed to be reliable or instantaneous. And as I've commented elsewhere on this thread, the problem now is that both you as an individual server admin have to deal with everyone else on the Net. And they've got to deal with you....
It's the sheer drudgery, more than anything else, of that, which doesn't scale. What's forgotten of the "golden age" of the Internet (1980 - 1992 or so) is that there were, comparatively, very few hosts. Dozens initially, a few tens of thousands toward the end of the period. Individual accountability largely worked. And while some individuals had direct connections, it was largely universities, a few employers, and government agencies who played the role of ISP / mail service provider. The system wasn't quite so chaotic as is commonly thought.
I recently saw a talk where someone held up a physical copy of the directory to the Internet circa 1985 or so. And it actually had everyone listed not once but 2-3 times -- by name, by organization, and by email address, or something like that. A not-very-substantial document, really.
Today's equivalent would have many millions of entries, I suspect.
As a user I also appreciate the ability to report abuse to someone I trust to deal with it.
But as someone operating various mailing lists for customers, the vast majority of the time when we get abuse reports from customers that "never subscribed", we have detailed history records showing that they did actually specifically take action to subscribe and confirm. This is consistent with pretty much every other thing - for every action in the systems, there are always customers that insists they have done no such thing.
Until we show them the audit trail.
Users are notoriously bad at remembering what they did. Which is not so odd given that most of the time the action is an inconsequential spur of the moment thing that they do or don't do as a matter of reflex and never think about again. But it makes "Report as spam" horribly abused.
And yes I Mark things as spam that may not meet the legal definition of spam, because companies tend to start sending you all sorts of crap once you sign up for their service.
In my ideal world you wouldn't have to pay a third party to send the email you would have to pay me for me to receive it.
It's never "some" spam. On my own mail domain, I get slightly more spam than wanted mail after spam filtering. If I didn't filter it would be tens or hundreds of spam items to actual mail.
The notifications question: well, in some ways people have moved to IM because it offers lower latency than email, generally better antispam, and usable notifications. Of course this is because all the IM solutions are centralized and can ban abusers more effectively.
I've only ever encountered one truly "militant" person against spam, and he refuses to accept mail from gmail because of NSA surveillance. Everyone else wants to avoid rejecting legitimate mail as much as possible while keeping email usable at all.
Lack of feasible antispam mechanisms is one of the things responsible for the decline of USENET.
If your idea is to overload google itself, you are going to need a bigger botnet.
Except for this problem, blacklists have never been a problem for me. I guess it's different if you send newsletters or other mass-marketing mail.
The anti-spam zealots ruined email for everyone, and they don't even have the honesty to admit what they did. "Hilarious and infuriating" is right.
A few minutes to configure smtpd. A few more to update DNS settings.
The problems are that:
1) the determination of "works well" or "doesn't" is almost entirely out of your hands. A DNSBL or major email service provider decides they don't like the tilt of your kilt and it's titsup.com for you. Been there, done that, shredded the t-shirt.
2) How much of a problem this becomes is a matter of who's inconvenienced, how much, and what alternative contact channels they've got. Generally, though, it's something of a PITA.
Things can work well for a long time. Or not. You're just never quite sure when they'll blow up.
And if that's sendmail smtpd, it's not "a few minutes" to configure if you're unfamiliar with it.
It's required basically no maintenance at all from me. As for monitoring, I check my mail daily anyway and I will notice it if it stops flowing.
A few weeks ago I actually moved and took down my server. I took an old netbook and made it a new, temporary server. Again that took me literally minutes.
Thankfully smtp is a robust protocol so if something fails, you normally have a couple days to fix it before mail starts dropping.
But what do you do when you're using third-party email service and they decide to kick you out all of a sudden? Happened to me with Google. Happened to many other people as well, also with other providers.
I ran my own debian-exim for about a decade until I got tired of the care and feeding. opensmtpd does look suitably low-maintenance and I might pick it if I ever wanted to do this again, but it was released after I gave up being an email admin. I still have the mail domain, but having the same email address for sixteen years means ending up on a lot of spam lists.
I have been running my own email server this way for about 10 years and I have never needed any kind of antispam measure except a simple .procmailrc.
Self hosting was once fun; now I hate it. Now I would only ever recommend getting a business gmail setup and using something like Mail Chimp. Not because I like either option, but fighting with blacklist operators and arrogant mail admins jaded me and now I refuse to support email servers for people.
Don't get me started on the web hosting side of this equation.
The raw mail flow may be useless without filtering, but blocking legit mail from a tainted IP addresses (tough to avoid anymore for the small business) is simply counter-productive. Our job as mail admins is to send and receive mail. We should let the receiving mail server tag the mail and let the mail clients filter.
We should never, never, never reject a message during the SMTP transaction. Just never.
There should be a new Fry meme: Shut up and accept my SMTP connection!
What you're seeing (and a conversation I've had numerous times, particularly with some individuals who seem to think it's all some Vast Conspiracy Against Personal Communications) is simply the challenges of dealing with email on a scale where you've got more peers than most people have email contacts -- where a peer is another peering email system.
It turns into a reputation management system. And it's a lot easier to deal with those reputations when you've got a handful of major public providers (Google, Microsoft, Yahoo, Aol, Inbox, etc.), and a few thousand major corporations. Once you get outside of the Fortune 500, even corporate emails get hard to deal with. At Krell Power Systems, we had a customer who required us to provide our email server IPs (already listed in both SPF and DKIM, natch) before they'd allow mail from us. The fact that they run a few machines powered by U-235 and are concerned about SCADA threats might have something to do with that level of paranoia.
But allowing all spam through, storing it, and relying on users / client software to filter it is expansive and quite error prone. The good thing about SMTP-time rejection is that it's unambiguous: any well-formed server will recognize that it's failed delivery, and in most cases the message is immediately bounced back to the sender. Accepting email and later trying to determine whether or not it's legitimate risks spoofing, Joe-Jobs, and silently-lost messages. That's actually far worse.
Much as I wish everyone could simply run their own servers, with systems as they stand now, it's just not possible.
The real problem for a small company mail server isn't setting up a respectably secure operation: set up SPF and DKIM and you're good to go. The problem is maintaining deliverability for anything more than very casual usage of mail into a very complicated ecosystem. At any point in time a random switch can flip on some heuristic somewhere and suddenly a large mail provider will reject you, or you're on a blacklist that provides no explanation or ability to remove your mails, and that isn't isolated: it will spread throughout the ecosystem, and leave a stain that can linger for a long time.
I've run a few-thousand person newsletter mail list for more than ten years, and fairly recently switched to SES because it's simply not possible to guarantee deliverability even at that modest scale running it yourself from your own servers. If you're not hooked into the ecosystem then you have to fight ongoing battles that only a professional in the mail deliverability business has time and connections to understand and effectively win.
Even then, you have to be very careful: don't mention sums of money (e.g. censor all discussion of research funding), don't trigger dumb spamassassin rules (e.g. sentences with lots of long words in a scientific paper), don't talk about known trigger words (e.g. no scientific articles on the role of growth hormone in mouse aging) and so forth.
It is crazy. The absolutely rational response to all of this is to outsource mail delivery.
To make sure that my messages are not treated as spam by the major email providers, I checked that my VPS IP is not in any of the spam blacklists and configured SPF and DKIM records. Gmail spam-flagged the messages for a while, but after a couple of months it learned they're not spam. Surprisingly, other email providers (yahoo, ms, aol) never batted an eye.
I'm quite happy with my own server.
Openspf.org has a page about that [1] which recommends email providers allow their users to configure authorized forwarding sources as exempt from SPF checks on their incoming mail, and default to not rejecting based on SPF when users haven't configured the preference. But in my experience few institutions follow this recommendation.
Once I ditched the SPF records, I've had no deliverability problems running my own mailserver.
GMail was the only one I had that problem with because few mailservers are using IPv6.
Gmail seems to accept mail from an IPv6 mx when the SPF record is "All the A records for all the MX records" (the headers says spf passed), but I'm not sure how others interpret it. Is it better to explicitly list the IPv6 addresses in the SPF record?
I've never really posted it up, but I feel like if you've read this article and are wanting more, this might help you. It's work-in-progress because I've been sidetracked from finishing the Client piece and also creating a Dockerfile that generates a group of self-contained images configured as recommended, but all the steps for the server are complete and tested.
You can find it at http://securemail.tristor.ro/#!index.md It's all written in Markdown and I accept pull requests at https://github.com/Tristor/securemail.tristor.ro
Hope that anyone in the process of setting up a mail server finds this helpful.
I am the author of http://gogs.info/books/debian-mail/chunked/ and I am working on something similar (I have switched to Dovecot and wrote a nice GUI for managing users and domain) but in the end it will be a set of Ansible scripts.
The things I did different:
* Amavisd-new instead of dspam and OpenDKIM
* SQLGrey instead of Postgrey
* unbound instead of BIND as a caching DNS server
I have been thinking on adding PolicyD for rate limiting accounts but then I get a lot of overlap with other services. How are you satisfied with postfwd?
I haven't really worked with Amavisd, SQLGrey, or PolicyD before. I've now switched to using unbound instead of BIND myself, but this hasn't yet been reflected in my instructions since I kind of stalled on updating them.
I like postfwd quite a bit for how I'm using it, which is to do hybrid greylisting based on DNSBL weighting. It allows me to reduce latency for inbound messages that are free and clear rather than greylisting everything. I don't really do any rate limiting or anything like that with it though, so I doubt I'm using enough of it to really form a clear opinion.
Good luck, let me know how it goes, and I'm happy to accept a pull request to add additional instructions for Postgres if you feel up to adding them.
If I did switch back to running my own show, I'd do Digital Ocean + Mailinabox via Docker.
Mailinabox aims to do all the things right out of the box:
When we started blogging the advent series last year (http://blog.fastmail.com/2014/12/01/fastmail-advent-2014/) I had a few support tickets asking "aren't you giving away all your secrets?"
We had a bit of a laugh about that, because we don't really have any secrets. The fact is, it's just a ton of hard work - new spammer tricks, new interoperability woes (ask us about SSL negotiation with other people's MXes some time, but make sure you provide the alcohol), and just trying to get the best possible performance out of the hardware and the software so we can handle both the person with 500,000 emails in their Inbox who never moves anything and the person with 10,000 folders each with 5 emails in it, and everything in between - and provide a fast loading experience for every single user when they hit the site, even though there isn't space in memory to hold that many mailboxes cached.
+ Far more configurable
+ Sieve scripts
+ The web interface is faster
+ Easy to access their engineers (hang around on emaildiscussions.com)
+ The contribute to Cyrus, etc.
+ Family/business accounts where not everyone needs to be an an expensive tier.
+ Standard IMAP, so works a lot better with most clients.
+ Some Dropbox integration.
- Mail gets blocked more often (e.g. there seem to be ongoing issues with Craigslist).
- They don't support labels. If you have a label-driven workflow, you are going to be frustrated.
- Extra storage space is much more expensive than Google Apps (which has e.g. 1TB for $10 per month).
- No real 2-factor auth, there is always an account with full access. Not a possibility to make a restricted IMAP account.
- Obviously fewer apps than Google Apps.
- Relatively slow support times (though I have never contacted Google Apps support).
- The Google Mail client is still much better than the Fastmail client (e.g. reflowing text, when the internet connection is spotty).
o Australian company with servers in the US (and Iceland). So, no real advantage wrt. the NSA et al.
Edit: I just noticed iredmail quotes this article on their front page.
site: https://crankylinuxuser.net
And I secured it down appropriately whilst also using multitude of external scanners to verify what I thought I did. It was somewhat problematic, mainly with the AUTH between dovecot and postfix.
And then I do a RBL search. Some idiot spammer did the dirty deed back in September. 3 RBLs, all lesser known ones. So, I responded as they requested, with a justification of why I should be removed. I did that today around noon.
I'm now on none of them. So the whole "shakedown" issue, at least for me, seems not to be the case.
You can launch the container in seconds and should be the fastest and perhaps easiest way to have a fully functioning mail server. There is a starter guide [2] and even a [3] video to get you going.
However a mail server unlike most other apps is not only complex to install - which we try to address - but also complex to run.
For most if not all users mail just can't fail and this needs a fair bit of knowledge of dns, smtp, spam prevention, security. I think in all but the most committed cases it can prove a bit overwhelming for the average user.
[1]http://www.flockport.com/containers [2]http://www.flockport.com/using-the-flockport-mailserver/ [3]https://www.youtube.com/watch?v=ysUswy8rGwM
I'm using Exim only because it was the MTA I knew how to configure. It's a simple setup, allows no relays, uses TLS when possible, a few bad actors are RCPT blacklisted, that's about it. I use a webmail program I wrote to retrieve and send mail. It's all quite basic.
Have had very little trouble with spam or or being blocked, which may be a benefit of keeping such a low profile. All in all, I've enjoyed the freedom and privacy of running my own email server. I suspect the secret of success is keeping it small, personal and inconspicuous.
I'm just not willing to store all my personal and business communication with a 3rd party. It goes against the entire purpose of the standard, and I figure the day it becomes impossible to run my own server is the day I no longer use email.
I ran a smallish email system for a year and a largish email operation for several years. It's a lot of work to do things right and the benefits are really small. The security related issues presented are very niche -- if you are concerned about data leakage for confidential discussions leaving your datacenter perimeter, you probably need to look at alternate solutions anyway.
Yeah, no.
You're subject to the same risk in a commercial office or colo, btw.
In the US, the Electronic Communications Privacy Act and Stored Communications Act provide protections here... In the event of a civil or criminal litigation, a subpoena delivered to Microsoft will be redirected to the customer or customer's counsel to respond to or quash. If the customer is non-responsive, Microsoft may be compelled to produce data.
There are situations where a criminal investigation will result in a search warrant directed at an individual or organization. Search warrants are a different ballgame... The standards are higher and the cops show up and take everything. Keep in mind in that situation, the cloud has no impact in terms of data privacy -- the police will literally knock down your door or pry the colo cage open to seize what they after.
If the nature of your business requires additional layers of secrecy or offshore servers in Iceland, you should not be using email.
Well it's not entirely true. They could have a strict client side encryption policy. However it could be even harder to manage than mail server.