How to set up a mail server on a GNU / Linux system | Hacker News Reader

How to set up a mail server on a GNU / Linux system

flurdy.com

I know it's a bit out of subject but I had to do this a couple time, and now, I simply use google app. It gets the job done, have a nice spam filter and more importantly, users are already used with google services. I'm not saying configuring the server with postfix is bad, far from that, but it might be worthwhile to look at google app as it may speed up things for you and your users.

I use Google Apps for personal email, but I was thinking of building in secure and monitored messaging to an idea I'm developing. In order to do that, I need to setup my own SMTP server.

Not necessarily. Isn't that what services like Sendgrid have popped for? I'm actually I'm the same boat as you with needing an app to send various notification emails which is why I'm reading this thread so I'm about to check out some 3rd party options now. I don't know about you but I always assume I'll have little to no users so I got a 512 Linode figuring I'd spare such a small server from doing so much work and buying a bigger slice when and if necessary. Besides the complexity of setting up an email server so your messages can be reliably received I'd go with a 3rd party to save the server from the load.

However, this is a great alternative to people looking to host their own, with some of the recent moves Google has been making. Also, it increases the bus factor of your communications, which is never a bad thing.

I agree. I've been on Google Apps for about five years now, and I routinely configure an apps account for almost all domains I buy. Never had any problems. Never having a "the e-mail is down" call while on vacation (or, really, anytime) is awesome.

I enjoyed "The Five Stages of Hosting" posted earlier (http://news.ycombinator.com/item?id=3526767). In that metaphor, Google Apps is the large, modern and comfortable house in a central location with most amenities and virtually rent free - only downside is that the landlord won't let you knock down the walls, and there's no way a grand piano will fit through the front door.

And that the landlord admits to coming into your house daily to read through all your mail and documents in order to sell you stuff. And then follows you around town to catalog what you buy, what videos you watch, who are your friends, what are they doing, and what information you're looking for, among other things because, you know, this will all make it a better experience for you.

That is exactly why I run my own mail server.

Also Google Apps broke on me and their support is absolutely utterly useless.

And when something gets blacklisted, TSHTF. It's much easier to sort it out yourself.

You forgot to mention the benefits of vendor lock in.

There is none. You can download all your email any time in an open, industry standard format (IMAP).

Calendar and contacts, as well as integration with a few other bits, provide a slight measure of lock-in. I'd still have to say that it's lower for Google Apps than for a lot of other products out there.

Actually, the Google Docs integration is probably the biggie.

Re: Contact Lockin: Your contacts should be in the Directory Server and synced to Google Apps. Any first class platform you will migrate to will sync to LDAP.

Re: Calendar I've done two calendar technology migrations for 500+ person companies - they are typically forklift upgrades done over a weekend. You basically lock in your resources (rooms, typically) a week ahead of time, have people rebook any forward meetings into those resources - and have everyone switch into the new system on Monday. As long as people have the right client (in Googles case, that would be a "Web Browser") there is no lock-in.

I could take a 500 person company from Google Calendar, Email, and Contacts over to Microsoft exchange with a team of three people in under a month, with maybe 2 days of disarray (monday) as people (who ignore instructions the previous week) update the mail servers and LDAP servers on their various Androids, iPhones, Macintoshes, etc...

Just make sure you keep your primary directory in your own LDAP server, and you will be good to go. Don't outsource the directory. And stick to something LDAP compatible.

Yeah, it's reasonably light, as I said. Training and end-user usage patterns (never something to be taken lightly) are probably your biggest issues.

Then again, the user experience with the leading alternative solution (MS Exchange) is so miserable that at one organization I'm aware of, the public announcement of a migration to Google Apps for Domains was greeted with a standing ovation.

Another interesting factoid: Hal Varian, co-author of Information Rules, which largely discusses strategic use of lock-in by both vendors and users, is Google's chief economist. I suspect this is a subject the organization understands well: http://www.amazon.com/Information-Rules-Strategic-Network-Ec... http://en.wikipedia.org/wiki/Hal_Varian

Yes, of course, there's that.

I disagree with the metaphor of the landlord reading your mail: As slightly worrisome as it is, having a person and an algorithm read your mail is not comparable. Of course, the risk is that the results of the analysis is leaked outside the algorithm.

I have taken a conscious decision that I don't care about this risk. I'm putting my money (yes, it's a bet) on Google being able to profit better from advertising to me without violating my privacy. It's a "life's too short" trade-off.

I've got a how-to for relaying SMTP mail thru google apps on a CentOS server using postfix here:

http://www.zulius.com/how-to/set-up-postfix-with-a-remote-sm...

That setup seems to only be useful for a local setup (i.e. a laptop with only one user) because you are setting up a single username/password for the gmail smtp server. You would only be able to send mail from a single gmail address, or am I missing something?

I've tried doing this (or something similar) - using the google SMTP server to send mail, and since the server sending the mail was in a different country to me, it thought that I was being hacked and blocked me from using the account.

Personally, I hugely prefer exim+dovecot to postfix+courier. This guide is the one I follow: http://hinterlands.org/wiki/index.php/DebianEximDovecotSquir... (although I'm not a Squirrelmail fan at all).

IMO, I agree with dovecot over courier, but any reason you prefer exim? I'm under the impression that postfix is regarded as one of the best examples of secure software to date.

Habit, mostly. The first mail server I ran for myself was postfix, and I managed to get in a nasty tangle. Exim configuration just seems to fit my brain better.

While in college, my housemates had a Linux mail server configured with individualized spam filtering; every user had an IMAP "spam" folder and a "ham" folder; they could move false negatives from their inbox to the spam folder, move false positives from spam to ham, and a nightly job would run and generate custom statistics for each user. It was remarkably slick, and for years I've been trying to figure out what that setup was. Does anybody have any ideas, or actual links to similar tutorials?

I'd just e-mail them and ask them. (And hope that your email doesn't fall into the spam box :) )

The "spam" and "ham" IMAP folders correspond to a certain file or folder(1) on the server. Email users have the learning commands in crontab. These are as simple as "sa-learn --spam /path/to/spam_folder; sa-learn --ham /path/to/ham_folder".

(1) file if the backend is mbox, folder if the backend is Maildir.

We did this for my old company, with Cyrus IMAPd and spamassassin. The mail directories are regular directories on the server, and then it's just a cronjob. I regret that we didn't think of Google's priority inbox, which would have been trivial to implement!

Make a cron job run this? http://spamassassin.apache.org/full/3.0.x/dist/doc/sa-learn....

I used this setup for years before switching to google apps using dovecot and dspam.

Doing a quick search this seems to be very close and up to date. http://www.owlfish.com/thoughts/dovecot-antispam-2011-03-21....

I do something similar for my customers via Roundcube/SpamAssassin. With a little bit of effort, you can get SpamAssassin to store its Bayes statistics in MySQL, and then you can have a "mark as not spam" and "mark as spam" function in Roundcube which ties back to sa-learn which will update the per-user Bayes data in MySQL.

Getting it to work right was a little fiddly though.

It probably was using a bayesian filter of some sort. There are many, I use bogofilter because my email app comes with it, and there is HNer jgc's popfile.

Actually setting up a multi-user SMTP and IMAP server on Linux is straightforward if you've done it before, or it is here-be-dragons stuff if you haven't. (Your distro might configure it all up for you prepackaged, or you could be in for a lot of reading.) Integrating the bayesian filter is probably the easy part.

If it is only for you, it is perhaps easier to integrate it with your email app rather than at the server level. Popfile will do that.

This is useful. I will review.

I have traditionally used the instructions here which are very mature and have done me well for years:

http://workaround.org/ispmail

They are absolutely bomb proof.

Yeah, this one is a bit improved stack, because it contains fewer pieces in critical places (ie dovecot can do all the authentication + delivery).

Does anyone here recommend building your own email system other than it being more private than hosted email? Are there any benefits (outside of privacy) to putting up your own email server?

... and on a -somewhat- related note: Does anyone here recommend 'premium' e-mail hosting, such as Rackspace E-mail? (http://www.rackspace.com/apps/email_hosting/rackspace_email/). What are the benefits of such services? I am trying to understand the benefits of these premium e-mail services to a postfix on a self-managed VPS.

The obvious one is that it saves you the hassle of running postfix on a self-managed VPS.

And you do get rather more than just postfix - there's webmail, IMAP, etc, all managed for you.

The obvious benefit is they manage it for you.

One major advantage of premium hosting services (and other SaaS providers such as SendGrid) is that for outgoing mail, your mail will be delivered and not marked as spam/blacklisted/dropped silently.

Because of the low volume of outgoing mail from a VPS, it's very easy to end up on blacklists.

Wait, how does sending little emails can get your server blacklisted? Is it because they don't care about the small guy, or is there some technical reason ?

It's really about how much a false positive affects your IP's reputation.

When you are sending out a lot of email, recipients see a volume of mail. If a few of these get marked as spam, no big deal, you sent out 5,000 mails in an hour, of which 2-3 were marked as a false positive.

If you are running your own VPS you are sending out comparatively fewer mails so the decision on whether you are a spammer or not is made with a lot less information -- and most recipients will assume you are a spammer immediately.

It's mostly because you're not yet proved to be NOT spam. When you start a new mail server on an IP there's zero diference between a legimate email server and a spamhost. Your IP will not be in spamhaus and other blacklists, but you might get stuck in McAffee and other more enterpricey filters, especially if some other IPs in your subnet are known to send spam.

What tildedave said.

I was on the wrong end of blacklisting situation and it was extremely frustrating. First, you don't know that your customers aren't receiving the emails. Second, you have very little clout in getting things cleared up. Third, it's just a huge PITA that wastes a lot of time.

I'm running a very small operation, but one bad instance of my IP being blacklisted prompted me to shell out the extra $3 /months to have Rackspace send my emails.

I have used pobox.com for years for email. They have excellent spam filtering. They also have excellent remediation for the few false positives I've had over the years. I can also only remember a couple of times they've actually been down that I have noticed, and I access my email fairly constantly (at home and via my phone)

If you only need a small handful of email addresses, often the hosted service can provide those for a lower (or similar) monthly cost to that of a VPS (assuming you're dedicating the VPS to just email).

The blacklist issues others have pointed out are real. Some IPs are on blacklists and you will not be able to get them removed without an outrageous amount of leg work. Most of the premium email hosting places make sure they're not on those blacklists.

Personally, I use FastMail.fm. I've gotten the impression that if TSHTF, they'll fix it fast. The price is reasonable. They're not Google. When I get a website up that can't be served via static pages, I'll be going the VPS or other cloud service, but for email, I'm much happier leaving that to a premium provider.

I want to insert a custom app into the email pipeline. Is there any other way?

Sure it's possible if you host it yourself. Where do you want to insert it?

He may have been replying to me by setting up a situation where hosting your own email would be beneficial.

I did mean in the hosted case. I want a web hook.

I'm not sure what you mean by a 'web hook.' Can you elaborate?

Each time I recieve an email, I would like my app to parse and then POST the email to a web server in real time.

If you want to be part of the e-mail pipeline, your application should use SMTP for both sending and recieving.

Use a spam service such as spamhero.com or google postini as the MX for your domain so you do not have to do spam filtering yourself, have the spam filter service deliver the mail to your application (over SMTP). Your app can do whatever processing it needs to do, and then deliver the mail to the MX server of the ISP you're using to host your mail.

This way you can use your app without any changes as a local smtp filter as well (e.g. if you're using something like amavis).

I'm not sure I agree. Integrating into an existing server, such as Exim/Postfix will give the security benefits with a much lower latency and lower bandwidth costs.

Sendgrid's Event API (http://docs.sendgrid.com/documentation/api/event-api/) does something similar to this. Their mail servers act as a gateway to allow your app server to both send and recieve email over HTTP. I've used it at work as part of a not-yet-released feature of our product.

Real men write their own email servers ... kidding ;P I think most people want the parse api. That is doing something when an email received. We were going to use that and the event api ourselves but I ended up writing my own custom email server with subethasmtp (in hindsight the lost opportunity cost was probably not worth it.)

I wrote an email server to do this but I highly recommend instead of using a RESTful POST you consider using a message bus like RabbitMQ. That is what we do. We have our custom email server push it to a Rabbit queue and then our app picks it up. That way if you shutdown the app you won't loose any emails.

There are multiple ways:

* use a service like http://mailgun.net/

* install http://lamsonproject.org/ on your server and define a forward from your hosted solution to your server

* run your own mail server and use something like procmail to execute scripts or combine it with lamson/something similar

Sure there are benefits with having your own email system versus a hosted solution (ie. you have the data on your server(s)), but it comes with maintenance headaches too.

We recently switched over to Atmail (http://atmail.com/) after years of running Cyrus+Postfix (with various anti-spam tools). Atmail is kinda the best of two worlds -- you have control over your data, but the software is easy to maintain. Under the hood, Atmail is using standard Unix/Linux components (dovecot, exim, spamassasin etc), but makes it easy to maintain.

For a few other companies I'm involved with, I've deployed Google Apps, just because it's dead simple and gives you no headache.

I've always run my own email for several reasons:

1. I can choose my domain (although nowadays that's not a problem - it was when I first started).

2. I can tweak spam and anti-virus filtering.

3. I can view the logs if I think something is wrong.

4. I can be as (in)tolerant of other mail servers as I want (I've noticed that Google is a bit lax in the rules it enforces with regards to rejecting mail).

5. I don't trust Google or other providers not to mine my email, or to back it up.

6. It's a useful learning experience.

Earlier I had used dovecot+postfix for my personal email system. The advantages are that if you roll your own you can do whatever the heck you want.

Huge disadvantage (other than the fact that power, static IP etc are all your problem) is that a LOT of email providers will think you are spam.

Now I use SES for outgoing (for my app) and Gmail for incoming. Works a lot better. I remember reading that Gmail lets you write filters and what not as well, but I have not examined it in detail. I'm sure its do-able.

The size of that guide illustrates why I switched to google apps + gmail. I was using a different stack based on FreeBSD, qmail, spamassassin, imapd and tcpserver, but about equally complex. It required regular maintenance and took up too much mental space.

The advice I wish somebody had given me: Don't use Debian's default configuration for Exim. Even though Debian's docs strongly recommend that you should do so. I'm sure their magic config stuff works brilliantly for somebody, but for most people it creates a vast, complex set of files that then breaks when you upgrade, leaving you with no email. Just write (or find) a config file.

This (almost) holds true for (almost) every piece of software.

I just recently setup an email auto-responder with a custom email server (in Java) and I will say it was a goddamn PITA. I did use the cited article and also this one is good: http://rimuhosting.com/support/settingupemail.jsp?mta=postfi...

Some things that are rather important:

1. Name your email server with "mail" some where in the name. Seriously don't call your email server crapbox.snaphop.com :)

2. Although you can send email as a relay on many different ports (2525, 8025, 587 (ssl)) you can really only receive on port 25.

3. You better have a PTR / reverse dns setup.

4. It takes at least a couple of hours before email servers will acknowledge you.

5. You can get away with out a SPF for a little bit but you really need if your going to blast a crap load of emails. Use this to test: http://www.kitterman.com/spf/validate.html

A couple of more gottchas are:

6. Some (most?) hosting providers are black listed and getting other email systems to recognize you as a valid agent is getting tougher and tougher.

7. You have to provide your own fail-over, or purchase it from someone else.

If you can live with delayed messages there is not really a need for fail-over as most server will retry multiple times (default iirc multiple days). But you should have at least another email address available for example to communicate with your provider.

8. domain keys (DKIM) ?

did this recently on EC2 and found these helpful

http://pauldowman.com/2008/02/17/smtp-mail-from-ec2-web-serv...

http://www.practicalclouds.com/content/guide/sending-email-e...

DKIM on Ubuntu - https://help.ubuntu.com/community/Postfix/DKIM

Microsoft SPF record creation wizard - http://www.microsoft.com/mscorp/safety/content/technologies/...

not sending that much mail, but noticed after setting up SPF/DKIM/reverse DNS that the mail I sent to myself didn't end up in spam folders at gmail and hotmail etc., without having to send via a service like authsmtp.com.

People still setup mail servers? amazing, I actually thought I was the last hold out to host my own email. I used to be all crazy into qmail (and then later postfix), with spamassassin, clamav, etc. It wasn't until about 5 years ago I ran into Surgemail while reviewing solutions for a former employer. http://netwinsite.com/surgemail/ it's worth a look if you are into hosting your own mail. I rarely, if ever plug commercial software, but these guys have really earned it. From their awesome support, development, and licensing model, it's just all around impressive.

I have much better recipe. You go to zimbra.com, download ZCS Open-Source Edition and immediately have all the functionality out of the box. As a bonus you have much more beautiful web-interface than ancient Squirrelmail.

Squirrelmail is indeed ancient looking. Years ago when I last ran my own mail stack (postfix + cyrus) I wondered why there wasn't a gmail-clone mail webapp. Perhaps there is now?

There's Roundcube, which is reasonably good:

http://roundcube.net/

For anyone thinking of running a private mail server off their home DSL line, don't do it. It's very likely your IP (even if static) will be in all sorts of spam block lists.

Instead, either use your ISPs mail server as a smarthost (if they allow it) or (better imho) get a small VPS of your own (e.g from Linode) and:

1). Install open dkim and set it to sign all your outgoing mail (make sure to add the relevant DKIM TXT records to your domains)

2). Add SPF records to all your domains

3). Make sure your server's IP has reverse dns setup

Your mail should then happily sail through all but the most brutal of spam filters.

I use my home server as the primary MX, and a VPS to relay outgoing mail via TLS and serve as a backup MX. This allows me to keep my email on a server only accessible to me, while avoiding the blocklist issues when sending mail.

Long ago I installed a small server for a university project that ran on linux (slackware with many customizations), at the time configuring postfix, dovecot, exim, etc. Was a pain and probably not worth my time for what it was used so we decided to go with Hula which at the time looked promising, was dead simple to get working and fairly performant given the low end spec of the server (P4 1.8Ghz, 256mb RAM).

Later on Hula struggled badly with Novell, was sold and forked, we tried to check out the forked version (Bongo I think) but so far the project seemed dead.

After a while we did a clean setup after failing a distribution upgrade (but hey 5 years updating without hitch on custom kernel and compiled software), we moved to debian to lessen maintenance, email was done via postfix+dovecoat+postgresql, which was a hassle (to say at least) to configure, funny enough this setup did not perform well, looking for a more consolidated solution we found Apache James, which looked fairly promising (being under the Apache foundation), the only downside was that it was written in Java, not that I have anything against it but that is another vm to install and maintain, we gave it a try and we have been very satisfied with the results, easy to administrate, very sane defaults, relatively easy configuration in case of tweaks (having it use our postgresql db for users was pretty easy compared to postfix/dovecot), it sucks a lot of memory but the machine overall feels even faster that with the postfix/dovecot stack.

So yeah, while its nice to have an email stack that follows the Unix philosophy, it can get very unwieldy for simple setups (while it may shine on complex setups where flexibility is needed).

It's funny that I've just switched from Google Apps to my own email server today. I've used GApps since 2007, but I've been considering to move away for some time, mostly due to privacy concerns.

It only took me a couple of hours to set up an encrypted VPS with Postfix and Dovecot, change the DNS and test that everything works ok. I'm somewhat lucky that I don't get too much spam and Thunderbird does a decent job of filtering it.

'Encrypted VPS'? You have an encrypted container on it? Have it installed in an encrypted container? Encrypt all incoming mail?

I run a UML VM with full disk encryption on a larger VPS. I only use it for email storage and services. MUAs connect using IMAPS and SMTPS.

Install the same in a couple of minutes with http://www.iredmail.org/

I don't want all of it though. You shouldn't be running a web server on your email server for instance.

Web server is installed for the web control panel, which can be disabled afterwards.

I do like step by step guides as they provide a higher level of control, but after you've performed a setup like that 2-3 times, you start to lean towards more automated installs as iredmail or custom written scripts. So why maintain your own scripts when you can start with a ready made package.

And the webmail...

You can do the same with Symbiosis, which is just a bunch of Debian packages on top of lenny (and soon squeeze):

http://symbiosis.bytemark.co.uk/docs/symbiosis.html

For personal projects I set up Google apps to handle inbound/outbound email for an admin email. Then postfix on the server in a null client configuration for outbound email only (eg. no-reply) type registration emails and so forth.

Off topic: I hate noreply email addresses. Why not send it from yourself or customer services?

The only valid reason I can think of is minimizing PLEASE UNSUBSCRIBE from your customer service spam filter.

For receiving email, why would you want to donate your bandwidth costs to spammers? I understand the desire to use Postfix though, so just use Gmail, or even Pobox (which has adjustable spam settings), then forward to your own server with some whitelisting in place. Then at least you've cut your bandwidth by about 99%.

For sending email, there are some steps for DNS and signing missing here that can result in your email going to spam with the major ISPs. It looks like he was going to add Domain Keys in v6 but abandoned it.

http://gogs.info/books/debian-mail/chunked/

This one was written by me, for Debian Squeeze...

I've become a very big fan of www.zarafa.com over the last year or so. It's open source, modular, works hand in hand with postfix, supports activesync phones, has a great webmail interface, strong documentation and, best of all, fully supports Outlook 2010 clients connecting in Exchange mode (if you want that).

IMHO, it's the best opensource groupware offering out there.

In case it helps anyone who is like me and isn't concerned about the most secure or optimal setup for all aspects of email and just wants the easiest way to be able to send mail on Ubuntu, what worked for me was this:

sudo apt-get install postfix

Then hit enter a few times to select the defaults.

Whenever I see a tutorial like this I think: "Why isn't this a script?".

Reply on news.ycombinator.com