Hacker News Reader

In my experience, it is not so much the installation itself, but the regular maintenance and the many subtle errors that can occur, which make running your own email server NOT as trivial as you make it sound :).

As for a few examples:

1. Mailservers around the world have different degrees of strictness in what they accept. Some require the sender’s mailserver to have a valid and matching PTR and A/AAAA (!) DNS records. Others don’t care. Some check black lists, some don’t. Some even resolve the MX record of your sender domain, connect to it and try to start delivery of an email to ensure your address is valid.

2. By default, postfix doesn’t warn you about undeliverable messages for quite some time (a week is the default, I think). So if there is any error in your config (e.g. I changed my DNS resolver config, then didn’t restart postfix), your mails will be stuck without any notice, for a number of days.

3. If there are SSL certificates invoked, they should be valid. Some mail servers will not use TLS at all, some will fall back to plain text, others will cancel delivery if your certificate is invalid.

These are just a few examples I have encountered recently, but every time they happen I am incredibly frustrated that emails either did not reach me (usually I detect that quickly) or are not delivered (detected only after a few days).

So this implies there is a market for a simple mailserver that just mimics gmail (or whatever) on your own server:

e.g.

apt-get install gmail-server

Search is IMO the hard part. I know you can do Lucerne, but getting that working well with email is hard. Until recently Outlook's search was crap compared to gmail. too, and mutt is still essentially "headers only".

Still, shouldn't be that hard to do as long as it is your own server and you trust it; I imagine a lot of the difficulty of gmail is scaling, and even a single user with 20-50GB of mail isn't in the same league.

If you were to outsource it to a service provider:

I'd love a way to do privacy-protecting search on my mailbox, either by building/maintaining a local index, or even more amazingly, some kind of cryptographic/data structure magic: do processing once either pre-encryption or on the local device, then add it to an index, with a configurable slider for data leakage vs. search quality. Search could execute locally and remotely.

(This is to allow you to use webmail with no local persistent storage, or a new phone, to search your email on a server, without trusting the server).

notmuch for search.

http://notmuchmail.org/

Agreed. I've handled over 100,000 messages with it with no problem, and I know people that have handled well over that. Instant full-text search, no matter how much mail you throw at it.

How about a good old desktop mail client? I don't really understand why so many people insist on only using webmail and then complain about the lack of certain features and/or GMail lock-in.

> and mutt is still essentially "headers only"

Mutt is more like "everything I can come up with from the command line." Sure, I won't make my grandma use it, but I've never needed any searching capability beyond grep and co.

Not necessarily. Try to do something like write up your email in Markdown, and create a script to automatically convert it to multipart/alternative with text and html versions. So far as I can tell the only 'real' way is to insert the script between mutt and the sendmail command, but that's less than optimal.

Yeah, I end up grepping organized-by-years Maildirs, but that's somewhat suboptimal.

Ever looked at Mairix?

No, looks interesting, will check it out.

Gmail search is actually pretty spotty.

I download my email to the Mac mail app. That has let me find some emails that were unfindable on the web interface (even when I searched by sender email!)

What if there were images on, say, AWS that let you spin up a mailbox instance? Packaging up a standard "I want a mailserver" system might be doable.

doesnt solve alot the issues (blacklists, SSL etc.) and is quite costly for just running email.

But they are solveable. Nice little business idea here.

I've actually wondered this myself. Why isn't there a good preconfigured drop in solution for this? It would be especially nice if it had a good web front-end that you could log into remotely. How hard would it be to develop something like this? I assume it would be a matter of forking an existing project and adding some custom configs. The hard part would be the web component.

Freedom box has talked and hopefully worked on doing this, but Im not sure if they have gotten as far as improving the interface of a webmail yet.

Still, I do think there exist some very talented grafic designers out there that would enjoy playing around with a webmail interface. Hopefully, articles like this will increase the demand and interest in run-it-your-self webmail packages, so one would do that and not spend all focus on building more CMSes.

IIRC, most webmail packages now (SquirrelMail, RoundCube) are just web front-ends to IMAP. It stands to reason that something more integrated, is probably more difficult.

There already are several solutions like this that have been around for years. Check out Zimbra. But still a pain to maintain and moving over to Google Apps several years ago, I never looked back.

Have to agree with this. I ran my own mail server for a few years and even though set up was easy, deliverability was a PITA (I was never able to successfully deliver to hotmail addresses even after doing everything possible on my end).

Then there's spam.. what a headache. Suddenly one day I realised that I have better things to do with my time and now everything goes through google servers.

Did you install greylisting and blacklisting? It really do reduce spam to ~0, and is a one time 5-20m configuration.

As for hotmail, I strongly remember doing a telnet to hotmail servers and succeeded in sending emails while writing commands by hand on my private computer. Thus without any special headers or other magic, I could send email without any issues. It would had been interesting to hear details on why hotmail rejected your emails. Default postfix installation?

I agree with both of you. Mail is hard, not easy. We should host our own email anyway, because the alternatives suck.

I wonder if there is a "Mail Admin Best Practices" doc floating around the web yet?

http://37signals.com/svn/posts/3096-behind-the-scenes-giving...

This is pretty good. :)

There have been a couple of "setting up a mail server" howtos come through here. I recall one that was huge in the depth that it went into (the depth also made sure that I never got around to reading it).

Subtle errors are a problem, as anyone who is a programmer or sysadmin will tell. It's a constant issue when programming in C or Javascript, and it's almost a job description for a sysadmin. Mail installations could clearly be improved here with testsuites that test the installation and not just the mail server code.

But to address some of those concerns.

While errors in the config can cause big problems, it's not very common for people to change it once it's installed. It's like doing changes to the kernel, Apache, or Xorg settings. Sure, things can easily be broken there but for the common case, the defaults are good once one has passed the installation wizard/guide.

In case of SSL issues between mail servers, I am rather sure that postfix only does a best-effort. Thus if your certification expires, they will fall back to plain text. However most CA's will warn you several times once it gets close to the one year mark, so it's not a very big workload to handle. If you know of an email server that will cancel delivery, please let me know as I would be very interested to hear it.

PTR is a real issue, but if you have static IP, PTR records should be rather easy to get. It is also a one time cost for the domain name. For a company network, a static IP and support is commonly included in the price. You email/call them, and they add a PTR. For private users, a VPN solution might be needed in some cases, but I'm unsure if it's that much of an requirement. Gmail seems fine in sending emails to locations without PTR, or receiving emails from servers without one. However I have not tested this fully or with other services like to Apple or Microsoft.

I only have a few small-ish mail servers under my care, and I couldn't agree more.

Reviewing the history of the mailop list [1] shows how frustrating it can be when a behemoth (AOL, MS, etc.) stops accepting your mail. It's bad enough when you're a sizeable mail operation; as a tiny standalone mailserver, you are not at all a priority, and it's possible you'll lose the ability to communicate with a significant fraction of the Internet's email users (regardless of how complete/correct your configuration) and with little recourse.

[1] http://chilli.nosignal.org/mailman/listinfo/mailop

Plus you need offsite backups, for both content and config... it's an escalating problem.

I haven't messed with the defaults on undeliverable messages, but my recollection is that Postfix sends you a "hey, it didn't go through, but I'm going to keep trying" email at 4 hours, and you get the final "I gave up" email about a week later.

Don't forget maintenance of the hardware itself, backups, and everything else that comes with running a server (assuming you didn't buy one in the cloud).

> assuming you didn't buy one in the cloud

Buying a mail server in the cloud is the most practical option. I doubt EC2 would peek inside your VM to censor mail as Apple is doing here.

Running a mail server at home can get complicated. Other than having to manage the hardware yourself, there are limitations on residential internet connections. Not only do residential ISP often provide dynamic IP addresses, they tend to block port 25 (at least outgoing) so as to counter spambots. Some spam filter may also treat mail delivered from your home server more harshly.

Though, if you really can't trust anyone (cloud providers, SMTP relays, or the network in general) you're be better of using PGP.

You are absolutely right about owning your own email, but it is also worth pointing out that no email provider in their right mind would irrevocably, silently delete emails based upon this insane criterion (where a movie script attachment contained a scene where a character views an ad for legal porn).

But Apple isn't in their right mind. They have never understood the cloud, they still don't, and every single Internet service Apple has ever produced, from eWorld to .Mac to Mobile Me to iCloud has utterly sucked gigantic balls.

Which makes relying on Apple to handle your email a lot crazier than relying on say, Google, Yahoo, Rackspace Mail, Tuffmail, etc.

If you have enough experience or make enough simplifying assumptions, anything is trivially easy. But it doesn't take much imagination or experience to see that bootstrapping your own email isn't trivial. For instance, I defy you to buy a domain name and a hosted debian machine without maintaining an existing email account.

As a developer / sysadmin who used to host his own mail and uses complicated half insane APIs every day, I know that even things that are reliable and easy to set up (such as debian servers) require maintenance and all the work and planning that goes into making something secure, reliable, and performant.

I don't see how anyone with experience could contend with a straight face that running a network service like email is a set-and-forget proposition. I remember having to brush up on my mail stack every time I needed to troubleshoot a problem, and over time it became clear that I had over-prioritized hosting my own email.

Edit: It's a fine hobby project though, as long as you're honest with yourself about what you're undertaking.

(1) use public email account to buy domain + hosted debian box

(2) set up mail server, create account(s)

(3) reset public email account with domain registrar + hosting facility to newly set-up email addresses

(4) kill public email account

You only need one for a very short while to bootstrap your own email server, you don't need it forever.

You're missing his point...

But I'll bite, if your mail server goes down - and it will - you now have no email. You can't email your hosting support. You try to log in to your host control panel to do a reboot or raise a ticket but forgot your password. You can't get the reset email. Somehow you manage to login and raise a ticket but won't get any email notifications when the technician responds asking for your server id number...

Get a VPS from another provider (e.g. EC2), restore system from backups, update MX records.

Hopefully this will encourage some people enough to run their own mail servers. It's far less problematic than most people think it is. Maybe it was hard to install and configure this in 1990s, but this is 2013. You buy a domain name, install a Debian machine, and do:

Thanks, but no thanks. I just went in the opposite direction a couple of months ago, switching to a paid Google Apps account so they can handle email for us. I couldn't be happier. Our Postfix / Cyrus / Squirrelmail setup worked fine, but then the server got compromised and taken over by spammers, out IP got blacklisted by most everybody, and right in the middle of trying to fix all that, I got sick and wound up in the hospital.

When I got home, I sat down to start trying to fix this mess and had an epiphany "WTF am I doing this? Hosting email servers is not a core competency for us, there is no competitive advantage to be gained here, and Google Apps is fairly cheap."

I switched over the same night and haven't looked back. After fighting with Postfix and Friends on and off multiple times over the past decade, I'm out. I want no part of hosting email, now, or probably ever again. I have much, much, much better uses for my time than fixing broken email servers.

It always surprises me that people can use a software library with complicated, half insane API's, but can't follow a single page of simple single steps installation.

I have yet to find an email setup that can be described as "following a single page of simple single steps installation". And even if I could, I still don't want to, because there just isn't any real point to it. Email delivery is a commodity, not a core competence. Outsource the hell out of that shit, IMO.

There are a few things to note here:

1) in some industries (like in finance) the regulatory requirements make it near impossible to use external email services. For example, there needs to be a record of every email sent out of accounts in the company that needs to be presented to regulators when they demand it.

2) People within the same company may slip up and send emails to coworkers that shouldn't be read outside the company. When you run the email servers, no other party reads them. It stays in the family, so to speak. When you use google or whatever service, they get a peek at the emails.

3) AFAICT Gmail is not HIPAA compliant, making it inappropriate for health care businesses.

Absolutely true, and I wouldn't argue against hosting your own email is there are such very specific constraints on your organization. We don't have any of that, and hosting our own email makes zero sense for us.

Or to put it another way... we're a 2 person, self-funded, bootstrapped startup. Our most precious resource right now is the time of myself and my cofounder. Every minute we spend tweaking and tuning an email server, is a minute we're not talking to customers, doing market research, working on our product, developing strategy, doing competitive intelligence research, etc. For us, that's not a tradeoff that makes sense.

And yet, here you are on HN responding to this thread. Just a little bit ironic :)

Heh, maybe, in a sense. But realistically, I can't "work" 24x7, and taking a break to check/post HN is part of how I stay sane.

That and then there are periods throughout my day (especially at the $DAYJOB, like right now) when there isn't anything productive I can really do on the startup, and there aren't any pressing assignments, so reading HN is about as productive as anything else. :-)

http://en.wikipedia.org/wiki/Irony

It's very very problematic. With all the spam, your mail server is not going to be trusted by any major providers. Their weak heuristics have no problem banning your 1 user mailserver, but they have gmail whitelisted to not ever ban them.

Agreed, this is the biggest barrier to running your own mailserver.

Even if you think deliverability is good for 95% of mail servers there's always some ISP that won't cater for you. So you're constantly having to be monitoring the logs to ensure your emails are being delivered.

Working professionally for a small company, which provide emails service for customers domains, i can say that this has never happen for us. not a single time.

Sometimes people get hacked, either because they pick a abc123 password (nowdays blocked by policy), or because a customer webserver CMS theme has not been updated in the last 2 years. When that happen, a few mail servers (mostly yahoo) has temp banned mails for a few hours. Mails still got delivered however.

So lesson to learn there is to not use passwords like abc123, and to actually update that 2 year old CMS theme. Or you don't combine webserver with webmail.

In practice this is nonsense. I have run my own email server since 1999 and "trust by major providers" has not been a problem for me.

I guess this is one of those YMMV cases. I could never deliver to hotmail addresses no matter how many hoops I jumped through.

Also: how do you know you've never had deliverability issues? If the server on the receiving end is blocking your mail it will happen silently.

I agree with you. Most professional email services have much less aggressive blacklisting than single-user email services run by their single user.

In many years of running my own server, this was never a problem. I checked blacklists on a somewhat regular basis but was never in them. And people I sent mail to usually replied, which means they are either psychic or my mail got delivered. (The problem comes when you start sending spam, of course, which is what many people do get blacklisted for.)

The problems with running your own email server are spam and the time it takes to keep things running. All the other issues are trivially solved.

yes yes yes! I 100% agree. It would be interesting to hear from someone with some legal knowledge. I suspect in many countries it's the case that if you host your email with a third pary (iCloud, gmail, FB, etc) you give up some rights to privacy ... whereas if you host your email on your own server that you physically control, for example in your personal home, your rights to privacy are greater.

Even on Ubuntu Server it's very very easy to setup a mail server. Linode has guides, and there are others on the interwebs as well.

Just make sure to implement a regular backup regime.

In my experience IMAP to my own mail server is WAY faster than dealing with Gmail (even web interface).

Note that if you host on a shared machine or a VM, then law enforcement is still likely to see it as your host's box with your data on it, rather than your box with your data on it.

The most conservative approach would be a co-located machine that you own (i.e. your physical property, so they probably need a warrant) or on a machine in your home.

I actually tried many times to setup a mail server on my debian machine but never could find a good tutorial. I gave up because I lack the time.

edit: I just tried again, for more than a couple of hours. First I had to guess that I had to install MDB2_Driver_mysql. Set up a DNS to match my /var/lib/roundcube and now I'm struggling to understand how I could sign-in since he didn't create any user for me.

Regarding MDB2_Driver_mysql, I point towards (http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=544694). If you want to use mysql and not default sqlite, use debian package roundcube-mysql and it will install the dependencies for you.

As for users, roundcube uses imap and thus do not create new users.

Anyway, GL, and if you want to have some fun, try out PostgreSQL.

//" install a Debian machine, and do: aptitude install postfix aptitude install postgrey install either a webmail (roundcube most commonly), or go to (http://www.postfix.org/docs.html). If a spam gets through, install blacklisting or spamassassin.//

You seriously think this is simple for non IT people?

No, I don't think this is simple for non IT people. Same goes for my web site code, or my python projects, or my shell scripts. If I gave either of those to an non IT person, they would have no clue what is going on.

This is as expected, as most people require some expertise to do technical work. I would have no idea how to fix the electrical system in a car, but I do expect that the mechanic knows how to figure it out if it one broke. Same, I expect a sysadmin or a programmer has the expertise to follow single steps installation guides.

It't even that simple for IT people. I've been working with email servers for years and ensuring deliverability was a NIGHTMARE.

>When you let someone else handle your email, they own your email. There is no postal secrecy law, no rule against reading. They can touch, modify or delete according to their whims. It's their email now.

In Germany, email is actually protected by the Fernmeldegeheimnis [1], even while the email is saved on the provider's server. The reasoning behind that [2] seems to be that the mail server's storage is part of the communication process, no matter how long the data sits there.

[1] http://de.wikipedia.org/wiki/Fernmeldegeheimnis#E-Mail [2] http://www.bundesverfassungsgericht.de/pressemitteilungen/bv...

I agree we should be more serious about email privacy.

But "There is no postal secrecy law, no rule against reading." Seems like something of an exaggeration.

See: http://en.wikipedia.org/wiki/Electronic_Communications_Priva...

I've written about this before, but I used to run my own mail server and it's no cake walk.

I used to do greylisting with spamd and ended up silently losing quite a lot of email. (Many mail hosts do not re-send from the same IP, meaning messages essentially get stuck forever.) Doing spam checks at DATA time and rejecting obviously bad emails outright seemed much more effective and less dangerous. I never managed to get Spamassassin to do this, but auto-rejecting languages you don't read also cuts down on spam a lot.

(I missed out on a consulting opportunity because the client's host of choice seemed to be a known spammer and my mail server's filtering was too aggressive about trusting blacklists. I chose to silently-reject those types of messages, so nobody got a bounce. Fortunately, someone was nice enough to ask me about it out-of-band so at least I was able to turn off greylisting and blacklisting before losing much more mail.)

I never found an imapd that scaled to having a lot of messages in a folder, so I ran a cron-job to move mail offline after 2 weeks (for mailing lists) and 1 year (for INBOX). Similarly, I never found a good client to use; Gnus had a very cryptic configuration that I could never believe worked, and mutt was not Emacs-y enough. Reading my email mostly consisted of waiting for Gnus and deleting spam. (I never figured out a good way to get Gnus to move messages marked as spam somewhere so that I could run a cron job to automatically train Spamassassin on the known-bad emails.

For search, I used HyperEstraier: http://sourceforge.net/projects/hyperestraier/

You set it up to run the ingestion program as a cron job every 10 minutes, and then you get a CGI that will show you matching emails for your query. Of course, you can't actually click the links and go anywhere unless you set up some sort of web-based email viewer. I never found anything I liked so I lived without webmail. (There are lots of options. All difficult to configure and probably riddled with security holes.)

Once your server is up and running, you need a secondary MX and a backup plan for your email. (I used Dyn.com's secondary MX hosting service. A lot of spam comes in through the secondary MX, so you can't just implicitly trust it. This involves more configuration.)

Finally, spam filtering uses a lot of CPU and RAM, so you have to pay for a rather expensive virtual machine. Linode's $40/month plan seemed mostly adequate.

I don't really like composing email in the web browser, but I've gotten used to it and $5/month for Gmail and $0/month of my time screwing around with spam filters seemed like a good tradeoff.

I use postfix without greylisting, instead I drop the message if it is going to a non-existent user (drops about 99% of all mail), then drop it if it comes from an invalid domain (there goes another 0.5%) and then a block list (cbl.abuseat.org). The rest is valid email, which gets spam/virus scanned and delivered to my mailbox. Not very CPU intensive at all...

Then it goes through amavisd that does the spamassasin checks, as well as verify DKIM and the like, and then it gets delivered to dovecot.

Now, Dovecot as an IMAP server is fantastic. Along with dovecot-pigeonhole I can sort messages into different folders server side.

Dovecot currently handles one mailbox for me that I archive a mailing list into ... 150k messages and counting, and no issues. Uses Maildir on the backend. Although, I do think at those sizes it comes down to good file system caches, and a good file system that doesn't have a problem caching the entire directory.

It works well for me, so well in fact that I am moving all my stuff from Google Apps for Domains back to in-house. For two reasons, 1. more control, and 2. I have had issues with Google Apps in the past, and even with a paid account the support has been lackluster. I'd like to know more about my email, be able to check logs if need be to see what is going on.

I used to have same reaction regarding greylisting, but changed my mind around 5 years ago. Have yet to see any issue with it, and I assume this is because homebrew email servers are no longer common or even uncommonly in use. I guess it could happen, but I have yet to find this at work for those I host email for, and same goes for all my private emails. It can be that because many commercial email providers uses greylisting, users of homebrew mail servers has either fixed it or gone over to postfix/exim/sendmail by now.

Putting a scalpel in your drawer is a simple one-step operation. Anyone can install a scalpel.

Using it effectively, and knowing the dark corners of scalpeling, is quite another thing. It would be easy to get into deep trouble with a scalpel.

This is only slightly more reasonable than handing people an axe and sending them into the woods because their landlord won't fix a leaking pipe.

Or maybe we need "dropbox for email" YC S13 company.

> spend 5-10-30m tops to do an email installation Hahahahahahhahahahahhahahahahhah

As an alternative, are there any secure email services on the web we can use instead?

The biggest issue is your domain getting blacklisted for whatever reason. It may look like the name of another spammer's domain name. The server you rent may be colocated on a rack with a spammer's box (or a box owned by spammers) and have weird reverse IP SNAFUs. Your very server may not have the latest 0-day patch and bet 0wned by script kiddies and, bam, again... Blacklisted.

Note that I do NOT know much about email servers and that is precisely the point. I eat iptables rules for breakfast, I can set up a Debian server from scratch and I can certainly configure a mail server. But there's more to it than meets the eye. Been there, done that. And we got burn.

I don't know "why", but all I know is that we had problems!

tl;dr: 30 team person startup and we tried to use our own mail server. Too many problems. We moved to Google Apps for domain and all our problems were fixed...

At conferences, especially security conferences, I tend to ask if anyone ever heard of a Debian base installation being hacked if it has auto-updates or is manually updated once a week. So far, my count is 0 and when asking self-proclaimed hackers, they also tend to agree that Deiban stable installation is very secure. The installation I got from 1999 is still running, and the count of breakins are 0. to add on that, it has no firewalls. I kind of wonder what happened that caused your server to be hacked. Insecure password?

As for using a previous owned spammer domain, I strongly question that logic. Black lists tend to be both conservative and done by copy-pasting. A blocked domain abcd will not block abc or abcde. If the block lits do not see mails from abcd within a month or so, they also tend to remove them to clear up space. This time frame is also way below common domain name expire dates.

Regarding reverse lookup, I assume that 30 team startup bought a company network contract from the ISP. Such deals should include support and static IP, which mean reverse lookup is a single email away from being done. Co-location contract might be different, but I wouldn't know.