Ask HN: Why is it still so hard to host your own email/calendar that just works? | Hacker News Reader

Ask HN: Why is it still so hard to host your own email/calendar that just works?

91 pointsthalev83 comments

It seems to me that email and calendar data are so personal that you shouldn't have to trust someone else to store and handle it for you. Yet when I try to find a way to host it for myself I get scared of the trickiness of the setup or the amount of maintenance required. Am I just looking for all the wrong things or is it in this day and age not possible, even for a reasonably tech savvy person, to host your own email and calendar securely? With securely I mean secure from all adversaries except for maybe nation-state adversaries. [edit] Like many people have pointed out in the comments an even bigger problem is being blacklisted by other ISPs. Any ideas how to deal with this while still maintaining control of where your emails are stored?

View on HN

This post [1,2] from last October discusses a major problem with hosting your own email, which is that the big email providers pretty much reject your messages until your "reputation" is good. The reputation concept is purportedly a spam fighting effort, but a more cynical view is to see it as an effort by the big providers to sabotage self-hosted email. More discussion here [3] and here [4].

[1] Original post, now defunct: http://liminality.xyz/the-hostile-email-landscape/

[2] Thank you archive.org: https://web.archive.org/web/20151121132739/http://liminality...?

[3] https://news.ycombinator.com/item?id=10405681

[4] https://lobste.rs/s/ckfyqd/the_hostile_email_landscape

Self-hosting is dead unless you are massive business.

I have a very small (on the side) hosting business that has been running for almost 6 years now and growing little by little. I have direct contact with my clients and they have it with me in case of need (if they need technical advice or support not necessarily directly related to the services).

And this is exactly the kind of service they were looking for, very difficult to have with a massive business company (have you ever opened a ticket with OVH?)

Then of course, the technical challenges to setup and maintain such services are high and a control-panel all-included package or a docker with everything in it is just a very small part of the work required.

Self-hosting email or self-hosting in general?

Email, but I would not be surprised if self-hosting in general become in practice impossible - self-host and your site only shows up on page 100 in Google.

Well this seems to be a dealbreaker, I can't have an email server that is likely to get blacklisted.

I posted that in part hoping that someone would chime in with a workaround/solution figured out in the 6-7 months since that discussion.

I happened to post comments in GP's cited thread and also another email server thread. You can read both of them to get a deeper understanding that personal email servers are hampered by "trust" issues more so than technical challenges.

https://news.ycombinator.com/item?id=10405945

https://news.ycombinator.com/item?id=10498988

Basically, you could conceivably get past all the technical set up. There may even be Docker containers or virtual images to jumpstart most of the set up. Or you can follow some step-by-step instructions.[1]

However, the problem is that SMTP servers originating from residential IP addresses by default are treated as home computers that have been hijacked by a malware spam bot. Using Bayesian reasoning, this is a rational filter because the number of zombie home computers sending unwanted spam vastly outnumbers any tiny amount of "good" people setting up legitimate SMTP servers.

Given today's realities of distrust-by-default (unknown SMTP server is guilty until proven innocent), it looks like the best strategy for a "personal" SMTP server is to pay for a virtual machine at a hosting provider (Rackspace, etc) whose IP address ranges are not blacklisted. You could then run encrypted email storage there so Rackspace has no visibility into private emails.

[1]http://arstechnica.com/information-technology/2014/02/how-to...

I run email for most of my web clients. We have never been blacklisted and my email servers were set up without any help but the docs from day one.

There are a number of things that can (and should) be done to prevent outgoing emails from your SMTP server not to be rejected or flagged as SPAM (or worse your IP address blocked).

A part from the obvious basics (IP reverse lookup working, TLS support, SMTP auth for your users..) you can also set up DKIM [1] and SPF [2] that help a lot and are well supported by big providers like gmail.

But more importantly not ending up in a blacklist is important, so if you have users using your service you should monitor the volume of outgoing emails to avoid someone using your server to spam.

There are a few tools that you can use to check your SMTP server MXToolbox and CheckTLS are the two I often use. [3]

I agree though that managing an SMTP server is challenging, I had a couple of issues I had to work on that I am sure would never have happened to a big provider.

[1] http://www.dkim.org

[2] http://www.openspf.org

[3] http://mxtoolbox.com http://www.checktls.com/testreceiver.html

You can do all of these things and still a certain percentage of your emails will disappear into the ether. The game is rigged and the little guy has lost.

I'm willing to accept email is rigged. Who uses it for actual communication these days anyway? I use mine primarily to get lifecycle emails from webapps I tried once and to reset passwords.

What's next, then? There have been lots of replacements for all of the old school protocols like IM and IRC. Even newsgroups have been (adequately?) replaced by forums and link aggregates and facebook. But, I've seen no attempt to replace email. I've seen lots of attempts to enhance email, but none to replace it completely.

>Who uses it for actual communication these days anyway?

People outside of tech bubbles.

People outside of tech bubbles use snapchat, facebook or whatever chat program is popular this week. I know this because I can't get anybody to return my emails. :(

Maybe you should stop self-hosting your email server :)

Defiantly there is a trend away from email outside of business, but in B2B email still rules. The ability to have a record of what was originally said and agreed to has saved my backside more than once.

What do you use to communicate if not email?

I use it constantly, as I have for years: one mailbox for work, one for personal stuff. My whole life flows through email, it sometimes seems. How else would it work?

More and more so Slack, it acts like email, chat, server communications tool, quite handy.

Hm. I hear about Slack but have never actually encountered it and have no idea why it is so popular. Perhaps someday its social network will finally connect up with mine and I'll find out why people care.

For the most part, facebook, twitter and sms. Trying to get people I know IRL to check their emails is a battle I lost about five years ago. If we're talking intra-company, then it's slack, hipchat, Yammer, google hangouts, trello, jira and whatever wiki system or wordpress is in use at the company I'm currently working for.

Email for me, as I alluded to in my previous comment, is mostly about interacting with automated systems. If I need to reset a password, I'll get an email with a link. My car insurance company and web host will occasionally email me asking me for payment. Google emails me to let me know I've logged in using a computer it doesn't recognize (like my phone).

I get a photographer, video maker or someone working in fashion (I am sure there are other cases, I know those) using facebook or <x social tool> for business communication because it is trendy or it is more convenient, but preferring it to emails?

There is obviously a problem here, there is so much talk about privacy and confidentiality and then you give all your communication in the hands of one company.

What I am trying to say is that there is a fundamental technical difference between emails and <faceslaktter>: The first is based on a distributed, well defined (RFCs??), open protocols and formats. The second? It is an application, running on the premises of some company...

Do you even read the posts you're replying to? Nowhere did I say I preferred closed platforms to email. I don't. I used the word "battle" in my previous comment for a reason. You're preaching to the choir.

> I get a photographer, video maker or someone working in fashion (I am sure there are other cases, I know those) using facebook or <x social tool> for business communication because it is trendy or it is more convenient

You know that's 99% of the world, right? You just described everybody but people who work in tech. How am I supposed to communicate with people who don't check their email?

> The first is based on a distributed, well defined (RFCs??), open protocols and formats.

Insecure open protocols and buggy formats. You know all emails travel in the open air, right? This was the primary reason I suggested we replace it, with a system that is secure, easy to use and that people can control.

I do read posts, and I wrongly implied you end up preferring other means to email because people don't answer to emails, sorry about that, but let me expand on my thoughts.

I get that too, people not replying, but I don't think this is only a technical problem unfortunately. With IM systems where you easily get notifications on your phone you have more chance to get a faster response, also when receiving large amounts of emails one tends to skip many of them.. frustrating, for the sender, but that happens to me too from time to time.

I do work in tech though, also in academia, and email is still the first choice (well, from my experience, also I am in Europe.. maybe this counts?)

As for the technical aspect, SMTP and mail transfer protocols are part of a distributed system, and a distributed system must be open, this of course leads to the SPAM issue, but after what, more than 30 years? we have a couple of strong server side implementations that work well. And let's not forget TLS... emails travel in the "open air" as every other protocol that does not use encryption.

I agree that a redesign would be helpful to eliminate some basic issues we still have with emails, but it looks like that at the moment most are busy inventing new proprietary protocols for new way to chat and send emoji around...

Email has not be to replaced, email just needs a better representation.

IMO Inbox by Google fixed that issue for me.

I thought so too.

Well, until you have done everything, — DKIM, SPF and checks are green… And gmail is continuing put your e-mail to spam every now and then :/

The thing is that it also depends on the client side (meaning the software client) and from the sender behaviour.

For instance, I have a client (person) who now and then sends out a newsletter to some hundreds of addresses. He does that directly from the webmail. I always catch him because I have an alert on the mail queue, and every single time I check it I notice that the messages have been delayed because the remote server has a filter on "too many recipients at the same time" or "server sending too fast" or similar.

This affects the way the remote server will consider further emails from your server, and change its "behaviour" temporarily. I use Postgray [1] that does very similar checks, and it is a great deterrent for incoming spammers. So I completely understand this strict checks from the point of view of a sys admin.

Then there is the email content and headers that are checked by spam filters... but that's another story...

[1] http://postgrey.schweikert.ch

What this means in practice is that you have to set up with DKIM and SPF at minimum. E.g.:

https://www.exratione.com/2016/05/a-mailserver-on-ubuntu-16-...

But: you must then also use a host that is going to give you a decent chance at a non-polluted IP address. The host of the server appears to be a big factor in the hidden part of the anti-spam ecosystem used by the big email providers like Gmail and Hotmail. This means that trial and error and experience in the ever-changing field is the only way to make a good choice. Expect to be moving around a bit as you figure things out. AWS EC2 is the best choice that I know of. Digital Ocean is the worse choice that I know of.

A possible approach is to set up your mailserver so that it relays outbound messages via AWS SES. SES has all of the advantages of a big service that puts in time to ensure deliverability, with few of the downsides, such as large cost.

For calendar apps, you might look at Horde (see an old setup example here https://www.exratione.com/2012/05/a-mailserver-on-ubuntu-120...) - I don't know how it matches up with other options. I moved away from it because I really didn't need a calendar as it turned out, and Horde is something like 10% application and 90% configuration system. It was a lot of work to get it set up and comfortable.

This was the problem I ran into when trying to self-host. Getting postfix etc set up was a pain in the ass, but I was getting flat-out refusal responses from big mail providers. Ended up pointing my MX records at google and paying them five buck a month to host it for me.

I don't have the same experience, I run my own mail server (that I wrote in python) on a personal domain and all it took was adding the correct SPF records to have my mail accepted by Google & co.

I've been running my own for years. On a VPS now. With SPF and DKIM my delivery rate is awesome. I host domain mail and send two large-ish newsletters (~4k each). Postfix, OpenDKIM, Dovecot in Linked.

Granted, takes a bit to set up. Up-side is full control. I have a robust black-list too, keeps loads of spam from even entering the syatem

I run my mail server using docker, specifically the following images:

  * mailserver: https://github.com/tomav/docker-mailserver
  * webmail: https://github.com/jprjr/docker-rainloop

Combined in docker-compose:

  version: '2'
  
  services:
    mail:
      image: tvial/docker-mailserver:latest
      hostname: your_hostname
      domainname: your_domain
      environment:
        SSL_TYPE: letsencrypt
      ports:
      - "25:25"
      - "587:587"
      - "993:993"
      volumes:
      - ./data/mail:/var/mail
      - ./data/config/:/tmp/docker-mailserver/
      - /etc/letsencrypt/:/etc/letsencrypt
    rainloop:
      image: jprjr/rainloop
      environment:
        NGINX: 1
      ports:
        - 80:80
      volumes:
        - "./data/rainloop:/var/lib/rainloop/data"

The docker-compose.yml and postfix-accounts.cf files are the only configs you need for a basic setup.

See the projects on github, especially docker-mailserver for further info on complete features of this setup.

That seems pretty cool. How long have you been doing this? For your primary email? How has it been working for you?

I've had this setup for about a month on my primary email, and it is working flawlessly. Even when I had to reboot the server, I was able to bring everything back online quickly.

I am keeping a lower priority MX dns entry pointed to an independent provider as a failover, but this is standard practice, I believe.

What I really like about the docker-mailserver image is that it has no database and that it is designed for simple updates (that is, docker pull && docker-compose restart).

Rainloop is also new to me (I previously used roundcube) and again I am very positively surprised: it works over imaps, so multiple accounts can be combined under a single login, it supports 2FA, manages both plaintext and html, manages your contacts, it supports openPGP (still in beta, I've not tried it yet).

It is especially difficult to stop others from just blackholing your email.

This serverfault question helps to illustrate the problem

http://serverfault.com/questions/434703/why-does-hotmail-sti...

The problem? Not enough traffic causing a low sender score. Hotmail isn't the only one; large and small ISPs across the world do it.

Hotmail is the worst though. I hate the engineers at Microsoft who designed their email filter.

I can recommend iredmail. It's been around for a long time and supports several major OS. There is also paid support should you need it.

http://iredmail.com/

thank you so much for mentioning. didn't know this.

anybody has xp with one of these?

http://mailinabox.email

https://poste.io

edit: "awesome list" of things: https://github.com/Kickball/awesome-selfhosted#complete-solu...

Wow qmail! Instant confidence.

Go download Zimbra (I'm not affliated with Zimbra, just have used it in the past), it's easy to setup and full featured.

The real trouble is keeping your domain off blacklists and managing security updates.

The real trouble is getting your emails delivered. The big companies have basically outsourced the spam filtering problem to the email service companies - an email from outside the club has a x% chance of being randomly marked as spam (x is some number between 1 and 5%).

After 15 years of hosting my own email server without issue I had to make the business decision a bit over a year ago that I can't afford to have 5% of my emails go missing and so now use an external service.

I see a lot of replies on the mail side of things, but I don't really see anything detailing solutions for calendars.

Any recommendations on "the rest": calendar, notes, contacts?

I've looked at owncloud before, but that's a fairly large-footprint php project, are there any decent self-hosted alternatives, either as a suite or as separate programs?

As far as self hosting calendars and contacts are concerned, I'd recommend using a caldav/carddav server, I personally like Baikal [1], and Radicale [2] seems to be another good lightweight option. Contrary to Owncloud these two servers do not provide a web interface to check your calendar/contacts, but there are clients available for most platforms, including web clients if you need one.

[1] http://sabre.io/baikal/ [2] http://radicale.org/

I've been using radicale, and it's been a huge pain in the ass. It took me days just to get the permissions right (behind Apache), and its interaction with DavDroid and the iOS calendars/addressbooks has been totally baffling. It doesn't help that the radicale mailing list seems to be no more.

Probably I'm blaming radicale for some of webdav's weirdness, but the whole process has been one of the most frustrating computing things I've done in years.

Thanks, I'll give those a go!

Just since no one has pointed it out, have you considered ProtonMail? Swiss based, fully encrypted, hosted email service. Should give it a look.

Sandstorm is quite a simple (at least relatively) and secure way to get set up with this.

afaict, there is no mail server in sandstorm. Where did you read this information?

Even if you host your own email server, your emails are still handled by routers and the destination server unencrypted. You are not trusting others much less than not hosting your own email.

If you have your own domain and you don't trust others with your conversations with others, I think you are better off hosting a chat server over an encrypted protocol, such as XMPP or IRC or https://blog.okturtles.com/2015/11/five-open-source-slack-al... . Persuading your contacts to use it is a problem though.

Calendar is a different thing though. As long as you have a CalDAV server and a CalDAV client you should be good to go.

Also related to selfhosting stuff: https://sandstorm.io/ is a project I've discovered some time ago and haven't been paying attention to, but it seems to have thrived pretty well.

I've run my own email for about 4 years now -- I'm fairly tech savvy, but don't know much about email -- so was pleased that my "quick" solution has worked well.

I use Apple Server to host the email via SSL on a dedicated server, really easy to configure.

The key thing is deliverability - initially I had some problems with my email going into junk folders.

To get around this I send all email via a service called PostMarkApp (http://www.postmarkapp.com) which allows you to setup SPF and DKIM records on your domain and therefore ensure emails are authenticated.

I get pretty close to 100% deliverability.

PostMarkApp also shows you open / browser stats, so it's quite useful when I wonder if someone has read a particular email.

Not sure if this would be secure enough, but has worked well for me with almost zero maintenance after setup.

Six months ago I followed the Arstechnica guide[1] to setting up your own email server. The guide is old, and some adjustments were needed, but the essential advice was good. I did SPF, DKIM, TLS certificates, basically everything I could possibly do to keep on everyone's good side. In the first month or so, I only opened up to a few users, and told them to be careful about how they used the service right in the beginning.

Never had a single issue with blacklisting or non-delivery.

[1]: http://arstechnica.com/information-technology/2014/02/how-to...

It's not perfect, but YunoHost is a very big step forward in that direction and it's making progress https://yunohost.org

And to fix the "ISP sucks" we have started to package an "just work out of the box" solution in the FFDN, a federation of local associative ISPs https://internetcu.be/

The trick is to use a "internet cleaning" VPN that gave you a static ip address and that this VPN is handled by an association in which you are a member and that you can trust.

I'm sure it's not a popular view here, but why would you want to? I moved mail, etc. to Google; gmail is by far the nicest mail client I've ever used. I use it for my business and private e-mails, I move other businesses to Google because it's just so convenient. Why would you want to do all that yourself for €4 a month? I really don't think anyone at Google would want to read my e-mails but even if someone would, do I really care? I'm all for a bit of privacy, but I've seen no evidence of data abuse at these large mail providers.

It's not for the money, I can think of more effective ways I can spend my time to save money. My reason is that I don't like the feeling of having these very personal conversations being stored on the harddisks of google. I might trust google right now with my email but when I no longer do they will still have those files and there will be nothing I can do to completely make sure they delete them.

The odds are that the other correspondent in your personal conversations is storing all their messages on Google's hard disks.

fair enough, but I'd rather not be one that is responsible for that

This is an excellent point. Your communication is only as secure as the weakest endpoint (in the same vein, perhaps your correspondents might be of the opposing viewpoint and might feel weird about their personal mails being stored by someone who's rolling his own IT and security).

Also, I don't want to store personal data on servers of a US company.

https://mako.cc/copyrighteous/google-has-most-of-my-email-be...

And what do you want to say with that?

Even if you don't want to store your e-mail on Google's servers, a substantial part will live there anyway since you are likely to communicate with people that use consumer Google Mail or Google Apps for Domains.

So you need to encrypt your communications, getting off Google won't change anything. Use GMail as a dumb transport pipe !

Add to that the fact a huge amount of email is/was transferred without encryption, and you would be foolish to assume any of your email is private.

A valid point.... although that situation is in flux. If Google's data is any proxy for the rest of the world, the fraction of encrypted-in-transit email is getting pretty high [1], Google reports 84/75% out/inbound.

[1] https://www.google.com/transparencyreport/saferemail/

> but why would you want to?

For me, philosophical reasons. I want to fight-back against the centralisation of Internet services. It's the same reason I run my own blog, host photos on my own server, pay slightly more for an unfiltered Internet connection and so on.

I can't 'win' on my own but its depressing that so many people are happy to jump on the centralisation because it's easy and convenient right now.

And what will happen in 10 years when you only have a choice of MegaCorp-1 or MegaCorp-2 handling all your online services, slicing and selling your personal data and bombarding you to consume more, because anyone running their own servers is a "digital terrorist"? Too late then.

> I want to fight-back against the centralisation of Internet services.

My sentiments exactly. Right now email is something anyone can participate in as long as you are using software that follows the (open) standards. It is at its core an inclusive technology, despite the (disheartening) issues mentioned in this thread about the pitfalls and hazards of getting your own mail-server going. It would be a shame if we were to lose this.

On the other side of the spectrum we have the dominant instant messaging platforms, where you can only participate if you run their proprietary software and identify yourself with your pre-approved identifier (your phone number in most cases). Or the centralized social media platforms that censor according to their interpretation of what is permissible.

I understand that leaving all complexity and the burden of grokking something as abstract as networked software is something most people are glad to leave in the hands of Apple, or Microsoft, or Google; but it saddens me that those of us who do understand how this stuff works (at least in a broad sense) appear so willing to just accept the death of a free and open internet for a bit of convenience.

A problem seems to be losing common media for message exchange. After experience with a bunch of instant messaging tools, invariably having to abandon them after focus shifts elsewhere, it's tempting to rely on something more of "common denominator". Email was that... and if it stops, then - what? phone?

This is an interesting problem of lack of standard. Hope http://seif.place will help with that :) .

Would it be possible to set something of a webmail client up with Mailchimp/sendgrid APIs? Get the flexibility and control of how your email works without the flagged as spam issue?

Any one using Mail-in-a-Box (https://mailinabox.email/)? Seems good to me.

I would support this if it wasn't based on Ubuntu, especially 14.04. That is the last disto I'd trust as a secure mail server.

Why do you think Ubuntu is dead last among Linux distros in security? From my experience, they patch things quickly and inform about security issues promptly.

http://www.ubuntu.com/usn/

Security is not only dealing with aftermath, although that is an important part of it. A security-first distro such as http://www.alpinelinux.org/ or OpenBSD, although harder to setup, would serve better long term.

1. They often run out of date package versions.

2. Their packaging team screws up packages regularly.

3. They have insane defaults of many package configurations.

4. They tend to do things their way rather than the standard way and come up with their own tools rather than using standard, well tested tooling / packages.

I could go on and on, but it's not a distro I'd ever let run on one of our servers, we used to run 100% Debian until CentOS 7 came out and in our eyes left it for dead when paired with the correct yum repos.

The first time I hosted my own email, on sendmail, it just worked. So I don't understand the question about setting it up. Configuring it for security took some study but it wasn't difficult. No more difficult than any other serious program I use.

Calendars are similar to forms on the web. Again, no more issues than normal so I don't get it.

Setting up an email server isn't hard.

Setting up an email server that won't get blacklisted by the major providers within a week of you setting up is difficult, time consuming, and often expensive.

You are saying that what I am doing will get me blacklisted, etc., but I am here to tell you it is not any of the things you say.

Self-hosted owncloud works for me except for the gruesome woes every single time I try to update it.

Does OwnCloud support email too? I wasn't aware of this.

No it doesn't, unfortunately. I misunderstood OP's questions at first. You can connect owncloud to a mail server (a la Roundcube) but nothing more.

Maybe one day they'll make a plugin that does the same that Roundcube and other PHP email clients are capable of doing, that would at least make it a little smoother to have as an all in one email client.

postfix (SMTP), dovecot (IMAP), davical (CardDAV for addresses, CalDAV for calendars) have been working well for me for more than five years.

I experienced blacklisting when I switched to a newer dedicated server hardware recently, which meant new IP addresses. The only mail server that didn't accept mails from my new addresses was Hotmail / Outlook.com though. It was fixed after filling in their form and waiting a day. Never had trouble with Gmail.

If you store your emails in a hosting provider then the hosting provider can read your emails. If you store your emails at your server at home then police can break your door plug in to your server and read your emails. Even if the disk is encripted they can read the encription key from the memory. Now I trust more Google than Microsoft, if you are non USA citizen of course. And I trust a foregn authority more than a local. This can change in for your case.

Well I live in The Netherlands and somehow I trust the Dutch police to only enter my house for a valid reason, unfortunately I can't say the same for Google/NSA reading my emails.

At least you know if the police have come for your mail.

In practice hosting your own mail provides some control over how to respond to a subpoena. If someone sues you the bar for getting the data by physical force is much higher than it is for having a third party turn it over.

On the other hand the bar for compromising your server is usually much lower than it is for gmail.

Because there is no money in that model.

Reply on news.ycombinator.com