Hacker News Reader

Comment by kardos | Hacker News Reader

This post [1,2] from last October discusses a major problem with hosting your own email, which is that the big email providers pretty much reject your messages until your "reputation" is good. The reputation concept is purportedly a spam fighting effort, but a more cynical view is to see it as an effort by the big providers to sabotage self-hosted email. More discussion here [3] and here [4].

[1] Original post, now defunct: http://liminality.xyz/the-hostile-email-landscape/

[2] Thank you archive.org: https://web.archive.org/web/20151121132739/http://liminality...?

[3] https://news.ycombinator.com/item?id=10405681

[4] https://lobste.rs/s/ckfyqd/the_hostile_email_landscape

View on HN

There are a number of things that can (and should) be done to prevent outgoing emails from your SMTP server not to be rejected or flagged as SPAM (or worse your IP address blocked).

A part from the obvious basics (IP reverse lookup working, TLS support, SMTP auth for your users..) you can also set up DKIM [1] and SPF [2] that help a lot and are well supported by big providers like gmail.

But more importantly not ending up in a blacklist is important, so if you have users using your service you should monitor the volume of outgoing emails to avoid someone using your server to spam.

There are a few tools that you can use to check your SMTP server MXToolbox and CheckTLS are the two I often use. [3]

I agree though that managing an SMTP server is challenging, I had a couple of issues I had to work on that I am sure would never have happened to a big provider.

[1] http://www.dkim.org

[2] http://www.openspf.org

[3] http://mxtoolbox.com http://www.checktls.com/testreceiver.html

You can do all of these things and still a certain percentage of your emails will disappear into the ether. The game is rigged and the little guy has lost.

I'm willing to accept email is rigged. Who uses it for actual communication these days anyway? I use mine primarily to get lifecycle emails from webapps I tried once and to reset passwords.

What's next, then? There have been lots of replacements for all of the old school protocols like IM and IRC. Even newsgroups have been (adequately?) replaced by forums and link aggregates and facebook. But, I've seen no attempt to replace email. I've seen lots of attempts to enhance email, but none to replace it completely.

>Who uses it for actual communication these days anyway?

People outside of tech bubbles.

People outside of tech bubbles use snapchat, facebook or whatever chat program is popular this week. I know this because I can't get anybody to return my emails. :(

Maybe you should stop self-hosting your email server :)

Defiantly there is a trend away from email outside of business, but in B2B email still rules. The ability to have a record of what was originally said and agreed to has saved my backside more than once.

What do you use to communicate if not email?

I use it constantly, as I have for years: one mailbox for work, one for personal stuff. My whole life flows through email, it sometimes seems. How else would it work?

More and more so Slack, it acts like email, chat, server communications tool, quite handy.

Hm. I hear about Slack but have never actually encountered it and have no idea why it is so popular. Perhaps someday its social network will finally connect up with mine and I'll find out why people care.

For the most part, facebook, twitter and sms. Trying to get people I know IRL to check their emails is a battle I lost about five years ago. If we're talking intra-company, then it's slack, hipchat, Yammer, google hangouts, trello, jira and whatever wiki system or wordpress is in use at the company I'm currently working for.

Email for me, as I alluded to in my previous comment, is mostly about interacting with automated systems. If I need to reset a password, I'll get an email with a link. My car insurance company and web host will occasionally email me asking me for payment. Google emails me to let me know I've logged in using a computer it doesn't recognize (like my phone).

I get a photographer, video maker or someone working in fashion (I am sure there are other cases, I know those) using facebook or <x social tool> for business communication because it is trendy or it is more convenient, but preferring it to emails?

There is obviously a problem here, there is so much talk about privacy and confidentiality and then you give all your communication in the hands of one company.

What I am trying to say is that there is a fundamental technical difference between emails and <faceslaktter>: The first is based on a distributed, well defined (RFCs??), open protocols and formats. The second? It is an application, running on the premises of some company...

Do you even read the posts you're replying to? Nowhere did I say I preferred closed platforms to email. I don't. I used the word "battle" in my previous comment for a reason. You're preaching to the choir.

> I get a photographer, video maker or someone working in fashion (I am sure there are other cases, I know those) using facebook or <x social tool> for business communication because it is trendy or it is more convenient

You know that's 99% of the world, right? You just described everybody but people who work in tech. How am I supposed to communicate with people who don't check their email?

> The first is based on a distributed, well defined (RFCs??), open protocols and formats.

Insecure open protocols and buggy formats. You know all emails travel in the open air, right? This was the primary reason I suggested we replace it, with a system that is secure, easy to use and that people can control.

I do read posts, and I wrongly implied you end up preferring other means to email because people don't answer to emails, sorry about that, but let me expand on my thoughts.

I get that too, people not replying, but I don't think this is only a technical problem unfortunately. With IM systems where you easily get notifications on your phone you have more chance to get a faster response, also when receiving large amounts of emails one tends to skip many of them.. frustrating, for the sender, but that happens to me too from time to time.

I do work in tech though, also in academia, and email is still the first choice (well, from my experience, also I am in Europe.. maybe this counts?)

As for the technical aspect, SMTP and mail transfer protocols are part of a distributed system, and a distributed system must be open, this of course leads to the SPAM issue, but after what, more than 30 years? we have a couple of strong server side implementations that work well. And let's not forget TLS... emails travel in the "open air" as every other protocol that does not use encryption.

I agree that a redesign would be helpful to eliminate some basic issues we still have with emails, but it looks like that at the moment most are busy inventing new proprietary protocols for new way to chat and send emoji around...

Email has not be to replaced, email just needs a better representation.

IMO Inbox by Google fixed that issue for me.

I thought so too.

Well, until you have done everything, — DKIM, SPF and checks are green… And gmail is continuing put your e-mail to spam every now and then :/

The thing is that it also depends on the client side (meaning the software client) and from the sender behaviour.

For instance, I have a client (person) who now and then sends out a newsletter to some hundreds of addresses. He does that directly from the webmail. I always catch him because I have an alert on the mail queue, and every single time I check it I notice that the messages have been delayed because the remote server has a filter on "too many recipients at the same time" or "server sending too fast" or similar.

This affects the way the remote server will consider further emails from your server, and change its "behaviour" temporarily. I use Postgray [1] that does very similar checks, and it is a great deterrent for incoming spammers. So I completely understand this strict checks from the point of view of a sys admin.

Then there is the email content and headers that are checked by spam filters... but that's another story...

[1] http://postgrey.schweikert.ch

I don't have the same experience, I run my own mail server (that I wrote in python) on a personal domain and all it took was adding the correct SPF records to have my mail accepted by Google & co.

What this means in practice is that you have to set up with DKIM and SPF at minimum. E.g.:

https://www.exratione.com/2016/05/a-mailserver-on-ubuntu-16-...

But: you must then also use a host that is going to give you a decent chance at a non-polluted IP address. The host of the server appears to be a big factor in the hidden part of the anti-spam ecosystem used by the big email providers like Gmail and Hotmail. This means that trial and error and experience in the ever-changing field is the only way to make a good choice. Expect to be moving around a bit as you figure things out. AWS EC2 is the best choice that I know of. Digital Ocean is the worse choice that I know of.

A possible approach is to set up your mailserver so that it relays outbound messages via AWS SES. SES has all of the advantages of a big service that puts in time to ensure deliverability, with few of the downsides, such as large cost.

For calendar apps, you might look at Horde (see an old setup example here https://www.exratione.com/2012/05/a-mailserver-on-ubuntu-120...) - I don't know how it matches up with other options. I moved away from it because I really didn't need a calendar as it turned out, and Horde is something like 10% application and 90% configuration system. It was a lot of work to get it set up and comfortable.

Well this seems to be a dealbreaker, I can't have an email server that is likely to get blacklisted.

I posted that in part hoping that someone would chime in with a workaround/solution figured out in the 6-7 months since that discussion.

I happened to post comments in GP's cited thread and also another email server thread. You can read both of them to get a deeper understanding that personal email servers are hampered by "trust" issues more so than technical challenges.

https://news.ycombinator.com/item?id=10405945

https://news.ycombinator.com/item?id=10498988

Basically, you could conceivably get past all the technical set up. There may even be Docker containers or virtual images to jumpstart most of the set up. Or you can follow some step-by-step instructions.[1]

However, the problem is that SMTP servers originating from residential IP addresses by default are treated as home computers that have been hijacked by a malware spam bot. Using Bayesian reasoning, this is a rational filter because the number of zombie home computers sending unwanted spam vastly outnumbers any tiny amount of "good" people setting up legitimate SMTP servers.

Given today's realities of distrust-by-default (unknown SMTP server is guilty until proven innocent), it looks like the best strategy for a "personal" SMTP server is to pay for a virtual machine at a hosting provider (Rackspace, etc) whose IP address ranges are not blacklisted. You could then run encrypted email storage there so Rackspace has no visibility into private emails.

[1]http://arstechnica.com/information-technology/2014/02/how-to...

I run email for most of my web clients. We have never been blacklisted and my email servers were set up without any help but the docs from day one.

This was the problem I ran into when trying to self-host. Getting postfix etc set up was a pain in the ass, but I was getting flat-out refusal responses from big mail providers. Ended up pointing my MX records at google and paying them five buck a month to host it for me.

Self-hosting is dead unless you are massive business.

I have a very small (on the side) hosting business that has been running for almost 6 years now and growing little by little. I have direct contact with my clients and they have it with me in case of need (if they need technical advice or support not necessarily directly related to the services).

And this is exactly the kind of service they were looking for, very difficult to have with a massive business company (have you ever opened a ticket with OVH?)

Then of course, the technical challenges to setup and maintain such services are high and a control-panel all-included package or a docker with everything in it is just a very small part of the work required.

Self-hosting email or self-hosting in general?

Email, but I would not be surprised if self-hosting in general become in practice impossible - self-host and your site only shows up on page 100 in Google.

I've been running my own for years. On a VPS now. With SPF and DKIM my delivery rate is awesome. I host domain mail and send two large-ish newsletters (~4k each). Postfix, OpenDKIM, Dovecot in Linked.

Granted, takes a bit to set up. Up-side is full control. I have a robust black-list too, keeps loads of spam from even entering the syatem

Reply on news.ycombinator.com