Hacker News Reader

Comment by userbinator | Hacker News Reader

I've noticed that "if it's not plaintext, it gets deleted without being read" seems to be a pretty common rule among Germans on the Internet, who also have a tendency to like specifying very exactly what they want of email to them. Here's a few examples:

https://www-user.tu-chemnitz.de/~heha/email.en.htm

http://problemkaputt.de/email.htm

https://www.gaertner.de/~neitzel/email-to-mn.html

http://www.karo-electronics.de/448.html

...and of course there's this:

http://arc.pasp.de/

View on HN

Second link points out a problem I was afraid of existing - big providers just stump on personal email servers.

>Hotmail is typically deleting all emails that I am sending. ... Monopolists like gmail.com won't accept any messages sent from my mail server.

I feel like I'm making this comment once a week on HN but I host my own email and I haven't had any major issue so far with "big email".

The main caveat is that you will have a very hard time getting your email accepted if it comes from a home connection IP range instead of some host provider but if you do have a dedicated server and follow the guidelines (SMTPS, DKIM, SPF etc...) it just works, at least in my experience.

Hosting one's own e-mail server is a totally opaque random crapshot. You may not have any trouble, but some other dude or gal will get their e-mail marked as spam without any way to tell what exactly is wrong and what to change.

First step when getting an ip for a server that will be a mail server is to check if the ip is not already blacklisted.

You can always get it unlisted.

After that don't start to send hundred of email by day. You need to build a reputation for your domain and ip.

As the parent comment says, set up directly spf, dkim and dmarc (also arc if you can). Rspamd can help you do that.

I've been running a personal mail server for 5 years with simply following those rules.

I've been running a personal mail server for twice as long with simply following those rules and my e-mail is still tagged as spam in Gmail when it's me who initiates contact (once the other party sends me an e-mail, reply or otherwise, I no longer get tagged as spam).

As I said, it's totally opaque crapshot.

Are you using DKIM and SPF properly? I've never had any problems with this (although, I'm not mailing from home ISP ranges)

Yes.

That's very strange, then.

I'd recommend trying http://dkimvalidator.com/ to make sure everything is perfect; having proper certs on your MXs seems to help too.

I run a pretty large email infra (~500k/minute) and I'd help you out if I can...

If you're still having problems with all that in place then why don't you just change IPs?

As I said, Gmail is opaque with regard to its spam detection.

As for your help offer, thank you, but I'm good. It's usually other people who want to contact me, I rarely send the first message ever.

Hmm well, I've found it to be pretty straightforward, I'm curious why this is happening to you though and there must be a solution...

I mean.. If you get a solid result from dkimvalidator but google is still shitlisting you then I'd definitely consider moving to another host/dc/isp at least.

Depending on your size, it might be best just to make this someone else's problem (if you can) -- like google, o365, etc..

I'm definitely not giving to somebody else my /var/log/mail.log, IMAP/Mutt access, sieve rules, nor ad hoc dedicated aliases for every website account I create. And on top of that, I would land on some US-based server for ease of illegal spying and my mail would be harvested for some advertising crap, all to solve something that is not a problem for me. Not happening.

Yep, that's the reason I host my own email also.

I'm working on a guide for setting this stuff up; still WIP [1] -- feedback appreciated!

1: https://medium.com/@cyberpunk_networks/nsa-proof-your-email-...

> I would land on some US-based server for ease of illegal spying and my mail would be harvested for some advertising crap

Does this not happen when you send your emails to someone else? Mad props if everyone you email uses encryption!

The real problem is Google is opaque in dropping such mail in recipients spam folder. Hotmail can sometimes be annoying in this regard too. As far as I've managed neither does the sane thing and add a header that report on how they've divided that "this mail is spam" - so the recipient doesn't have anything to go on either, other than hoping "mark this as bot spam/add to address book/send email/reply to address" help Google/Ms treat it as "not spam".

Maybe it's worse for non-English mail (maybe the statistical models are biased against "not English" even for people whom don't have English as a native language).

But I'm quite convinced google's (and to a lesser extent Hotmail/ms') "magic" spamfiltering is subtly (but annoyingly) broken.

There have been a few discussions regarding deliverability:

Why does Gmail hate my domain? | https://news.ycombinator.com/item?id=9855030

How to Avoid Spam Filters | https://news.ycombinator.com/item?id=10465639

Hotmail | https://news.ycombinator.com/item?id=14210939

ESP | https://news.ycombinator.com/item?id=14201704

If there are others I'd appreciate a link as I try to connect them!

I'm in a similar boat. I've moved ISPs a few times throughout the years, and each move it takes a few months to finally get things settled. I'm in the US, and had the best luck for deliverability when I was on a Comcast Business account.

My current ISP, a local fiber provider, was not great getting going. Most of the IPs that they have are in at least 1 spam database, and it took a while for the ISP to reach out to the database maintainers themselves. Even then, since they're a small ISP, the IPs are still blacklisted. The ISP wasn't even a company when the IPs were added to these blacklists.

After a few months they were able to assign me an IP that wasn't in a blacklist somewhere. I still randomly have issues with the big providers though - gmail is probably the most annoying. Like dozzie, my SPF, DKIM, and DMARC setups are all valid.

Overall, I really enjoying running my own mail server. Every now and then there are a few annoyances, but it's worth it in the long run.

> You can always get it unlisted.

How do you do that? The vast majority of blocklists I've interacted with have been unwilling to deal with questions, instead responding only that if you fix "something" (not always specified) automated measures will remove it from the list eventually.

In my experience it has also been extremely difficult to deal with people using blocklists. It's easy to find a bunch of people using .tor.dan.me.uk rather than .torexit.dan.me.uk "just to be safe". Frankly, I'm not sure why the former list exists in the first place other than to be an arse? What threats do entry/relay nodes pose to you?

I worked for two very large ESPs and never really had many problems with blacklists from a deliverability perspective.

Whenever one of our mailer IPs was blacklisted by one of the big targets (hotmail, gmail, etc) it would only be for 24hours after which I'd put it back into the pools (although, at least initially perhaps for our more reputable clients to warm it back up before letting the more dodgy stuff through it again). If you host your own NS's for the sender domains flipping SPF ips/ranges is not too hard (it's all automated for us, anyway).

The big boys work that way at least in my experience. I'm sure sometimes you'd hit a 'somedomain.foo' domain which is using a blocklist style thing that their (usually inexperienced) sysadmins think prevents them from receiving spam; but they're not worth arguing with. If you're doing email at volume shifting sends to another range for that client is usually 'enough' to get them through if they care that much about that one segment to ask you to do it.

If it's not then we'd usually refund instead of trying to negotiate with such admins.

Honestly though, blacklists have never been an issue for me and I've sent a ridiculous amount of email over the last 10 years...

Maybe https://www.mail-tester.com/ would help. Also https://mxtoolbox.com/diagnostic.aspx .

If neither of these tools highlight any issues it's pretty strange indeed.

If I ran an evil email monopoly, I would systematically drop all email from say _half_ of the email servers out there. This way, nobody would be able to make a case that we're abusing our monopoly position, but at the same time running an email server is hard and the internet is filled with wildly varying experience reports (which is the best FUD there is).

This way, fewer people start email servers and, therefore, fewer potential competitors grow up.

Obviously I have no way to tell whether Google really does this, but if they would, the result would look exactly like this HN thread.

I'm probably totally wrong here, but note that there is absolutely no incentive for the large email providers to try to fix this mess and make email the free distributed network it once was.

I've gotten DKIM, SPF, DMARC, etc all verified on my Digital Ocean droplet (and removed from various greylists) but still get spam checked by AT&T / SBC Global, who is the main offender of keeping a private list.

I've been removed for now from their lists... but they tend to re-add IPs from previous ranges for no reason.

This is the most German thing I've seen since Octoberfest 2016.

It seems weird to me to have to find out the preferred way of communication to this level of detail, I already loathe that one cient of mine that only uses Facebook messenger for all communication.

Thunderbird has settings for each of your contacts for this reason. You can set if they want HTML or text only mails.

I would love to only use plain text e-mail. However, how does this work in practice when you are not a well known person (albeit in a niche) that decides the rules?

In work I have to accept whatever my colleagues, clients and bosses will send me. For personal communication I use mail with two people. And services that let you choose whether they'll send you plain text are practically inexistent.

Many (most?) emails can be read in plain-text format even if they were written in HTML. The email often comes encoded with an "alternative" plain text version; of course you need a client that will show you that version instead. In my experience most personal email (i.e. not automated/form mails) I receive has a plain text alternative version that works fine.

For emails sent by an evil client that doesn't provide a plain text version, you can consider a tool that attempts to convert HTML to plain text (though I don't have a suggestion for this yet). Fall back on reading the HTML version only if the above fail.

- experience from working on / using my own mutt-like terminal email client.

> For emails sent by an evil client that doesn't provide a plain text version, you can consider a tool that attempts to convert HTML to plain text (though I don't have a suggestion for this yet).

I use emacs to read my email, and w3m to render HTML email as text. It does a good job. Newer emacs includes its own web browser (eww) but I have not tried it for HTML email rendering since I'm happy with w3m.

Replies always go out in plain text (my preference) but if you want to send HTML there are ways to e.g. write your email in org-mode markup and have it converted to HTML when you send. But IMO if you want to send "rich" email then just use a client that does that natively.

Mutt + Lynx does it well, too

Similar to the other child comment: elinks can format HTML as plaintext.

I have this as my .mailcap and it Just Works (TM) with mutt:

  text/html;  elinks -dump %s; nametemplate=%s.html;          copiousoutput

You can also use pandoc, I had better results with this:

> text/html; iconv -f %{charset} -t UTF-8 | pandoc -f html -t plain --wrap=preserve; copiousoutput; nametemplate=%s.html; description="HTML eMail";

I automatically delete anything with an attachment, that's so effective I didn't bother going any further with it.

I'm still waiting for people like that to automatically email back a CAPTCHA to the sender, if the sender is unknown, and not on a whitelist yet.

The proof of work approach is better.

Or how about a (micro-)payment?

(I suppose it would be better for the environment than wasting energy on otherwise useless computations)

How much work would you propose is sufficient that grandma doesn't mind but spammers will be severely hampered in their sending and in a manner that doesn't require the receiver to store too much state?

Only a small amount is necessary. There is a protocol called hashcash (used by Bitcoin but generally applicable) that can be easily used with email [1]. You basically get a header like this:

X-Hashcash: 0:030626:adam@cypherspace.org:6470e06d773e05a8

You can choose how much work you want to do and the recepient can specify thresholds for minimum work required. This header is all they have to store, and it's easily stored through existing email infrastructure. Your mail server can do it without your client's help, too. If you spent a second on a proof I'm sure grandma wouldn't notice, but it would be very difficult for spammers to do the same.

[1] http://www.hashcash.org/

That does not sound terrible useful.

For one, it'll only work if both sender and receiver use it. Which means for 99.99% of mail traffic right now, it is utterly meaningless.

edit: Plus you also literally wasted everyone's time and energy.

I'd rather favor some kind of mechanism like grey-lists that don't require sender opt-in otherwise it'll be a dead technology just like GPG. (You can probably count the number of GPG emails within 1000 average mails on one hand)

We make incremental change. Adoption starts slow, then some middle players pick it up, one big player grabs it and then it proliferates. I've been working on a mail client for some time and I have plans to write a mail server, and both will have first class, opt-out support for hashcash.

That sounds like a lovely pipe-dream.

I severely doubt any big player picks it up on grounds of it being cheaper and less energy intensive to simply greylist or blacklist.

How about sending (by email) a link to a proof-of-work-website, so the hash function can be computed in javascript/wasm in a browser?

Anyway, I agree on the waste of time+energy.

Thanks. I love the examples.

Reply on news.ycombinator.com