Hacker News Reader

Whether by monopoly or sheer blindness, Google leans really heavily towards only letting massive institutions through.

I know a few people whose professional work involves a lot of delivering explicitly-requested, non-spam emails. With most email providers, they don't go to spam because they aren't spam. If they get flagged as spam servers (usually right after spinning up an instance, when the confidence is low), they can contact someone and get it sorted out.

With Gmail? They have extensive lists of tips and tricks to "sneak through" emails that their users specifically asked for. They 'prime' servers with extra-safe traffic to avoid getting frozen in the low-confidence early emails. And they still regularly run into problems. They just don't have the size to be trusted by Google, or to get business contacts to solve the problems that do arise.

It's a pretty concerning state of affairs - Gmail will deliver anything sometimes, but it seems to be a very high bar (vastly higher than "not spam") to ensure you don't regularly drop messages.

This is exactly my experience sending legit commercial emails to gmail. We jump through every hoop, we implement every suggestion, and they drop our email on the floor every time. I have no problem with deliveries anywhere else. They DGAF.

The idea that their spam filter is particularly good is ridiculous too. I get more spam to my gmail accounts than to my self hosted mail accounts with greylisting and spamassassin.

>greylisting

Don't do that! Emails should be instant, not with stupid greylisting delays.

I will do that, and you should too.

Nothing in the RCFs says emails are supposed to be instant. Reliable, yes. Instant, no.

First, greylisting is HIGHLY effective. Something like 95% of email connections are spam. Almost all of them fail without ever hitting disk, via greylisting. If they actually retry, then the chance that the sender IP or email content is in an RBL is MUCH higher, so spam that actually retries will be more likely to be rejected at HELO by an RBL or filtered after it is accepted. The incidence of false positives with grey listing is VERY VERY low. Using greylisting means that very few emails need to be accepted and then filtered. Those are the false positives that really hurt. The sender receives no bounce, and the recipient probably doesn't look at his spam folder every day, especially when it is full of hundreds of junk emails. Greylisting makes email more reliable.

In addition, better greylisting systems note senders that reliably retry and automatically whitelist them. My mail server actually delays less than 2% of legitimate emails due to greylisting.

I'm completely confused by Gmail's spam filter. We've already discussed its extreme tendency to false positives, but it's quite good about false negatives - no spam hits my inbox.

Except... Google claims >90% of spam never even reaches your Spam box, only the ambiguous messages. So why do I get 10x the spam at Gmail as at any other account? Are they not actually good at detection? Are they just a high value target that gets hit more often?

Or (my guess) are they trying to get clever? Google Doc's spellcheck is infamously bad because they're trying to use machine learning approaches without having a simple English dictionary as a fallback. Is Gmail trying to detect solely on email sender/content and totally abandoning approaches like greylisting?

I had my personal domain hosted with Google. One of the reasons I moved away is that emails that I was sending to other Google Apps domains were bouncing as spam when the destination address was an alias. The bounce came from Google Groups, which is what Google used for handling aliases within Apps domains.

e.g. https://pastebin.com/u48DAaLP

That particular example was sent via SMTP, but I had the same problem when sending via the Gmail web interface, and it occurred sending to at least three different Google Apps domains.

After I moved my domain off Google Apps (I switched to Fastmail for a variety of reasons, but that issue was the kicker), I was able to send to those same addresses without issue. In fairness to Google, I was on the Google Apps free tier at the time, so there was nowhere to go for support.

ARC was designed to fix that kind of edge case, and Gmail was one of the first to start signing and validating ARC within the last couple months - so this may no longer be a problem.

http://arc-spec.org/

What's the edge case? That the destination address is an alias? That I was using a personal domain? That Google implements aliases using Google Groups?

I never understood what was causing these bounces. The entire email chain occurred within Google's control, passed SPF and DKIM checks, my domain isn't on any blacklists nor used for spam that I know of, and isn't used for anything but personal emails. And I've had it registered for years. Nor was it the content of the message. I had a friend with another Google Apps domain setup an alias for testing and when I mailed that address it bounced the same way, regardless of the message body. Emails to other addresses in his domain were not marked as spam... it was only Groups aliases that bounced.

Ok so as much as I promote self-hosting email, you're right. I've also had trouble with mail originating on my server to gmail and only gmail accounts. I did recently discover that I had some configuration issues via mail-tester.com though (my rDNS didn't match the HELO of my server) so now that I'm getting a 10/10 on that, things seem a bit better.

I definitely felt that gmail was being too aggressive on this too. I sent a bunch of detailed replies to middle/high schoolers working on science projects when they reached out through my public-education webpage and when they went to gmail I never got a reply but if they went to some other server, the conversations usually continued nicely. That was pretty frustrating.

Fully agree. I am in the same situation as you, and I even relay my emails through my ISP SMTP to avoid the issues such as 'residential IP marked as spammer'. And yet, Gmail will randomly classify simple personal emails. It is the only mail provider to act like this, I never had any issue with any other provider (ISPs, CSPs...) And to add insult to injury, they have this page: https://support.google.com/mail/answer/6227174?hl=en&ref_top... but as I send too few emails, it is completely useless.

> I set it up with SPF and DKIM properly, triple checked. And yet, for whatever reason, sometimes my email end in spam.

It's probably your IP address.

Sure, SPF and DKIM say you are allowed to send these emails, you are who you say you are, etc, but you also aren't a Sendgrid IP address (or someone similar) so you're treated as less trustworthy.

I could be wrong, but I believe many email servers take these things into consideration (please let me know if I'm mistaken!)

This is fairly common - if you want a method that gets you around this at the cost of some privacy, you can use Mailgun or a similar service for free as your relay, which should pass most spam filters without issue.

Mailgun has similar issues though. We advise those signing up for accounts on our open source community site not to use Hotmail since mailgun emails tend to get binned by Hotmail. Not sorted to spam or bounced, straight up deleted.

Is this anti NN.

No.

However, it's still Real Stupid.

I'm pretty happy with the results of spam filtering based on sender reputation.

You can appeal to DNSBLs to have your IP unlisted.

Not always feasable.

I run my email through google for my domain, and the IP assigned to my mail server is actively used to send spam (like, daily active reports on SORBS), so I can't even fill out their form to get it unblocked.

As a result, I need to email my bank from a different email address. So stupid.

>I run my email through google for my domain

Well, here's your problem. Have you thought about self hosting?

If you're literally running an e-mail server from your cable/DSL connection, you're much more likely to be marked as spam, regardless of doing everything else correctly (SPF/DKIM/reverse DNS), because there are lists like SORBS DUHL [1] and RBL-DNS DUL [2] that index these. Some servers block entirely, others just use it as an indicator that your mail is more likely to be spam.

Many spam filtering systems work on a 'points' system, so being on the dynamic IP list adds a lot of points, and if you also have other things that add points (being a mailing list, having 'unsubscribe' links, attaching images, having links) it might be enough to put it over the edge and get marked as spam.

You can do a lookup [3] to see if you're on any of these lists.

[1] http://www.sorbs.net/faq/dul.shtml

[2] http://www.rbl-dns.com/dul.shtml

[3] http://multirbl.valli.org/lookup/

Although what you say is true, I've been running my own mail server out of my house for years with proper spf/dkim/dns and have never had problems of my mail being sent to spam.

I set it up with SPF and DKIM properly, triple checked.

Check your reverse DNS as well. It helps if it matches the HELO advertised by your mail server.

If you haven't already, it can't hurt adding your sending domain to https://www.dnswl.org

Thx for the hint, I didn't know about this ! Just added my domain details to it, hopefully it helps a little.

Yeah, I’ve seen the same, to the point that I’ve actually considered getting a Gmail account for my business, because customers not receiving important e-mail is bad for business.

And I think that's half their goal. Why fix something if it gets you more customers (direct or roundabout)?

Yeah, it certainly looks rather sinister.

> Having a mailing list doesn't necessarely mean the content is unwanted. I'm subscribed to HN replies for instance. I want those email. If it started going to spam, I might consider "hey it's a few false positive". But for the HN replies guys, it would be maddening to be unable to fix the problem because Google can't be arsed to have a support channel.

I'm not sure this argument is related to the rest of your post, but FWIW, GMail/Inbox puts emails from mail lists into their own "Forums" category, not in the spam folder.

SPF & DKIM are only the start of the story. Use a non consumer IPv6 address, TLS, and DMARC. Have perfect DNS (including correct reverse lookups). Sign your zones. Monitor your IP and domain status with every reputation list on earth. Sign up for the postmaster tools at Google and Outlook.

Running your own email ain't a hobbyist thing anymore, you gotta play like you're in the NBA. I'm doing all the above, my deliverability is solid. And pay attention to the changing landscape - for example, I will be implementing ARC real soon.

> Running your own email ain't a hobbyist thing anymore, you gotta play like you're in the NBA

I'd disagree. I've been running my own just fine for years. Yeah, you have to put some work into it, but spf/dkim/dns have been suiting me just fine.

I think this is a matter of perspective. My response was to someone who, in my assessment of what they'd listed, had more of that work to put in. Note they were also talking of using email as a file transfer protocol, which you may not be doing. Also your years of operation may (as with my 17 years of spamless service) have built a very positive reputation for domains and addresses.

I mean, DMARC, ARC and TLS make perfect sense (though I have none setup because DMARC is nuclear, I didn't know about ARC, and TLS is just up for the submission port for me). But signed DNS ? What does that protect against, malicious registrar ?

And the postmaster tools don't work for me on google because the volume of email I send is too low. So there's that.

On a sidenote : It's about time something like ARC comes out. If I understand it correctly, it should fix many setups that involve mail forwarding. That'd be a godsend for me, I'll probably set it up soon enough.

Eh, you have the wrong approach. Just do everything and assume they all contribute to reputation. Maximise, don't minimise, your appearance of being a well-run ESP that takes internet citizenship seriously. For example, zone signing: I don't know if it makes a difference to domain reputation, but it might, so I do it.

DMARC is not a nuclear option, no-one starts out with the reject policy, in fact most of my domains are still p=none and I monitor the reports.

You absolutely can and should - irrespective of how much information it gives you in return - verify your domains in Google's Postmaster Tools, monitor your IP addresses and junk mail tagging at Hotmail via Microsoft SNDS/JMRP, and setup DMARC reporting e.g. using Postmark's weekly digest.

I strongly suspect your lack of TLS is hurting you in particular. For this and other reasons besides I advise you to enable TLS for all inbound and outbound. And use a real cert.

How are you subscribed to HN replies? I had something set up a year or two ago, but it stopped working.

http://www.hnreplies.com/ works well for me.

Thanks!