An email list’s open rate drops over 50% after changes to Gmail spam filtering | Hacker News Reader

An email list’s open rate drops over 50% after changes to Gmail spam filtering

detailed.com

130 pointsiamben151 comments

As much as I'd like to reduce my reliance on "cloud" services, Gmail is one of those that I just can't quit, since they've pretty much solved the spam problem for me. I'd rather a few false positives get filtered than the deluge that comes with inadequate (self-hosted) E-mail.

Sorry if this guy's "audience building" business is affected, but as an end user not worried about "my open rate", I appreciate Google's aggressive filtering and urge them not to let up.

That’s fine for you, but believe it or not, there are actually people who sign up for newsletters voluntarily because they want to stay informed about something. Google shouldn’t try to override the user’s preference about receiving newsletters.

Mark a few of the messages as important and you won't miss future ones. Given how pervasive getting signed up for email lists is even when I don't really want to be on them, the opt-in model makes more sense for me than the opt-out model.

True, but the solution should be double optin not something managed by Google.

Mailing lists that aren't double opt-in are way more likely to be managed by Google on behalf of the user than those that are. The spammy mailing lists are way more likely to get spam-filtered by users who were "opted-in" to them by the mailer.

Google's solution is necessary because bad actors in email abound.

More or less any business that sells domains also offers POP3 or IMAP mailboxes with spam filters and some web mail interface (which I'm not using).

That's what I've been using for my two primary domains (vanity personal and business one) since 2004 and 2006. Those filters work pretty well. I can't compare with Google's one but they're enough. I prefer to get some extra spam than losing a business email and maybe a new customer.

Maybe there is a way to keep using Gmail, read from an external POP3/IMAP mailbox and disable Google's antispam on messages fetched from there.

Let us know how you feel when Google starts filtering out transactional emails for your company and your chargeback rate goes up as a result.

A reasonable sentiment, for sure. I'm curious though, have you tried self-hosting email recently? Spam-assassin with razor2, pyzor, and Bayesian learning is really working pretty well for me. I train every 6-12 months based on my most recent 10k spams. I get a few false negatives but I can tune my spam threshold so that swings to a few false positives with one quick vim session.

Dovecot + Thunderbird even gives you client editing of server-side filtering based on all sorts of criteria, and self-hosted webmail (e.g. MailBox) is pretty good these days as well for when you're on the go.

It's been on my list of things to do "some time later," as I understand the tools have gotten much better. I suppose my belief that self-hosted E-mail necessarily means inadequate spam filtering may be a relic of a time long past.

Spam-assassin soaked up too much memory for my cheap, tiny virtual server. I recently set up rspamd, which seems to work as well and, while it is still the largest process on my machine, hasn't forced a reboot yet.

Dspam used to be great, but now it's mostly unmaintained :(

Urg.

The thing is, I run my own personal email server. It's not used for spam ever. It's only personal email to family members, friends, and the odd reach out to support. The volume of email is so small it's ridiculous. I might send up to 10 mail (1 receiver each) per month. I set it up with SPF and DKIM properly, triple checked.

And yet, for whatever reason, sometimes my email end in spam. Generally when I send attachment (like pictures). But sometimes it's just completely arbitrary.

How about this : for you it might be a "few false positive", because so much of the wanted mail traffic originates from google in the first place. But for the few of us that use mail as was intended, this becomes a huge problem.

Having a mailing list doesn't necessarely mean the content is unwanted. I'm subscribed to HN replies for instance. I want those email. If it started going to spam, I might consider "hey it's a few false positive". But for the HN replies guys, it would be maddening to be unable to fix the problem because Google can't be arsed to have a support channel.

From my POV, Google is way past "aggressive filtering" territory and way into "monopolistic behavior".

Yeah, I’ve seen the same, to the point that I’ve actually considered getting a Gmail account for my business, because customers not receiving important e-mail is bad for business.

And I think that's half their goal. Why fix something if it gets you more customers (direct or roundabout)?

Yeah, it certainly looks rather sinister.

Ok so as much as I promote self-hosting email, you're right. I've also had trouble with mail originating on my server to gmail and only gmail accounts. I did recently discover that I had some configuration issues via mail-tester.com though (my rDNS didn't match the HELO of my server) so now that I'm getting a 10/10 on that, things seem a bit better.

I definitely felt that gmail was being too aggressive on this too. I sent a bunch of detailed replies to middle/high schoolers working on science projects when they reached out through my public-education webpage and when they went to gmail I never got a reply but if they went to some other server, the conversations usually continued nicely. That was pretty frustrating.

How are you subscribed to HN replies? I had something set up a year or two ago, but it stopped working.

http://www.hnreplies.com/ works well for me.

Thanks!

If you haven't already, it can't hurt adding your sending domain to https://www.dnswl.org

Thx for the hint, I didn't know about this ! Just added my domain details to it, hopefully it helps a little.

> I set it up with SPF and DKIM properly, triple checked. And yet, for whatever reason, sometimes my email end in spam.

It's probably your IP address.

Sure, SPF and DKIM say you are allowed to send these emails, you are who you say you are, etc, but you also aren't a Sendgrid IP address (or someone similar) so you're treated as less trustworthy.

I could be wrong, but I believe many email servers take these things into consideration (please let me know if I'm mistaken!)

This is fairly common - if you want a method that gets you around this at the cost of some privacy, you can use Mailgun or a similar service for free as your relay, which should pass most spam filters without issue.

Mailgun has similar issues though. We advise those signing up for accounts on our open source community site not to use Hotmail since mailgun emails tend to get binned by Hotmail. Not sorted to spam or bounced, straight up deleted.

Is this anti NN.

No.

However, it's still Real Stupid.

I'm pretty happy with the results of spam filtering based on sender reputation.

You can appeal to DNSBLs to have your IP unlisted.

Not always feasable.

I run my email through google for my domain, and the IP assigned to my mail server is actively used to send spam (like, daily active reports on SORBS), so I can't even fill out their form to get it unblocked.

As a result, I need to email my bank from a different email address. So stupid.

>I run my email through google for my domain

Well, here's your problem. Have you thought about self hosting?

If you're literally running an e-mail server from your cable/DSL connection, you're much more likely to be marked as spam, regardless of doing everything else correctly (SPF/DKIM/reverse DNS), because there are lists like SORBS DUHL [1] and RBL-DNS DUL [2] that index these. Some servers block entirely, others just use it as an indicator that your mail is more likely to be spam.

Many spam filtering systems work on a 'points' system, so being on the dynamic IP list adds a lot of points, and if you also have other things that add points (being a mailing list, having 'unsubscribe' links, attaching images, having links) it might be enough to put it over the edge and get marked as spam.

You can do a lookup [3] to see if you're on any of these lists.

[1] http://www.sorbs.net/faq/dul.shtml

[2] http://www.rbl-dns.com/dul.shtml

[3] http://multirbl.valli.org/lookup/

Although what you say is true, I've been running my own mail server out of my house for years with proper spf/dkim/dns and have never had problems of my mail being sent to spam.

I had my personal domain hosted with Google. One of the reasons I moved away is that emails that I was sending to other Google Apps domains were bouncing as spam when the destination address was an alias. The bounce came from Google Groups, which is what Google used for handling aliases within Apps domains.

e.g. https://pastebin.com/u48DAaLP

That particular example was sent via SMTP, but I had the same problem when sending via the Gmail web interface, and it occurred sending to at least three different Google Apps domains.

After I moved my domain off Google Apps (I switched to Fastmail for a variety of reasons, but that issue was the kicker), I was able to send to those same addresses without issue. In fairness to Google, I was on the Google Apps free tier at the time, so there was nowhere to go for support.

ARC was designed to fix that kind of edge case, and Gmail was one of the first to start signing and validating ARC within the last couple months - so this may no longer be a problem.

http://arc-spec.org/

What's the edge case? That the destination address is an alias? That I was using a personal domain? That Google implements aliases using Google Groups?

I never understood what was causing these bounces. The entire email chain occurred within Google's control, passed SPF and DKIM checks, my domain isn't on any blacklists nor used for spam that I know of, and isn't used for anything but personal emails. And I've had it registered for years. Nor was it the content of the message. I had a friend with another Google Apps domain setup an alias for testing and when I mailed that address it bounced the same way, regardless of the message body. Emails to other addresses in his domain were not marked as spam... it was only Groups aliases that bounced.

I set it up with SPF and DKIM properly, triple checked.

Check your reverse DNS as well. It helps if it matches the HELO advertised by your mail server.

Whether by monopoly or sheer blindness, Google leans really heavily towards only letting massive institutions through.

I know a few people whose professional work involves a lot of delivering explicitly-requested, non-spam emails. With most email providers, they don't go to spam because they aren't spam. If they get flagged as spam servers (usually right after spinning up an instance, when the confidence is low), they can contact someone and get it sorted out.

With Gmail? They have extensive lists of tips and tricks to "sneak through" emails that their users specifically asked for. They 'prime' servers with extra-safe traffic to avoid getting frozen in the low-confidence early emails. And they still regularly run into problems. They just don't have the size to be trusted by Google, or to get business contacts to solve the problems that do arise.

It's a pretty concerning state of affairs - Gmail will deliver anything sometimes, but it seems to be a very high bar (vastly higher than "not spam") to ensure you don't regularly drop messages.

This is exactly my experience sending legit commercial emails to gmail. We jump through every hoop, we implement every suggestion, and they drop our email on the floor every time. I have no problem with deliveries anywhere else. They DGAF.

The idea that their spam filter is particularly good is ridiculous too. I get more spam to my gmail accounts than to my self hosted mail accounts with greylisting and spamassassin.

>greylisting

Don't do that! Emails should be instant, not with stupid greylisting delays.

I will do that, and you should too.

Nothing in the RCFs says emails are supposed to be instant. Reliable, yes. Instant, no.

First, greylisting is HIGHLY effective. Something like 95% of email connections are spam. Almost all of them fail without ever hitting disk, via greylisting. If they actually retry, then the chance that the sender IP or email content is in an RBL is MUCH higher, so spam that actually retries will be more likely to be rejected at HELO by an RBL or filtered after it is accepted. The incidence of false positives with grey listing is VERY VERY low. Using greylisting means that very few emails need to be accepted and then filtered. Those are the false positives that really hurt. The sender receives no bounce, and the recipient probably doesn't look at his spam folder every day, especially when it is full of hundreds of junk emails. Greylisting makes email more reliable.

In addition, better greylisting systems note senders that reliably retry and automatically whitelist them. My mail server actually delays less than 2% of legitimate emails due to greylisting.

I'm completely confused by Gmail's spam filter. We've already discussed its extreme tendency to false positives, but it's quite good about false negatives - no spam hits my inbox.

Except... Google claims >90% of spam never even reaches your Spam box, only the ambiguous messages. So why do I get 10x the spam at Gmail as at any other account? Are they not actually good at detection? Are they just a high value target that gets hit more often?

Or (my guess) are they trying to get clever? Google Doc's spellcheck is infamously bad because they're trying to use machine learning approaches without having a simple English dictionary as a fallback. Is Gmail trying to detect solely on email sender/content and totally abandoning approaches like greylisting?

Fully agree. I am in the same situation as you, and I even relay my emails through my ISP SMTP to avoid the issues such as 'residential IP marked as spammer'. And yet, Gmail will randomly classify simple personal emails. It is the only mail provider to act like this, I never had any issue with any other provider (ISPs, CSPs...) And to add insult to injury, they have this page: https://support.google.com/mail/answer/6227174?hl=en&ref_top... but as I send too few emails, it is completely useless.

> Having a mailing list doesn't necessarely mean the content is unwanted. I'm subscribed to HN replies for instance. I want those email. If it started going to spam, I might consider "hey it's a few false positive". But for the HN replies guys, it would be maddening to be unable to fix the problem because Google can't be arsed to have a support channel.

I'm not sure this argument is related to the rest of your post, but FWIW, GMail/Inbox puts emails from mail lists into their own "Forums" category, not in the spam folder.

SPF & DKIM are only the start of the story. Use a non consumer IPv6 address, TLS, and DMARC. Have perfect DNS (including correct reverse lookups). Sign your zones. Monitor your IP and domain status with every reputation list on earth. Sign up for the postmaster tools at Google and Outlook.

Running your own email ain't a hobbyist thing anymore, you gotta play like you're in the NBA. I'm doing all the above, my deliverability is solid. And pay attention to the changing landscape - for example, I will be implementing ARC real soon.

> Running your own email ain't a hobbyist thing anymore, you gotta play like you're in the NBA

I'd disagree. I've been running my own just fine for years. Yeah, you have to put some work into it, but spf/dkim/dns have been suiting me just fine.

I think this is a matter of perspective. My response was to someone who, in my assessment of what they'd listed, had more of that work to put in. Note they were also talking of using email as a file transfer protocol, which you may not be doing. Also your years of operation may (as with my 17 years of spamless service) have built a very positive reputation for domains and addresses.

I mean, DMARC, ARC and TLS make perfect sense (though I have none setup because DMARC is nuclear, I didn't know about ARC, and TLS is just up for the submission port for me). But signed DNS ? What does that protect against, malicious registrar ?

And the postmaster tools don't work for me on google because the volume of email I send is too low. So there's that.

On a sidenote : It's about time something like ARC comes out. If I understand it correctly, it should fix many setups that involve mail forwarding. That'd be a godsend for me, I'll probably set it up soon enough.

Eh, you have the wrong approach. Just do everything and assume they all contribute to reputation. Maximise, don't minimise, your appearance of being a well-run ESP that takes internet citizenship seriously. For example, zone signing: I don't know if it makes a difference to domain reputation, but it might, so I do it.

DMARC is not a nuclear option, no-one starts out with the reject policy, in fact most of my domains are still p=none and I monitor the reports.

You absolutely can and should - irrespective of how much information it gives you in return - verify your domains in Google's Postmaster Tools, monitor your IP addresses and junk mail tagging at Hotmail via Microsoft SNDS/JMRP, and setup DMARC reporting e.g. using Postmark's weekly digest.

I strongly suspect your lack of TLS is hurting you in particular. For this and other reasons besides I advise you to enable TLS for all inbound and outbound. And use a real cert.

I switched to FastMail from GMail earlier this year, did not notice any increase in spam to an address I've used publicly for at least 15 years.

I've been on Fastmail going on a year now, and spam was my biggest worry coming from gmail. But honestly I don't think I've received any spam in my inbox at all since doing the switch, and only occasionally had to check the spam folder for a missing email I was expecting. It's been surprisingly solid I think. I'm very glad to be off gmail.

"I'd rather a few false positives get filtered than the deluge that comes with inadequate (self-hosted) E-mail."

The main reason you need a spam-filter is because your address ends with @gmail.com. Man I've lost important mails due to gmail, I'd never use it for anything remotely serious. Works fine for interactions within google-services though.

But apart from that there is no decent client and their notification support is not reliable either, nor can you rely on their spam-filter. Why even bother with email if you are going to use gmail?

> As much as I'd like to reduce my reliance on "cloud" services, Gmail is one of those that I just can't quit, since they've pretty much solved the spam problem for me.

It's overrated. And like mentioned in this article: it makes regular email delivery unreliable.

For me FastMail has largely replaced gmail. I'm much more happy with them than I ever was with gmail.

And they have an actual support channel.

Edit: the "gmail is the only real option" meme is getting pretty tired. They were clearly the best option 10 years ago, but times have changed. No need to repeat old truisms.

Back before I switched to fastmail I'd sometimes stop receiving newsletters. Because Google randomly decided the newsletter I've been receiving for ages and never marked as spam is suddenly spam.

Fastmail on the other hand only classifies spam as spam (with way less than one spam mail per month getting through) which for me is far superior to Google's "I have to check the spam folder regularly because anything can end up there"

I would love if I could set the spam filter to include "audience building"/"growth hacking" emails. I have TENS OF THOUSANDS of emails in my inbox because every startup wants to "boost their engagement". Startups have given me Pavlovian email anxiety.

And what about email newsletters you subscribed to? Those you _want_ to receive?

Because is about those ones the article is about.

> Gmail is one of those that I just can't quit

Literally just started to switch back today from Fastmail

> since they've pretty much solved the spam problem for me

#1 reason. I tried Fastmail for a month or two but I've only just hit the 200 spam messages to train my "personal" spam filter, all of them "greetings of the day" website/SEO related. Realised I've been fighting to get to this personal filter level and then when it enabled.. I received another spam message.

I'm out. Love the idea of not relying on Google for everything but I guess for now spam filtering is just one of those things Google does good.

You just barely trained your spam filter and are surprised that you got another spam? Why are you giving up so early? The spam filter only gets better over time, and you're quitting without giving it a chance.

Also note that you can tweak your spam preferences. For example, I have mine set to mark things at spam at a score of 4.0 instead of the default score of 5.0.

Gmail user here: I like the fact that Gmail's spam filter just works. The false positive rate is incredibly low for me.. things rarely end up in my spam folder that don't belong there. And I didnt have to configure anything. Why should I want to struggle for so long to get the same level of functionality?

Back before I switched to FastMail, I did actually have to keep an eye on my Gmail spam folder, because stuff I wanted would end up there every now and then.

And the very subject of this article is emails that people want to receive getting filed into spam by Gmail (but not by other mail hosts).

It was the #1 reason. Other issues were I had about 480 emails in my inbox. I'm not sure where the difference is but I've always managed an "Inbox Zero" with Googles' Inbox

There are other things I missed from Inbox too such as reminders "hit t - set reminder", snoozing emails till another time and I have a few people that contact me via Hangouts so I ended up loading Inbox occasionally anyway to see those messages.

No fault on Fastmail really, if I'd never discovered Inbox I'd still be using it. In regards to spam I just can't be dealing with training a spam filter, I was reminded it was one of the reasons I switched from self hosted to Gmail in the first place.

Edit: For what it's worth the SEO and website spam emails weren't getting a spam rating at all, most were <2.0

Edit 2: Also while looking at the spambox for that last message I now realise a lot of my server backup status messages were going into the spam box with a 7.1 spam rating.

Edit 3: I'm going to stop editing after this one. Edit 2 was on me, the backup messages were sending direct rather than via mailgun.

FWIW there are other mail clients that add snoozing behavior to arbitrary mail providers. For example, the Spark app will snooze emails on my FastMail account. I don't personally use that functionality, but I know it exists.

I switch between mobile and multiple desktops frequently unfortunately, appreciate the suggestion but I don't think that would work for me

Spark has both an iOS and a Mac app, though if you're using other platforms that doesn't really help.

Fraid not, I'm a Linux sorta guy. Very jealous of some of the tools Mac gets but not quite enough to budget for a Mac :)

That's hilarious: A newsletter about how-to-spam is treated as spammy and the author is upset. In the end, everyone thinks they're one of the good guys.

> The aim of this blog is to help anyone with a website find proven content ideas that they can use to drive more traffic and links to their own sites. The concept is already working, with one recent article helping a reader get on the front page of Product Hunt.

This is instantly what I thought too. Essentially this guy is teaching how to get rich by using SEO, catchy gimmicks/ideas, and the likes. The spam teacher themselves get filtered, how ironic.

Yes, welcome to the world of Viperchill - the shadow king of get rich by teaching others how to get rich schemes. I wouldn't be surprised if Google personally targeted him, I would do :)

Sadly this is happening to a lot more people than just me otherwise I would not have written about it.

Heh - I wouldn't be surprised if the language used sets off the filter. Machine learning isn't great at abstractions like "writing spam" versus "writing samples of spam someone else might write", so I imagine this sort of topic is particularly prone to filtering.

About how to spam?

Lol, that's quite the jump you made there.

If you read the article, you'll find there are many other industries which (suddenly) have the same issue.

Exactly. World's smallest violin plays.

With a title of "Launching my new daily blog (and podcast)", and sending in bulk, I'd expect a good spam filter to send it to spam. "I’m Starting a Brand New Live Case Study. Over one hour of free training in today's video..." was also flagged as spam. On keywords alone, those titles just scream "spammer".

I only skimmed this so forgive me if I missed something. But could this simply be what happens to you when a sufficient number of email recipients classify you as spam, and you then reach some critical threshold?

>> I changed my sender email address to see if would help

But if the pattern of content got flagged as spam by Google users, the sender address might not be the only determinant.

>> I was relieved to discover that others with newsletters were having similar troubles very recently.

But if this results from crossing some sort of user-classified spam threshold, one would probably always be able to find some recent "victims".

This is not intended as a criticism -- I'm sensitive to Google's seemingly arbitrary power -- just an idea to consider or rule out.

That seems an unlikely thing to have happen to multiple high-profile newsletters at once in a short span of time. Particularly, if these people aren't seeing big changes in their users, why would the spam threshold be hit now? Did a large chunk of email readers suddenly decide enough is enough?

What if it happens to some high-profile newsletters over every/most any short span of time you might pick?

Then you'd expect this article to appear on the FP every week, surely? It seems more likely to me that whenever people write about something out of the ordinary, it would not happen ordinarily - otherwise there wouldn't be anything to write about.

No you'd expect people to be complaining about it somewhere at some frequencly...and it hasn't been established that isn't the case. However, if you search for "Google newsletter suddenly spam folder", you can easily find lots posts that are superficially similar-looking, from various time periods.

Maybe Google changed the threshold? Lets say 20% of recipients have historically classified the list as spam, and Google just adjusted the spam threshold from 25% to 15%.

That would be exactly what the article is describing. GGP was suggesting an event which Google didn't directly control.

Another possibility is that the spammers have adapted and rolled out the next phase of their plans. After creating a million or three Gmail accounts (aged, so they're all months old, and staggered, so they're not all created at the same time, and with some amount of fake "organic" traffic such that they seem they're owned by real people), the obvious first phase, as a spammer, is to sign up for your own newsletters, and mark those as "not spam" from your three million fake accounts.

As a spammer, you've got to assume Google's going to catch onto your first three million fake accounts and mark both those accounts, the email servers' IPs you use to send, and the email list itself as spam. Consider your first three million gmail account burned by that point.

The less obvious second phase then, using your next three million fake, aged, but-with-fake-email-traffic-to-make-them-seem-real Gmail accounts, is to sign up for not-spam newsletters, but mark those as spam. The wishful-thinking hope is that with enough of those marked as spam, the wider public will see more articles like the linked article, and ever so barely push back on Google, who then have to figure out another way to distinguish signal from noise in a vastly complicated and ML-enchanted algorithm.

I suspect it's a consequence of Google not caring whether or not you're actively subscribed to the newsletter, and treating it as spam based on all other signs.

Checkboxing "[ ] I want your newsletter" on some website doesn't inform Gmail to whitelist that site.

Similarly, it could have been 3-4 consecutive newsletters with particularly low engagement that put him across some threshold. Not people marking as spam, just not opening or opening and quickly deleting.

Personally, I can't remember the last time gmail marked anything important spam. Maybe a newsletter whose absence I hadn't even noticed, but nothing actually important to me.

I just wish I could get gmail spam filtering on my cell phone. Robocalls are out of control -- 4-5 / day. (I've looked at some of the apps / subscriptions for this, but I'm not ready to give a relatively unknown 3rd party access to my contacts or to adding contacts, which they generally require.)

Try making google voice your main number? They had solid robo filtering if I remember right. I'm on Fi now and I can't remember the last time I received a robo call.

Adding myself to the no-call list and marking the numbers as spam in the phone made them disappear for me.

> a sufficient number of email recipients classify you as spam

From Google's end, I can understand this decision. But from the consumer end, I only want that logic used for de novo email sources.

Even if many people think a newsletter is spam, I'd want my personal patterns to override that. If I've opened letters from the same sender repeatedly and never marked one as spam, it shouldn't much matter what other users have to say. If I've ever explicitly taken a sender out of spam and not put them in spam, that should basically settle the matter.

I regularly lose emails I've requested to Gmail spam, and even when I've un-spammed the same sender multiple times. I appreciate that Google achieves pre-emptive blocking via a large userbase, but I don't appreciate when the wisdom of the crowd overrules my own behavior.

Google sends me reports on the spam we get to one of our mailing lists.

I hadn't seem them in a while and found that they themselves were being sent to the spam folder.

edit - I'd forgotten about this actually, and checked, it happened again today.

It's apparently happening because people send me spam, so I mark it as spam. Because I've marked things sent to that email address as spam, they're saying their own report is now spam. The description of the problem is sounding more python-esque as I continue so I'll stop there.

Can confirm — Google's own Spam reports are going to the Spam folder. Was it the word Spam that triggered it? lol.

The spam problem has never been solved, no matter how good spam filters may have become in the recent years. It's still an arms race, with a front that is moving one way or another every few months. Even today, a spam filter is a piece of software that you trust to throw away e-mails without letting you read them, based on its own decision that you don't want to read them.

It seems insane to me that people would willingly let a program decide to throw away inbound mail based on unspecified rules. And I nearly start frothing at the mouth when someone suggests that, since their software decided to throw away my e-mail, it is somehow my responsibility to fix that. It isn't.

If there were thousands of different anti-spam solutions, and one of them behaved badly, then senders wouldn't care, and users would vote with their feet (or wallet, or app store) until the behaviour was fixed. But since Google has a near-monopoly on e-mail, it's now up to the senders to do the work and jump through the badly-documented ever-changing hoops.

Do they really have a "near monopoly" on email? How near are we talking here?

In the younger, private mail account demographic, I’ve seen as much as 80% of some e-mail lists being Gmail accounts. For business, Exchange is still king, but Gmail is gaining on them.

It's scarey stupid how much the "big 3" own email now. Google, Yahoo, Hotmail. When any of them change up their spam fighting techniques it causes ripple effects across the entire email ecosystem now.

Have Yahoo mail subscribers on a Mailman mailing list? Not if you're not running the latest Mailman which has a new workaround just for them. Yahoo started bouncing legitimate mail a couple years ago due to the normal way mailing lists have ALWAYS handled sender addresses, since before Yahoo existed.

I get it, sometimes desparate times call for desparate measures (for fighting spam). But jeeze, it's super annoying to have to break your package manager to get an upstream mail server to support one user who uses one certain mail hosting service.

I send newsletters (that people double opt in to and want to receive) to about 360k developers each week and a good 90% are on Gmail or have their domain on Gmail. This is very specifically developers, however.

> It seems insane to me that people would willingly let a program decide to throw away inbound mail based on unspecified rules.

See, you say that, and until a few days ago I'd have felt the same. But that was before I registered the first .com domain I've ever owned, and oh boy has that brought them out of the woodwork - I've gotten more spam in the last week than in any of the past ten years prior! Not just email, but phone calls, too; I even heard from one of those tech support scams, and it's been so long since the last time that happened that I couldn't even tell you when it was.

It seems to be settling down in the last day or so, which is nice. But if it picks up again, I'll absolutely be installing SpamAssassin on my MTA host, for the first time in the going on two decades I've been running my own mail.

To be clear, I totally agree with your point about Gmail - it's far too centralized and unaccountable for my own comfort, too, which is part of why I don't use it for any mail I care about. But a lot of my correspondents do, which means I'm nonetheless in the position of either taking assiduous care to maximize my deliverability, or risking my mail landing in spam folders that nobody ever really looks through. Not a good situation, but at least until someone comes up with a plausible means of addressing it from the outside, here we are.

I don't know where you registered your domain, but I have like 100 domains, and I've never been contacted about them (except by the registrar of course). Most of them have valid and router MX.

GoDaddy, and maybe that's my problem. On the other hand, I've been doing business with them for many years, and never had anything remotely resembling this happen before. I also didn't use whois privacy, but, again, I've made that choice on many domains before, and likewise never seen anything like this result from it.

This is due to a new business model. Where companies scrape ICANN data base for newly registered domains and then sell it to service providers.

That squares with what I'm seeing, yeah - a lot of cheaply automated nonsense trolling for a few suckers.

Is there anywhere I can look for more information on how this model works in detail, and how to avoid the same problem in future? It's a real pain in the neck. Thanks!

Best way is to opt for whois protection in your first year or for first few months. Here is a mail I received offering the service.

Hello,

My Name is Prk Aa

We are providing whois data in very low price

Daily more than 100,000 domains are registered

Reply Me with your contact number & skype id.

Warm Regards Prk Aa

Lessons learned, I suppose...

Except it's not new at all. Since I first got my email on my primary domain in 2008 or so I've had admin@, postmaster@ etc., Along with the address in whois all being spammed.

Some domains have been used as shit magnets and you can inherit that mess if you pick it up quickly after it's been dropped.

I wouldn't believe it myself, until it happened to me. I've run small mail servers for various domains for years with only "average" amounts of spam. Then one time I picked up a personal domain of my handle (not this one) which I'd wanted for years and I discovered it finally was free. The moment I setup an mx for that domain I got a firehose of spam coming in to it. In fact it was so bad I had to unreg the mx and leave it for another year before it was worth doing anything email-wise with the domain. Just the bounce processing was driving cpu and bandwidth more than I cared to deal with, even if it wasn't receiving any local mail.

I checked the IP and the domain before I went with either of them, and their reputations are both clean, so I don't think that's it. I've definitely had that kind of trouble in the past, though.

Wait until you start getting physical mail based on your DNS entries. :-)

It'll take some doing for anyone to beat the long conversation I had to have, years back, with a representative of a very large film studio, who were concerned that a domain I'd registered might verge uncomfortably closely on their mark.

Worked out in the end - it only took them a few minutes actually looking at the content to realize, on the one hand, that it was clearly not being used for any commercial purpose, and on the other, that there was no meaningful risk of confusion between my domain name and their mark. They were happy to wait to buy it until I was done with it. But it was a surprising call to receive at seven o'clock on a Saturday evening!

I get more than 3 hundred spam emails daily. My inbox would be absolutely unusable of it wasn't for spam filters.

I don't want to turn it into a contest, but across my domains I receive in excess of 3000 spam emails a day.

At least personally as far as I can tell the spam problem has been solved by gmail. I can't even remember the last time I saw a spam message in my inbox and can count on one hand the number of times I've had to go into the spam folder to find a legitimate email.

> Even today, a spam filter is a piece of software that you trust to throw away e-mails without letting you read them, based on its own decision that you don't want to read them.

The way I set up rules on my mail server, they all just go into a spam folder that I browse every once in a while. If anyone is going to delete my shit, it's me.

It's not a Gmail issue!

By using an ESP (like Aweber, Mailchimp, etc) means that you are sharing part of your "reputation" with thousands (or even millions) other senders. So even thou most of the ESP are quite efficient in cleaning-up their customer base from spammers, the more clients they have, the more it's difficult.

Here you can find an pretty accurate and updated analysis of ESP's clients "quality": http://mainsleaze.spambouncer.org/may-2017-in-spamtraps-esps... Aweber is #6, which is not a good signal.

That said, you may switch ESP by choosing one that provides a you custom "identity" such as dedicated IP, DKIM, PTR, SPF records (which usually means quite a lot of setup and costs) or choose an ESP that doesn't have tons and tons of clients, eventually with a free version.

http://www.emailvendorselection.com/ is an updated ESP directory. You are of course also welcomed to try mine: http://www.mailup.com

" such as dedicated IP, DKIM, PTR, SPF"

DKIM, PTR and SPF really ought to be enough, right?

Yes, but you can't have a custom PTR without a dedicated IP.

When I saw the friend was using Mailchimp, it made a lot more sense.

I get lots of spam from Mailchimp (ie. people subscribing me to their newsletter, without my consent), and complaining to Mailchimp about it seems to have no effect.

So since Mailchimp allows bad actors to send mail with their service, it’s not entirely unfair that Mailchimp mail gets flagged as spam.

The author uses Aweber, not Mailchimp. Besides, Mailchimp is quite strict with their opt-in policy and they don't hesitate to suspend accounts that ignore it.

One of the author's peers uses MailChimp and saw the drop. https://detailed.com/wp-content/uploads/2017/07/paul-jarvis..... That's in the article.

"Did you read the post"...that always shuts down discussion.

You're right, it's unnecessarily hostile. I edited that out.

> The author uses Aweber, not Mailchimp.

Sorry, I wasn’t aware that was a mail service, my bad. His friend (from the iMessage convo) was using Mailchimp, though.

> Mailchimp is quite strict with their opt-in policy and they don't hesitate to suspend accounts that ignore it.

They might say they’re strict, but almost any time I give my e-mail address some where online, I’m automatically signed up for some Mailchimp newsletter, and Mailchimp can’t even be bothered to reply if you report it.

Mailchimp does not care at all. I unsubscribe daily for lists never subscribed to. They come back and spam again...

> Mailchimp is quite strict with their opt-in policy and they don't hesitate to suspend accounts that ignore it.

And still I get quite a bit of Mailchimp-sourced spam I never opted-in to.

Gmail is not magically going to white-list recipients for receiving some spam that they opted into on some random website it knows nothing about.

People sometimes opt-in in good faith, and then when they get the e-mail, they realize it is garbage. They tell their e-mail client that it is junk, rather than going through the ritual of unsubscribing. Probably, in may cases, they probably don't even make the mental connection between what they received and the site they visited where they checkboxed something; they don't recognize that what they're receiving is something they signed up for. Their reaction is "WTF is this crap?" and report it as spam 500 milliseconds later without giving it another thought.

If enough users flag the posting as spam, Gmail will probably learn that and help the remaining users.

After reading this I checked my spam folder. Found an email from one of my best friends from highschool, who I've sent 100s of emails back and forth with, letting me know he would be in town in my spam folder... Well guess I didn't want to see him over the holiday weekend. GAAHAHAHHAH.

See as how a lot of my friends use their own domains / hosting for email I guess I'll be spending a lot of time in my spam folder.

What's really bad is your friend had no idea you never saw his email because it ends up in a spam folder and not bounced back instead

Google could get really evil in this regard. But then again Facebook has been doing this for years. "We didn't think the content was relevant to this audience. Pay us some money and we will allow your email to reach more people..."

This is already the case today. I get so much irrelevant mail in my main inbox that I ignore a lot of legitimate email in the process. And you can buy Gmail ads which appear in the top of your inbox.

Perhaps businesses are becoming too agressive at requesting email addresses, people are providing fake email addresses, and the recipients are responding by flagging it as spam?

I bring this up because I started receiving suspiciously legitimate spam a few years back. No red flags were raised by the email itself (e.g. no indications of phishing), the companies exist, the recipients exist, everything was geographically consistent. It was as though certain individuals were providing my email address as their own to businesses yet not to friends or colleagues.

When you consider how agressive some of these websites are with newsletters, I would not be surprised if people were providing fake email addresses just to get rid of the prompts. I would not be surprised if Google started flagging those newsletters as spam, even if the odd user genuinely wanted them.

The hilarious part about Google's spam filtering is that I have crons that send logs in emails to my own address using Google's own smtp servers (that require authentication with my own credentials) and they end up getting placed in spam.

I bought electronic content that was sent to me via signed-email and it got into spam ... somethings not right.

All the spam I get is newsletters from the perspective of the spammer. F that. My email should be for people.

Well, if you didn't sign up for it then it probably is spam. But lots of people want to get newsletters and if you want to get it, then it isn't spam.

Even if you did sign up for it it's probably spam. That's the real issue here. Given how aggressive businesses are with signing up people for unwanted newsletters it seems completely natural for Google to be suspicious of them.

Yep. I waste an hour (probably) every week unsubscribing from various newsletters and marketing campaigns that I don't care about. Most corporations have largely taken "this person has given me an email address" -- even if one is necessary to use their service! -- as free license to hit them with all sorts of marketing messages. Really, these are spam -- unsolicited, unwanted email. It's just that "spam" has taken a stronger, more negative meaning.

Some years ago I got a reimbursement from google regarding expenses they were paying me. I though they had totally forgotten about it and contacted them.

Even the google representative knew their own financial related emails were systematically being tagged as SPAM on gmail.

That's how aggressive it was.

I couldn't stop laughing for minutes.

I've had emails from family members go straight to gmail's spam folder on two different occasions, and a third occasion in which the mail appeared to have sent according to the sender's mailserver, but never showed up anywhere in my gmail account. From what I could tell glancing around some forum posts, mine was hardly a unique experience and is reliably solved (well, in the first two cases) only by explicitly whitelisting every single address from which you expect to receive correspondence. I'm glad to see more attention to this issue of overzealous filtering.

I work for a small non-profit and we've had the same issue. We use SendGrid to send out our email newsletters and the open rates have fallen from ~27% to ~13% over the past month.

Does anyone here have contacts at Google who might be able to help? [Edit: I've also reached out to SendGrid support to look into this]

Looking at the data more, the drop is due to a ~73% drop in Gmail opens. Hotmail and Yahoo opens remain stable:

1) See drop in overall open rate here: https://www.dropbox.com/s/j8grg3r684zshhz/Drop%20in%20open%2...

2) See drop in Gmail open rate vs. Yahoo and Hotmail here: https://www.dropbox.com/s/0fdvbddwno27mgf/Drop%20in%20Gmail%...

Since you didn't provide your domain name, I can't troubleshoot for you.

I recommend reading my comment https://news.ycombinator.com/item?id=14720344 and following the troubleshooting steps I outlined.

We use the news.pariyatti.org domain for these emails. If you're willing to help troubleshoot, that would be much appreciated. Otherwise I'll try to work through your recommendations, although a lot of it is very new to me so I'm not sure how far I'll be able to get. You can reach me via email at my HN username @pariyatti.org if you wish to discuss further.

In a previous job, my team was responsible (among other things) of the company's newsletter. Nothing great but nothing spammy either, and we made sure to follow anti-spam law by having the newsletter subscription being opt-in and having a clear and easy unsubscribe flow. We didn't send the emails ourselves, but rather used a big player in that area.

What I learned is that sending emails and not getting treated as spam by any of the big email providers is an absolute mess of tinkering and hoping it'll work. Not all of them would categorize our emails as spam, but Outlook.com started doing it out of the blue, and even a year later we didn't have a clue as to why or how to make it stop. Support for email senders is as good as for users, meaning getting anything else than a canned response wasn't happening.

I have zero sympathy for these people. My spouse for example often gets added to mailing lists when purchasing a product (just like we all do). Instead of clicking the unsubscribe link, they'll just "open" the email without reading it to mark it as read in their inbox, and immediately go on to the next one. These are essentially spam that my spouse is too lazy to mark as such. I have to wonder how many of the "opens" the author's lists get are just like this. "Oh, that stupid list I got subscribed to somehow. Mark as read." I have a very deep suspicion that they're mailings are essentially spam to a lot of people who receive them and this is correct classification by Google.

Just a anecdote, but this month there was a significant phishing attempt emulating an Australian electrical provider which got through Gmail's spam filter. Gmail has now updated their filters to detect it, but it's now catching genuine email as well.

That said, it's been a long time since Gmail has had a false positive that I was surprised about (i.e. not a spammy newsletter that I subscribed too)

DETAILED.COM has a bunch of issues that would easily trip SPAM filters.

-------------------------------------------------

TL;DR: DNS, SPF, DKIM & DMARC are not properly configured for DETAILED.COM. Fix and test the issues below.

-------------------------------------------------

Start here: https://mxtoolbox.com/domain/detailed.com/

1. SPF Records

Issue: You should only have 1 SPF record but you have 2.

Combine them. https://www.spfwizard.net/

Current:

  detailed.com.           3600    IN      TXT     "v=spf1 include:smtp.groovehq.com ~all"

  detailed.com.           3600    IN      TXT     "v=spf1 mx include:send.aweber.com -all"

Should be:

  detailed.com.  IN TXT "v=spf1 mx include:send.aweber.com include:smtp.groovehq.com ~all"

2. You SOA record is incorrect

Your current authoritative DNS servers are:

  detailed.com.           172800  IN      NS      ns.sourcedns.com.
  detailed.com.           172800  IN      NS      ns1.sourcedns.com.

Your current SOA is:

  detailed.com.           3600    IN      SOA     ns.liquidweb.com. admin.liquidweb.com. 2017042600 86400 7200 3600000 14400

In your SOA, ns.liquidweb.com is incorrect. It should be ns.sourcedns.com

3. Missing DKIM records

Check if AWEBER is sending the correct ones.

Use the steps outlined here https://www.phishingscorecard.com/ to find your DKIM key and test to see if it's working properly.

4. Setup DMARC to get reports on your failed email

If you had set this up, you would be getting notifications from Gmail, Yahoo, etc as to why your email is not being delivered properly.

I suggest doing a 14-day free trial via https://dmarcian.com/ Once you create an account, they will provide you with RUA & RUF addresses and process the XML responses for you. Don't forget to add the _DMARC TXT record to your DNS.

5. Test your email using https://www.mail-tester.com/ This site is fantastic and will help you narrow down on actual delivery issues.

The core of the issue seems to be an anti-phishing feature that Google seems to have dialed too high in some cases

But yeah, communication channels are getting more and more clogged with irrelevant communications (not even span alone) so it's getting tough to get across the barrier

Gmail is not a good provider if you care about seeing all of your emails all of the time.

It does a good job getting rid of spam but if you need to check the spam folder for missed emails then it makes sense to leave the spam in the main inbox and save a few clicks

Huh, weird. I haven't noticed this and I send a lot of email newsletters.

I wonder if they've started handing SPF or DMARC misalignment differently. I've noticed a lot of those kinds of problems with newsletter-type mail in the past.

Good thing everyone uses gmail right? Centralize everything under one company! Centralize all the things! I use gmail :(

If Google wasn't able to find the problem, then they probably should rollback.

They can't hurt their customers this way.

You say that like Google cares, over the years they've amassed a large swatch of consumer email users (probably due to Android) - they have basically gotten to a position where they set the rules and if it hurts you tough luck.

Email deliverability is a hard problem these days, spam has caused us so much grief that large email providers are doing really annoying things in their spam filters to combat the problem but I've noticed more and more legitimate email is getting caught in the crossfire. Unfortunately, I don't see this being fixed anytime soon - and it's personally making my life increasingly difficult as someone who had to give up self-hosting his email and switching to a hosted Zimbra service just to stop getting flagged by Google as spam (even though I passed DMARC/DKIM and SPF validation and had a non-blacklisted IP).

> as someone who had to give up self-hosting his email and switching to a hosted Zimbra service just to stop getting flagged by Google as spam (even though I passed DMARC/DKIM and SPF validation and had a non-blacklisted IP).

There is a trick how you can get out of the spamfilter: Send mail, regularly (like a few mails per week) to friends/family that use Googlemail, and tell them to mark your mail as "not spam". After roughly a month, you’ll have gathered enough points that Googlemail won’t mark you as spam anymore.

The people sending these newsletters aren't even their customers, for the most part. They're customers of AWeber, Mailchimp, etc.

And for the people receiving these emails? This is a nuisance at best. Although if 99% of the population stopped receiving emails from their friends, etc, THEN this would be a bigger mainstream issue. Right now? Nobody's worrying (especially in Google right now) except a few newsletter senders.

You mean their product? People don't pay for gmail, so they're not customers.

If you can't figure out what is being sold in the situation to keep the business afloat, what is being sold is you.

They're measuring "open rate" by putting links or images in the email that have uniquely identifying information, right?

As much as I'm frustrated with Gmail's spam filtering (I also use a personal email server, and it's annoying to get my personal email filtered by Gmail), I absolutely loathe seeing that tracking information in links, even if it's from a mailing list I specifically subscribed to. Often times they're newsletters that are also available on the web. I'd rather just go to ddg to search for the web version to read.

Of course, that's probably a case of https://xkcd.com/1105/

tl;dr have your SPF, DMARC and DKIM in perfect order, 'cos Gmail has started filtering hard on those.

Reply on news.ycombinator.com