Why does Gmail hate my domain? | Hacker News Reader

Why does Gmail hate my domain?

bitbin.de

310 pointsstbenjam142 comments

Hi, I'm Nicolò and I work in the deliverability team of Qboxmail.com. We have run into similar problems in the past. The reason of the behavior of Gmail with your domain is not easy to understand, a theory could be that in the past someone has used it for sending spam, and even if it was not your server, you are paying for this.

The solution we have used, it was bring Gmail to understand that your emails have been sent from a real user. You can simulate a conversation between your domain and other Gmail accounts. Send a first email, with a realistic text, to a Gmail account, if it goes in spam, mark as safe (remove it from spam), and reply to it, always with a realistic text. Continue to simulate a normal conversation, sending 4/5 emails between the 2 account. Re-do the operation from another Gmail account, but this time do the opposite: start from the Gmail one.

It not bulletproof, but it has worked in the past for us.

So the way to get off from spam blacklists is to send spam mails between two addresses? Looks reasonable.

You send emails that Gmail looks wrongly like spam. If everything is clean, it is reasonable to think that it is matter of little fixes, and to "help" Gmail to understand that domain doesn't send spam.

There is no way that this method would work id the domain it does actually send spam mails.

Could always start legitimate conversations with a friend who marks you as safe. :)

Sending about 250,000 emails per month for over a year now (all double optins), we found out that both gmail and yahoo has what we call "do you listen" rule.

Basically even if user expressly add you to their contact list and pull you out of spam, etc, if you sending them just your email ignoring what they reply back to you, eventually you get back to spam. I think this is very clever and is based on simple human rule that most follow when conversating over email: when people reply to email the do not start a new email, they hit reply and previous email from their convo lands on the bottom of this new email.

Once we started emailing new emails with adding content from addessee previous email to the bottom, our emails start getting back to addressee major mailbox.

And this make perfect sense on how to detect a real conversation (gmail has your sent items so if incoming email contains something you wrote before, its an indication you have a real live conversation with someone)

You are right. In the end, it always a problem of volumes and proportion between spam emails, conversation, ecc..

If your numbers are low, so even small change can make a big difference

Sounds like re-training Google's spam filter with a few false negatives (by marking them as such). Am I right?

exactly. That is spam detection programs work. If you re-train them, when they think that the volume of spam sent from that domain in acceptable and normal, they will let its email pass.

Google mail's spam filter delivers spams to users; it just classifies it into a different folder.

In this case, the mail was bounced at the SMTP level, and the bounce is (strangely) identified as coming from Google Groups.

"Mark as safe" will not SMTP non-delivery problems, I think, just the mis-classification of e-mail as spam at the folder delivery level.

Is there a way to automate this procedure?

If there was, it would defeat the point...

(There's a minor typo in your website: SMPT should be SMTP)

thanks! fixed!

Interesting - I cannot access this website because the domain is blocked by my company. We have 2 enterprise blacklist solutions to stop people from going to websites they shouldn't.

The reason given is "parked domain".

Edit: I am referring to the bitbin.de domain

Thanks for the warn. We are going to verify it!

It is a problem with the policies of your company. I don't know why they should have blacklisted our site, maybe they want to be sure that your email are managed by them :)

If you are curious, try to go on our italian website: qboxmail.it

Sorry, I was referring to the bitbin.de domain from which this post is written. I was providing vague proof that the domain's history may have caused it to be blacklisted.

Notice the comment on the story:

"I had the exact same problems in the past. For me it helped just to register and immediately cancel a trial of Google Apps for the domain. It’s annoying to have this as a necessary step but at least it was like that done in just a few minutes (after I tried to find a contact email of Google for a much longer time). I tried it because I had other domains on the same server which didn’t had this issue. All domains which weren’t filtered had a Google Apps account in the past. So I thought it’s worth a try, and yeah it was solved."

Thanks, may have to pay Google the $5 (or thereabouts?) to host my mail for a month and consider it a cost of freedom from them long term.

After 2+ years of hosting my domain myself, it doesn't seem possible to build up enough reputation with them personally.

If the Google Apps anecdote is true, it's likely to be a design flaw in their anti-spam systems - think of the number of spammers who could register domains, associate them with a $5 google apps account, and spam millions of Gmail recipients. If you think about the kind of metrics/outcomes the Gmail anti-spam team uses to measure success, it's hard to imagine a scenario where that would be good for them.

What this looks like to me is a reputation problem with one or more domains, either in your headers or your message content.

From a domain reputation perspective, you don't generally need to 'build' reputation in the same way you often do with IP reputation; rather, Gmail tends to tag domains with negative reputation only when they've observed unusually bad behavior associated with the domain.

Sometimes this can happen due to domain forgery or spoofing. Sometimes it's because a sender is doing overly aggressive email marketing. Sometimes it's just because your domain shows up in someone else's bulk email.

With full authentication, spoofing is unlikely; I also don't see any indication you're doing email marketing here, and your IP 88.198.5.136 doesn't appear to generate much more than a trickle of volume. So - perhaps the latter? Has anyone included a link to one of your blog posts in an email newsletter, for example?

It would also be helpful to see full headers and content; it's great you're doing SPF/DKIM/DMARC, but they're not going to prevent all issues - there might be something elsewhere in your message that gives us a better idea where the problem is.

Finally, keep in mind that anti-spam systems are highly dynamic, and results can change - sometimes in as little as a few minutes - based on the latest data feeding into the system. If you're lucky, you might see this resolve on its own fairly quickly.

you are absolutely right, and this will most likely be fixed tomorrow morning as a result of this thread.

There's a difference between

>anti-spam systems are highly dynamic, and results can change - sometimes in as little as a few minutes - based on the latest data feeding into the system. If you're lucky, you might see this resolve on its own fairly quickly.

and

>this will most likely be fixed tomorrow morning as a result of this thread.

How was she 'absolutely right'?

I didn't take Boris' comment at face value. I took it to be a joke.

I think he understands that it's different. He was humorously implying that the issue will be resolved by a Gmail person seeing the thread, as opposed to some AI spam filtering.

yes but not totally joking, I've definitely seen that happen with G a few times in the last few yearsq

Google does read threads about them, but seldom comments. When Google Local first came out, I mentioned on Slashdot that most big companies tended not to "claim their page" on Google Local, and that Google hadn't even claimed their own page on Google Local. Google claimed their own Google Local page within hours.

If its true, its likely it's only a small part of the score but who knows, it might be enough to push my domain into the good graces of gmails spam filters.

I doubt you'd get away with doing it for a domain sending legitimate mail, or at least not for very long.

They offer a free trial. I don't think you even need to pay them the $5.

If the domain is bitbin.de and the mail server is hosted on the same server, it's probably Hetzner being scorched-earth for deliverability, just like any other major hosting provider will be. Email from AWS, Rackspace, Heroku etc. get a drastically higher starting spam score because people fire up servers there to spam.

Running the email through something like Amazon SES or Mandrill would probably be helpful. Both have generous free tiers.

This seems to also be the case with DO, I setup a mail server on an instance there and can't escape people's spam boxes.

FWIW, I am able to send mail effectively from my Linode box, (passing SPF and DKIM).

Linode has always been very strict about keeping their IP ranges clean.

The fact that Linode's lowest tier has been $20/mo until recently ($10/mom nowadays) also helps. That makes Linode unnecessarily expensive for people who just want to burn IPs, since they can easily get an IP for $5/mo with DO or even less with low-end VPS providers.

Check the IP against the spam blacklists: http://mxtoolbox.com/blacklists.aspx

You might have to spin up a bunch of servers to find one which doesn't have a blacklisted IP. Then you have to build the IP's "reputation" with the various providers before you can send a significant amount of email through it.

Thanks, I just assumed their whole block would be blacklisted. I'll try some more.

Not blacklisted, just treated with higher suspicion by default.

These days, there are only a handful of IP-based blacklists where a listing result in more than low single-digit impact to your delivery rate; moreover, among industry insiders, Gmail isn't known to use any public IP-based blacklist for deliverability.

>Gmail isn't known to use any public IP-based blacklist for deliverability

Good to know! I bet they have their own internal ranking system which is far more accurate than the public blacklists.

I'm curious to know why you think they'd be any more accurate than public blacklists. About a fifth of the spam that hits my domains originates from gmail servers, and the majority of people I know complain (when asked) about gmail's tendency to spam-bin legit mail.

I would wager that Google handles a large enough cross section of all email traffic such that their internal statistics on the trustworthiness of domains/ips would rival any other blacklist system.

I've never had issues inboxing on gmail accounts using SPF/DKIM/DMARC + Sendgrid, even when sending 125,000+ emails (legit!) per day.

Accuracy in spam filters is a difficult thing to measure, particularly if you don't have access to internal metrics on filter performance.

The primary limiting factor for most blacklists is not scale, but simply the fact that most of them have no more than a few data sources - most commonly spamtrap data. It's useful, but it's not a comprehensive enough data point to accurately evaluate mail.

Having dozens or hundreds of data points available - things like how many recipients open a message and spend time reading it, how quickly they seek out a message when initially opening their inbox, or a sending domain's pagerank - gives Google a considerable edge in assessing overall mail quality.

(Caveat: outbound filtering is often more difficult than inbound filtering - perhaps in part because there are fewer data points available when assessing outbound mail.)

Agreed on all points!

> majority of people I know complain (when asked) about gmail's tendency to spam-bin legit mail.

I observed a former boss, who was very technically competent otherwise, using "mark as spam" instead of delete.

I guess if he could underestimate that button so can a million other people.

GMail may not use public blacklists, but they definitely have IP-based reputations that feed into other blacklists.

Just a few years ago, I personally experienced this sending email between two GMail accounts, and then I noticed that my (workplace!!) IP address ended up on a public blacklist within an hour. Fortunately I noticed fast enough to get it removed before any mail delivery was affected.

I ran my personal mail server on DO for about a year or so without issue. I would definitely try a new instance--I didn't even know this was a problem.

Amazon SES has a pretty poor set of IPs due to the issue you mentioned about firing up servers to send spam.

Mandrill is top notch, just like other professional email providers such as Mailgun and Sendgrid.

Are you thinking of EC2?

SES (as with Mandrill, Sendgrid, etc.) runs on a separate block of IPs and is very strict about spam/bounce reports. Very easy to get yourself throttled or cut off entirely.

I run a server on Hetzner that sends maybe 100k messages a week, and appears to have no problems with delivery. It doesn't even have SPF or DKIM set up. And I'm pretty sure that if messages were not getting delivered, users would complain. At least that's exactly what happened the one time there was a configuration screwup [1] that made some email providers reject mail from that server.

So I don't know if that theory explains the issue either, though it could be a component.

[1] https://www.snellman.net/blog/archive/2014-12-05-how-buying-...

> I run a server on Hetzner that sends maybe 100k messages a week, and appears to have no problems with delivery.

That volume (assuming non-spammy content, of course) would over time build up a positive reputation for your IPs, outbound addresses, and content.

A new or low-volume IP on Hetzner is going to have a harder time overcoming the initial suspicion Gmail saddles it with.

That's probably more true in general than it is in this particular case. Gmail does IP reputation differently than other major ISPs, and domain/content reputation are very different beasts entirely.

Gmail's approach with a low-reputation IP is generally to throttle it aggressively; mail still gets through, just not very quickly if you're sending in bulk.

Domain reputation is not initially distrusted unless it's a brand-new domain (registered in the last 30 days or so), and generally doesn't come into play unless there's very unusually poor metrics (think sky-high spam complaints) associated with the domain. Content reputation is applied in a similar manner.

The headers I'm looking at should most likely give Gmail enough distinct identifiers to work with to isolate it from broader reputation issues at Hetzner, but the post only included partial headers, so I can't quite say that conclusively.

Always worth checking if the IP you inherit is in some DNSBL. A number of Hetzner IPs I've gotten were tainted.

It's oh so much fun trying to get them removed.

Amazon SES or Mandrill would probably be helpful. Both have generous free tiers.

But you are still sending all your outgoing email through an external 3rd party. Not ideal. Also, these days every email is a top-reply orgy of all correspondence ever sent, so your SMTP relay essentially gets a copy of your entire conversational inbox (via "QUOTE ALL THE THINGS" reply garbage) as well as your outbox.

While I don't doubt this, I often see from DIY types some really lazy smtp habits. If they don't have their reverse DNS matching the forward DNS as well as the HELO name matching that as well, then, yes, that will be a major red flag for anti-spam systems. Reverse DNS is pretty important and weighed heavily.

smtp delivery is something of a black art, but at the very least those things need to be set correctly. Its also worth mentioning that if you have 'spammy' applications that opt people in to mailings then a lot of people just get pissed off and click the 'junk' button over and over instead of unsubscribing properly. So think twice before auto-subing customers or enabling 'mail me replies' by default on forums and such.

I also learned the hard way that once you're on Cisco Ironport's spam list, then you really can't get off. There's no one to talk to. Its supposed to be automated. In practice, that means months before your domain is unblocked. I find most Ironport admins just use that sole list even while Cisco recommends they use many and weigh the average. After a stupid config change that opened our smtp to relaying, we got off everyone's blacklist but IronPort. IronPort is pretty much the defacto standard in several types of enterprise. My fix? Get a different IP address for the mail server.

I send a LOT of emails each month (email newsletter business - yes, legit!) and ran into an separate but topically related and amusing problem recently.

My newsletters are aimed at developers, and one issue went out and was considered by Gmail to be a 'phishing' attempt. I couldn't figure it out. Several issues later, another one was picked up the same way and I figured it out.. In both issues, one of the items was linking to domains that looked a bit like this "www.0x10abcdef.com" (this is NOT the actual domain) - basically a domain that looked like a hexadecimal number. I ran numerous tests and Gmail always considered mails with links to domains like this to be phishing attempts.

I reported this as a bug (since nothing was wrong or reported with the domains in question, it was basically Gmail's filter being in error) but no idea if it was ever resolved.

It's probably specifically penalizing anything matching /https?:\/\/0/ because you can specify an IP address using hex or octal in most common browsers.

hex: http://0xD83AC02E

octal: http://033016540056

That's my guess, at any rate.

Yes, URLs like that are commonly used in phishing emails, so that's my hunch as well. They're just not recognizing that such strings can appear in domains so I suspect it's a simple bug but probably not one they'll fix.

OT: What does the red-flower-on-the-sun doodle mean?

Argentina National Day 2015

Edit: You can usually find out what they mean by going to https://www.google.com/doodles

I plan to build a developer mailing list but am hesitant to actually do so because all the advice I read tells me to do things that I personally find morally reprehensible.

tl;dr: I would blackhole my own mailserver.

Have you any advice for me?

Im planning to use confirmation as is done by free software mailing lists but my concern is how to flog my website without offending anyone by flogging my website.

use mailchimp or sendgrid or sailthru instead of self hosting it.

If I include graphics in my mail, are the graphics hosted by an http server or included in each mail?

it is generally up to you, you can inline them (MIME/base64? in the message) or remotely serve.

it'll usually serve remote http pixels for "open tracking" either way though

That's my whole point.

I regard such tracking pixels as morally reprehensible. While I know most of my subscribers will disable remote images anyway, quite likely they would think poorly of me for serving them.

All the stuff I read about email marketing is all about all the kewel things one can do with email bugs.

consider using your own software, but relaying through mailgun then. dumb relay, other than bounce management and notifications it won't try to append garbage to your messages.

You regard finding out whether someone you have sent a marketing email to actually opened that message as 'morally reprehensible' somehow? I know that blanket surveillance and government intrusion is a bad thing, to be minimised, but I'm not sure that also makes recipient tracking for one's own marketing purposes evil. If done right, cookies, email bugs and similar technologies are benign, or even beneficial to the recipient... It's all about finding out what the customer actually wants by observing what they do, since when you ask them, they often don't really know.

Not OP, but -- Yes. Morally reprehensible because it removes choice from the user/customer.

I might tolerate your initiation of contact, but I will not tolerate your observation of my reaction, without consent.

For this reason, I will not click on links with obvious tracking parameters. I strip them out first, or come to get the information some other way.

I'm cool with some discovering that I read their eMail.

I am not cool with them discovering I read their eMail while receiving pleasure in a hoyse of ill repute.

Recipients of email containing tracking beacons are generally not aware that such things exist, did not give permission for them to be used, and generally speaking, if they were aware of their existance, would opt out.

So if you use them, you're taking advantage of peoples ignorance. Seems morally reprehensible to me...

> quite likely they would think poorly of me for serving them.

Mandrill sets open tracking to 'off' by default[1] and no doubt Sendgrid, et. al make it optional as well. Same goes for link tracking.

(I use Mandrill for emailing dev-related stuff and was also sensitive about this given the demographic/privacy)

https://mandrill.zendesk.com/hc/en-us/articles/205582907-How...

> it'll usually serve remote http pixels for "open tracking" either way though

given how most clients/web-mails filter these by default, is this of any use? Only users which explicitly click on "show images" will get tracked, and the rest won't even show see your email properly.

And GMail for example will proxy the images as well, so they cant be tracked as easily.

Part of the gmail's spam filtering appears to be "crowd-sourced". People clicking on "Report Spam" or "Not Spam" on your emails in their gmail-inbox.

How about asking a few of your friends to select your emails in their gmail spam folders and click "Not Spam". Hopefully that gets the ball rolling and the situation improves...

I've had the same problem myself.

My best guesses as to why my domain has been dinged:

1) It's on a VPS, that IP may be flagged already

2) I often use a VPN, and sometimes send emails out through my server using it. This probably raises the red flags.

3) It's a non-standard TLD, .co

4) I don't use any Google services with that email address/domain (I assume doing so adds some level of measurable trust)

5) I'm not in the address book of a lot of gmail users, because this is my private server, that I use only for job seeking, and personal communication.

This has really done damage in the past. I've applied for jobs, and heard back weeks after they hired someone, letting me know, woops, I ended up in their spam folder.

Trying to meet with a friend? I can't email them, I need to use Facebook or Gmail... welp.

This is super annoying. I kind of understand why it happens, but it's just a little sad that building your own fort, so to speak, is so impossible.

Things I tried to do to mitigate this:

1) Made my web domain https only (why not?)

2) set up DKIM and SPF (didn't seem to have any effect)

3) proper SMTP authentication, secure port only

4) Reached out to Google via the typical forms they offer, and heard nothing, obviously.

That's all a bit presumptive and inflammatory for what amounts to pure speculation on the part of the author.

Google stands to lose a lot more from a potential PR disaster for burning former customers who move away from hosting than they do from trying to convert a tiny portion of users to a free mail hosting service.

My own cousin blackholed my gmail because gmail sends so mych spam. I had to ask him to configure an exception just for me.

If you dont want to receive spam use lafn.org. I dont know how one gets an account there but I expect it doesnt cost anything.

If you want a full VPS use http://prgmr.com/ - "we dont assume you are stupid.". Its a hosting service operated by neckbeards for the benefit of their fellow neckbeards.

Quite cheap, I get mine for free because I help them with their marketing.

i got mine for writing a tutorial for setting up archlinux on prgmr on their wiki. :D

I run a personal mail server there too. I'm very happy with them: I was in the free tier for years, after they acquired the hosting provider I had been using. I recently asked them to start billing and am very happy to be a paying customer.

If you run your own MDA, you could do a lot worse than to filter incoming mail against https://www.spamhaus.org/zen/; the free tier is extremely broad, and the coverage is excellent -- I get maybe one spam email a week, tops.

I'm also happy with Spamhaus Zen generally, but spam sent via Gmail is one category of email it won't stop, since it only blacklists pure spam sources, and Gmail is obviously a legitimate mailhost. I get a steady trickle of spam from Gmail addresses, mostly SEO-consultant spammers who grabbed my email from the whois info.

I dont regard SEO consultants as spammers.

Whenever they solicit me I respond immediately with links to my own SEO articles. Of course the very best SEO is for ones own website to go viral at an SEO board.

One enterprising young South Asian was obviously a clueless newbie so I gave my reply a great deal of care. A few hours later he responded with:

"You are my SEO master."

I dontbreally offer SEO consulting but to claim that I do results in lots of Google Juice.

Sort of off topic, but I saw an ad for prgrmr.com on a Safeway checkout conveyer belt divider in Mountain View, CA On Shoreline road near Google. I was wondering how effective this advertising has been.

I am going to give Luke The Smackdown.

I have devoted years to beating a clue about marketing through his pointy skull but then he complains that he has to take contract comouter janitor work to pay his data center.

I expect he shops at that dame Safeway.

The way he needs to market was established in the 1960s by the stanford alumni association but Luke refuses to Read The Fine Manual.

He knows all about Open Source though.

I would like to use prgmr.com because I appreciate the no-nonsense way they seem to do business but (especially with the now-worthless Australian Dollar) they're so much more expensive than DigitalOcean.

offer to barter something as I and kbar did. They have a really good wiki maybe you can get service in return for writing a HOWTO.

Hosted gmail is not free unless you've been with them for a while. They started charging for domains a number of years ago.

I hit a similar problems when sending automated internal emails to a Google Groups address at my company. The problem was fixed by adding the following footer: "To unsubscribe, email <my-email-address>."

Uh oh. I created an "unsubscribe" filter a while ago to get rid of spam. It's been great for keeping my inbox clean but I just don't see stuff with an unsubscribe line anymore.

It's not a good idea to manually create filters that 100% block. You can't think of all the special cases like this.

Exactly, especially based on an not-so-unique word

Doing a quick search on my inbox, there are some relevant emails containing the word 'unsubscribe'

I just filter it into a separate folder and skip the inbox but I've definitely missed some emails because of it.

Wow I have to try this trick. I have endless problems with some emails going missing (hotmail and the rest of microsoft’s domains are the worst).

Anecdotal information:

I run my own mail server and manage a number of small business clients who have mail servers. Email trust is getting more and more tedious.

Recently I was resolving a domain registrar issue with Network Solutions. They required forms filled out and signed, a copy of a utility bill from my client, a copy of my ID...

I bundled up everything and emailed them the scans. I contacted them 5 business days later, they claim to never have received it.

I sent it again while I had a rep on the phone, it went into their spam hole, probably due to size of attachment.

They helpfully suggested I get a GMail account to send the same message.

They are my registrar, they host my DNS, including my MX record. I have an spf record...

I thought it was pretty farcical, and a sad statement of digital trust/authenticity.

Some of my clients are giving up and just going with the flow, I have had several conversions to Google/Microsoft cloud-hosted solutions for email...

An spf record only helps preventing emails sent from phishing servers. It is inherently meaningless to spam filters if the trust with the domain owner itself is misplaced.

True but if they're raintrees' registrar, they should already be able to connect the domain with the person/company who registered it.

I'm very curious about that as well as I am thinking of moving my personal domains away from Gmail and it would really suck to start landing in spam simply on the basis of not being with a major mail provider.

I hope this story gets traction and someone on the Gmail team finds it and comments.

The author jumps to an unfounded conclusion which is pretty irritating and probably will make everybody who could help him not want to help him.

That said I've never seen a DSN like the one in the screenshot. It certainly is not generated by the gmail spam checking system, because gmail does not bounce spam. Gmail either rejects spam at SMTP DATA time, or delivers it to spam folders.

> The author jumps to an unfounded conclusion which is pretty irritating and probably will make everybody who could help him not want to help him.

Apologies if you took it as a conclusion, it was only hand waving speculation, but google does have an incentive to make it difficult for the little guy...

> Gmail either rejects spam at SMTP DATA time, or delivers it to spam folders.

They do if you have DMARC enabled, and a REJECT policy, although they still seem to ignore it sometimes and I have no clue my mail is sitting in a spam box.

It's entirely true that Gmail accepts mail at much higher rates than other major ISPs, and that their default handling of unwanted mail is to place it into the spam folder - but the truth is they can and will bounce mail if it's bad enough. It's just extremely uncommon for most people - even email marketers - to ever run into a situation where that happens.

> Gmail either rejects spam at SMTP DATA time, or delivers it to spam folders.

This is a million times better. Meanwhile, microsoft accepts it, sends it at passed on DMARC reports, but discards the mail silently without it even reaching the spam mailbox.

I've had mail sent via my own server rejected by gmail because of a missing Message-ID header. The 550 reject message was the standard "Unsolicited Mail detected" text, the same mail was accepted without causing any fuzz once the Message-ID was added.

Why on earth wouldn't your mail server add that for you?

Because I'm writing my own ;)

https://github.com/cmail-mta/cmail if you're interested.

I'm still wondering how does it happen that I never ever had as many problems with spam as I have with anti-spam systems and goddamn google in the first place. Actually, at this point it well might be only google. And it's hard to ignore when more and more emails in your contact book have "gmail" in them. Makes it seem that something as simple and fundamental as e-mail now belongs to google. Just ridiculous.

Seems like author hasn't done enough troubleshooting and has jumped to a conclusion. My recommendation is to run through google's troubleshooting steps on the issue:

https://support.google.com/mail/troubleshooter/2920052?hl=en

Also make sure a DMARC record is setup.

At the end of that, it asks about whether your server passes "the diagnostic tests" -- but I didn't see a link to which tests it was talking about.

It's MXToolBox[1]. To be fair to OP, it's very popular.

[1] http://mxtoolbox.com/diagnostic.aspx

I have been through this, and the end result is a contact form that no one responds to. I filled it out a few months ago and again this week.

DMARC is enabled (and in one of the screenshots, Google shows it passing). I get reports from them, but it provides no insight into their decisions.

Spam filtering/tagging is more annoying than outright bans. Microsoft banned my IP from their email services, but helpfully sent bounces so I could appeal. This is the email equivalent of hellbanning.

I know that on HN "me too" comments are not well seen, but I just have to this time, because I am really frustrated by this as well (as said before: https://news.ycombinator.com/item?id=9812157).

This part sums up as I see it as well:

I can only think this is intentional on Google’s part – they have a near monopoly; the vast majority of mail I send these days goes to Google – and if a small company is running their own mail server is too much of a hassle, then maybe they’d buy Google Apps. It’s bad, anti-competitive behavior on Google’s part. Shame on them if its true. I don’t know if it is, I can only guess, but they certainly have an incentive to make it difficult for the little guy.

I’m just a geek that likes running my own servers. My pleas to Google’s impersonal forms fall on deaf ears, and I’m getting tired of telling everyone I e-mail to check their spam folders.

I just tell people I email that their free email provider is fucking them out of their legitimate messages.

I have the same with my own server and Yahoo, after my first announcement of switching from Gmail to my own server Yahoo sent me a message that I was blocked forever. I have SPF set up but not DKIM. I also don't have a valid cert (I'd have to renew every year). Gmail never gave me any problems though.

Yahoo apparently has procedures to deal with this but they are difficult to find. The annoying thing is that now I'm not even receiving any notification of the blocking anymore. The mail just disappears.

Btw, mail from my Drupal system also always ended up in spam. But I kept removing the spam tag and now it is just goes into regular mail.

Network effect. Google blocks non-GMail email, eventually you are forced to use them as your service provider. Can either be deliberate or happy unexpected outcome for Google.

I use Google for my email, and they still count some as spam. Hell, it is sent from Google's own servers! It still doesn't help.

I've had similar problems over the last year or two with Gmail and with Outlook/Live Mail.

Our 2 person small time company sends a few hundred mails a month at most. We replay to someone on Gmail and get spam-binned. They whitelist us, we reply again and get spam binned. We send the mail via a major provider and it gets through.

On Outlook a website mail form sending emails to a hotmail.co.uk address was getting blocked - the server has the same IP it's had for years, the form has been used for years, the recipient has whitelisted the email address. I forget what eventually fixed it, think it was addition of a reply-to address; quite ridiculous.

In both cases they are long term domains with real ID info that hasn't changed, the domains have been on the same IP held by the same ISP for at least 3 years and owned by the same owner used for the same businesses for at least 10 years. Both domains are long-term registered in (Google|Microsoft) analytics.

Yes I can see that such domains could be purchased by spammers and the prior owners may not change their ID info and the new holders may be able to purchase space on the old server and so keep the IP address (despite the established ISP having strong anti-spam policies) and may be able to then send out spam emails, but who would whitelist those emails???

IMO on either Outlook or Gmail if you whitelist something, even if it were spam from a known spammer, then they should let it through (sanitised if needs be). If they wanted to they could add a "99.999996% of others blocked this but you have whitelisted it, do you want to block emails from YourBestFriendWhoSendsSpam@theirISP.com in the future?".

/rant

I've had and still have similar issue like you do. I have my own smtp server, which sends some alerts to my google apps account. I've setup my Google Apps account to white -list all email arriving from my mail server, whitelisted source domain, added spf records, etc - but my mail still being blocked. First, Google told me that's because I use IMAP and I should stop using it - only webmail, or exchange on iOS devices - fine, still blocking. Then I was told, that I can't attempt connection to the second MX server if first fails - done, still have problems. Two or three month later, they confirmed issue on their end with firewall blocking DDoS by blocking whole /24 subnet, with no fix in sight.

Remember, I am paid user and I went through rounds of bs with their support.

I feel sorry for those, who have to send email to Google Apps/Gmail as part of their business.

(I've posted this as a comment on original story, but decided to duplicate here, if someone will find this info helpful)

>What can I do except move to a hosted provider ?

To answer your question about what you can do, you can send mail to Google legal asking them to accept your mail gmail users - just copy your whole blog post, I'm sure they'll action it right away:

Antitrust is serious business. (They already have trouble in the EU for it - https://www.google.com/search?q=antitrust+eu+google - by the way I think it's completely unfounded.) It takes them seconds to whitelist you, and this really is a "15 seconds could save you $15 million on your next antitrust case" and anyone in Google legal can probably see that.

EDIT: I don't see how this got downvoted. This literally answers OP's question about what else he can do.

Antitrust is serious business, but seeing as there is no clear antitrust issue here, I'm not sure what you think getting lawyers involved will accomplish other than making it impossible to help you since your case is now an impending legal action.

I've had similar, unfathomable problems with Gmail delivery for a long time, until I added a PTR record for reverse IP lookup for the IPv6 version of my IP address. That did the trick, I haven't had delivery trouble since, even without DKIM (but with SPF).

Google just posted http://gmailblog.blogspot.com/2015/07/the-mail-you-want-not-...

"but sometimes these wanted messages are mistakenly classified as spam. When this happens, you might have to wade through your spam folder to find that one important email (yuck!). We can help senders to do better, so today we’re launching the Gmail Postmaster Tools."

https://gmail.com/postmaster/

Some years ago I ran a mail server on a dynamic IP. I basically contacted Google and had them white-list me. And at the same time, some small providers totally refused to accept mail from me because I used a dynamic IP :P

Check-list before you attempt to get white-listed/ban-lifted on Gmail: Send bulk with "precedence: bulk" header. Use spf and dkim (optional). Always send from the same domain. Have your users "opt-in" to receive mail. Have a public e-mail policy. Basically, don't require users to enter their e-mail address.

My 2c: I know it sucks when your mail doesn't get through, I've had the same issue with att.net, but gmail.com has never been a real problem for me.

I have a lot of mail that gets forwarded to gmail, including from cron that goes to my own mailbox at gmail. I sometimes have to unmark it as spam, but not too often. My IPv4 doesn't even have a custom rDNS -- only provider-specific one -- nor have I bothered to implement DKIM, although I do have SPF and am also registered for Webmaster Tools (although I somehow doubt that really matters).

I had a problem when I sent a message to about 20 associates and used BCC. Google apps for business then just marked me as spam. I would get lots of rejected email messages. The people I sent to were all people I knew who were attending an event. Just gmail being a dick about BCC. So now I send email through sendgrid from within gmail. Have you tried using an external smtp relay? It's more likely your IP address than your domain that is flagged.

I feel your pain. I have had similar problems over the last 15 years that I have been running my own mail server, but mostly with hotmail when they were the gmail of their time. (they would accept the mail and then silently drop it)

I would maybe change the SPF record to fail instead of soft-fail (~all to -all).

I have a @gmail.com account that I use for testing whenever I change something.

Please make sure that your domain or email address is not blacklisted by Spamhaus. You can check the status here: https://www.spamhaus.org/

It is the blacklist removal center. I removed my email address from this list and since then, everything seems to be working perfect.

> A few days ago, I attempted to e-mail a company regarding an online e-commerce order I had placed, from my personal address.

What? How does e-mailing a company end up with a bounce from ... Google Groups? (See attached screenshot of the bounce.)

Maybe the original outbound message had something funny in the To: or Cc: recipient lists.

The problem might be the IP address of the mail server and not the domain. Where are you hosting the server? Is it is your personal server? If so, what is the ISP and is it a residential connection or a business connection? If it is hosted by someone else, what is their reputation as a source of spam.

I never had an issue with gmail blocking my mail server, but I was getting flagged as spam by AOL every sixish month because my server wasn't sending them enough mail... The company I work for resells an antispam solution with outbound filtering, so I just relay through that now

What network are you hosted at? Perhaps your netblock is notorious?

Of course it's deliberate, it's part of Google's ongoing business model of ring fencing the internet.

People tell you not to run your own mail server as it's a nightmare to keep on top of all the security aspects, and yes that's a thing. The greater problem is getting your mail delivered to the vast majority of people with gmail/hotmail/yahooo mail.

I always wonder how companies like Mailchimp stay out of Google's blacklists.

Money?

"The needs of the many outweigh the needs of the few." -- Spock.

I'd rather have a few geeky people like me not have the fun of self hosting an e-mail server, than waste millions of man hours round the world because of people dealing with spam.

If you must hack, however, use something like Mailgun. It is more hacky in a way that you can program your incoming mail the way you want, not just install a mail server with a few commands.

That...doesn't make any sense. The author doesn't appear to be spamming or doing any kind of mailing list traffic. He is an individual communicating with other individuals. That's kind of the whole point of this Internet thing we have going here. Besides, he's only being singled out by Google; no other providers seem to have a problem with his domain and Google's own headers show that everything is fine.

It's a coherent opinion which makes sense, even if we don't agree with it. The claim isn't that the author is spamming, the claim is that the availability of effective spam filtering ("needs of the many") trades off against and ultimately outweighs the author's reasons for running his own small web server for his personal email ("needs of the few").

No, the opinion is not even logically valid because there is nothing that shows Google's punishment of his server is required to reduce spam.

It's like eliminating poverty by killing all poor people. Yes, it would work, but the side effects are worse than the outcome.

Logical validity only means that the argument works given its premises, not that the premises are correct.

Soundness and validity are often confused in common parlance, if you attempt to interpret his comment in the best light then it seems well reasoned.

Your 'majority rule' analysis is flawed as the ability for individuals to access the internet without relying on [specific] mega corps or other major organisations serves the needs of the majority. The ultimate end of requiring people to satisfy a corporations demands before being allowed to communicate using the common means (email here) is anti-democratic, it gives too much power to those companies.

First, it's about triggers. You are not allowed to carry guns on an airplane. "But I didn't shoot anybody." You didn't, but the people who do, carry guns. It's a negative trigger.

Second, when a technology becomes extremely mainstream, it begins to generate federation and structure around it. It's called evolution. This is higher in services where stakes are higher, and where giving an individual excess power can screw up other individual's lives. It is the same reason you cannot install your own cell tower. You can start a tiny in-house telephony service, but you cannot complain if AT&T declines to integrate with it.

Third, the openness is a wild, wild west concept. If openness allows him to install mail server, it also allows Google to block it.

Fourth, you underestimate the amount of turtles the spam cave has. A lot of very smart people, leave mail servers on for years sending only a few emails before using it as a spam weapon. These servers can be rented for high price on the black market. If you were a measly C++ program, it is impossible for you to distinguish his good motives from a bad person. From what Google knows from 99% cases, this server will start sending spam any moment now.

Fifth, you underestimate the effect spam has. This guy suddenly starts sending out phishy Apple looking emails, asking people to change password. Next thing you know, Jennifer's pics (or your wife's, or your daughter's) are online.

We, as hackers, drink way too much Kool-Aid.

TL;DR: The Internet is over.

>You are not allowed to carry guns on an airplane. //

I don't think that's a good analogy. It misses the crucial point that the emails are being whitelisted by the receiver but still get blocked. A better analogy IMO would be that you invite me in to your living room for a chat, I say I'll bring the gun you wanted to see, you say that's fine. Then Google come along and muscle me away at your door because I have a gun. Then another day I ring and say can I come over for a chat, I say I have the gun you wanted to see, you say that's fine and that you've let Google know you want me to come over. Then Google meet me at the door and say I can't come in because I have a gun - I don't even get to knock, they don't ask you if you want to see me despite me having what you requested, they pay no attention to your request to them to let me in.

In case it's not clear what is happening outside the analogy is that I'm sending a message with content (the gun) that Google thinks is harmful. You want that content (to see the gun). You tell Google you want that content (whitelisting my email address, marking my email as not-spam) but they continue to block the email. What's worst is they block the email now without notifying and ignore the whitelist (they meet me at the door). You never get my emails that you want (never get to see the gun), because Google have made an erroneous assumption based on generalities and ignored the interventions of their customer.

Meanwhile - inside the analogy - a third party can bring a gun to your friends house but they happen to hire Google doorstaff themselves ...

It has the complexion of a protection racket: "Wouldn't it be a shame if you used some other supplier and your emails didn't get through, oh no. /s". Then as soon as the "you paid Google" flag is raised [eg by giving them access to all your emails] suddenly the exact same messages get through to the exact same people.

Google may be completely innocent but it stinks real bad.

Yeah, but we are spending millions of man hours dealing with spam so I don't see how Google messing with this guy is helping (assuming his speculation is correct).

Google puts resources into reducing spam and does reduce spam, so it would be even worse than it is if Google didn't do anything. The author's troubles may be side effects of some of the things Google does at scale to filter out spam.

Reply on news.ycombinator.com