Ham or Spam? Gmail Not to Be Trusted for Important Mail
jacquesmattheij.com
jacquesmattheij.com
From what I've seen while debugging such issues the problems were always from misconfigured outgoing servers and their dns, so I would be very interested in knowing why this has happened.
ps. You might consider enabling comments on your blog so there is a place for such discussions.
I guess this is Google's way of telling me that I should switch to Gmail if I expect to see my mail being delivered.
She uses gmail and his messages were dropping into her junk folder. She almost exclusively uses her phone for her gmail access, so she never saw the messages.
Since gmail doesn't give anything like SpamAssassin headers for messages, there's nothing I can do to troubleshoot this aside from trial-and-error.
Gmail's support for this is not very good - for example, if someone with a SPF record e-mails an address that forwards to gmail it gets marked as an SPF failure because the forwarding server isn't an approved sender for the sender's domain. Unless your forwarding server supports SRS, which is very rare.
To put it into perspective, don't complain about the car, the road, weather, or the cop when you get ticketed for hitting a parked car in the rain while speeding.
Email is highly complex, and GMail are one of the few that are doing it right! (and per RFC's instead of making their own standards like some other unnamed companies do :)
Random checks are random and do not belong in a system like this.
Yes, email is highly complex, I'll grant you that but if you start adding a random element to it then it gets more complex without any apparent reason.
I'd be perfectly ok with all this if I had made changes on my end but I fail to see how not having made changes on my end this suddenly started happening and I'm supposed to be the one at fault.
Random checks do belong in a large scale mail system such as GMail. Their system only checks 1 in x emails for DKIM/SPF.
Could you imagine running DKIM/SPF checks on EVERY single email coming in? That would take considerable compute power, not to mention increasing memory requirements in a system built to minimize such things so that it can be scaled properly.
Your problem is simple. Make your own MTA compliant to any applicable RFC's and GMail will just work. Skipping important items like SPF and DKIM will ALWAYS cause intermittent issues with various other MTA's.
>> I'd be perfectly ok with all this if I had made changes on my end but I fail to see how not having made changes on my end this suddenly started happening and I'm supposed to be the one at fault.
You're making the assumption that every delivery is going to the same server, running the same code, which tests the same checks... every time. - It doesn't, and won't. Outcomes can change in a system designed to flex and deal with current problems.
I'd imagine google has by now figured out how to make a cache work.
> Could you imagine running DKIM/SPF checks on EVERY single email coming in?
No, but I can imagine keeping a set of cached data per origin address.
> You're making the assumption that every delivery is going to the same server, running the same code, which tests the same checks... every time. - It doesn't, and won't.
Indeed. Unreliable by design.
I believe it is just a well meaning extension of Gmail's spam filter but it has real anti-competitive results. I had to take the pragmatic course of action but I now view email as inherently unreliable.
If any Google employees are here, consider adding the rule "if email address in (list of emails addresses sent to) then NOT SPAM". It would make a lot of people's lives easier.
Additionally I'd like to still receive emails from friends even if they mention vi@gr@ too much.
This blog post is extremely worthless. The 'root cause' he mentions at the end isn't discussed at all.
SPF records are REQUIRED to get past Gmail's spam filtering.
Add the SPF record, damn.
A properly configured email server would have both SPF and DKIM signing.
Google's spam filter is a complicated black box. You're not going to get far understanding it by prodding it. But you're certainly not going to pass it by ignoring SPF.
I don't know what your history is, but there's piles of bugs that I've had to fix by just trying random things that seem vaguely related to see what happens. If I refused to try anything until I was 100% sure of the root cause, then lots of bugs I've been assigned would never have been fixed, and many more would have taken 10x longer.
If you are still getting some emails through, they must have other characteristics that make them look less like spam than the others that are getting filtered.
Just add the SPF.
This issue surely does not affect me alone hence the warning, the sender may have no idea their messages get dropped at all.
SPF is a DNS record on that domain name. You control that record, not your email provider.
Set that and watch this problem disappear.
I do occasionally check my spam folder and there's always a few false positives - but usually nothing critical (although admittedly I might have missed some that were and never known it)
So - day to day - I currently regard Gmail as 'good enough' but some of that is down to blind trust with little verification.
Can anyone else share their experiences - positive as well as negative?
Could be that other GMail users don't know how to unsubscribe from GitHub/other notifications and mark these as spam to get rid of them which introduces false positives into the spam notification system.
I'm increasingly finding the following scenarios to occur when using Gmail:
- Letting email I've previously marked as SPAM (same subject, same sender) through the SPAM-filter as valid email. Again. This week.
- Marking non-SPAM email from known contacts I've emailed to and received emails from in the past marked as SPAM for completely non-obvious reasons.
So you DO get spam, and you DO get false positives, meaning you also need to check your SPAM folder for important email.
Basically the one feature which once made Gmail great is now utterly broken, and since for most parts of the internet people now equate Gmail with Email, Google has now officially broken email which was supposed to be decentralized and thus not vulnerable to stuff like this.
Great stuff there, big G. Great stuff. From the rest of the internet: Thanks a million.
Slightly histrionic, don't you think?
This was the the status quo a few years back and still is for many mail providers.
Though certainly 'I've previously had an email conversation with this person' should override any spam detecting heuristics.
On one occasion the problem was so sever that we switched to using (the irony) a google docs document to continue the conversation.
Like Jacques I've run my own mail server for decades, from sendmail, to qmail, to postfix there hasn't been all that much to worry about other than dealing with folks who would try to break into your system through bugs, or exploit your server to send spam.
But Google is such a tempting target for spammers, probably carrying a notable fraction of all the worlds email. Once you get "inside" you can deliver your spam to millions, if not billions if you're careful. So I understand their problem.
That said, it would seem possible to set up a system for "whitelisting" yourself with Google, something like Clear for travellers, an API key perhaps. But that would actually increase the cost of running gmail and that would cause it to lose even more money and well that is something Google can ill afford these days. And I realize that Google is unable to post strict rules "if you do this we will accept your email" because, well spammers would do all those things.
We are but collateral damage of the battle amongst those who would exploit the network for gain.
One one of the sysadmin professional organization mailing lists, email from one of the people speaking for the organization started to be marked as spam because the person had DMARC instructions on their domain, but messages remailed from the mailing list server weren't authorized. In this case, Google's mail servers were acting as instructed. I sent the person copies of the headers showing why Google's mail servers were taking this action, as, apparently, did a number of other people. The person became irritated and decided that people who use Google's mail servers needed to fix their own problems.
If your DMARC configuration is causing your messages to be marked as spam, you really should look into that.
If you appear to speak for a sysadmin professional organization, and your DMARC config causes your email to be marked as spam, you really need to fix that, and consider that blaming the recipient mail servers does not do good things for your credibility.
Of course, there are plenty of things that can go wrong other than DMARC, and some of those might well be Google's fault. You can typically track down the reasoning for a Google mail server spam decision from the headers of the delivered message.
Like others, I'm interested in what root cause you've determined, if you can share.
Google is pretty efficient at eliminating phishing scams. But sure enough I found an email from my actual bank, offering me a credit card, tagged as phishing. Then again, when I looked at the sender domain I was half convinced this was as accurate as its categorisation of all the other phishing attempts
(Dear Santander. If I have to Google the yoursantander.co.uk domain because I'm not confident it's actually yours, you probably shouldn't be using it to try to get me to sign up for credit cards. If the top result is a three year old open letter written by someone else complaining about the same negligent approach to internet security...)
Similarly, there's an Amazon deal email in there, but there's also an explanation on the top that it's "failed amazon.co.uk's required tests for authentication", which might well not be Google's screwup
I was less impressed when Gmail filtered an innocuous email from my dad...
Two reasons.
1) Email is not built with security in mind. Don't suggest SSL as it only protects (somewhat) while in transit and not at rest. You can also bet that not every recipient you send to is using an SSL-enabled POP3/IMAP connection to check mail, so you can't control that end of the conversation.
2) It has no delivery assurance and wasn't built with that in mind. You can never be sure if it arrived, or will be read (by an actual person).
That said, if you need to transmit important documents, use FAX (if the destination has their own (not shared) FAX. Or simply drop it off.
If you need delivery assurance, pick up the phone and ask if they got it, or send if via courier with signature request on delivery.
Whether it is useful or not depends on the contents of your spam folder and already one person in this thread reports that for them it certainly was useful.
I'm not sitting here making stuff up and I seriously resent your suggestion that this is false.
There's zero evidence to show that, "Gmail not to be trusted with important email.", as the title suggests. None of it is factual. All of it is presented as-is with only what appears to be a very poor understanding of MTA configuration issues.
The answer has been here all along early in the thread. DKIM & SPF yet the complainant has simply shrugged it off with, "Well that would cause consistent issues." (which is also false.)
If he can not do basic troubleshooting or understand the importance of the SPF/DKIM settings in a modern MTA system, and as a basic starting point, he can not assert that Google can not be trusted with important emails.
So. False, and useless.
1. some Gmail users have told me they don't receive email from my non-gmail server/address, even though I set up SPF, reverse DNS lookup and everything else that was recommended except for DKIM. (The domain was bottoms.christianjaeger.ch.)
2. I've gotten a lot of false positives from mailing lists in my Gmail spam folder for a long time. Now the part that really pisses me off is that Gmail's learning mechanism, if present at all, does not seem to work at all. Why otherwise would mails from the same mailing lists, the same authors, with the same kind of content (e.g. "[PATCH]" in the subject, patch in the mail body) still make it to the spam folder with the noted reason "looks similar to other mails treated as spam"?
3. On top of that, every about 10 mails I mark as spam the Gmail UI asks me whether I'd like to submit it to the Gmail spam team to help improve the service. I never got any notice from them, nor does this seem to have helped.
4. I think I might be spending more time now going over my spam folder and unspam false positives there than I was going over my inbox in the past and tag the false negatives when I was hosting my own mail and used spamsassassin and grey listing.
5. It's getting worse and worse. Now all messages from my servers (crontab and other system mail) are going to spam, too[1]. So basically I now have to move to something else, and I'm ready to use my own hosting again, but I don't know what I should do to solve the "Gmail users won't receive my mail anymore" problem.
[1] e.g. From address "chris@leafpair.com" with "Return-Path: <chrisirc@a.christianjaeger.ch>", or From "bottomsmail@a.christianjaeger.ch" with that same address as Return-Path. (Also a header from Google says "Received-SPF: pass (google.com: domain of chrisirc@a.christianjaeger.ch designates 78.47.51.206 as permitted sender)" or "Received-SPF: pass (google.com: domain of bottomsmail@a.christianjaeger.ch designates 78.47.51.206 as permitted sender)" for the latter.)
Edit: add examples
To stop the trapped emails ending up in the spam folder, I had to create a custom filter: simply marking mails as 'not spam' in the spam folder does not do the job.
And the filtered mails now have a yellow banner plastered across them saying "This message was not sent to Spam because of a filter you created."
I'm going back to Mutt and my ISP's mail server.
One of the more annoying nuisances hitting me personally is from a YC startup, Zenefits. I really dislike them and their moronic hey-buddy spam and that I can't reasonably blackhole the originating network for a while.