Lessons learned while protecting Gmail | Hacker News Reader

Lessons learned while protecting Gmail

googleresearch.blogspot.com

143 pointshektik65 comments

I am a victim of these "lessons."

I run several email newsletters. One of them consistently gets marked as a "phishing" attempt by Gmail. They do not get marked as phishing if the email includes no links – but if I include a link, it immediately gets marked as phishing. This disables all of the links in the emails. I regularly get emails from subscribers complaining that the links do not work.

I once sent an email to all of the Gmail subscribers, asking them to explicitly mark my emails as trusted. Many of them did. But, it did not help – they still get marked as phishing attempts. I've implemented SPF and DKIM. It still didn't help.

The problem: Once an email is flagged as "phishing", the links are not clickable. On mobile, there is no way to make them clickable. Most users don't know how to make them clickable, even when it is possible.

All of the emails come from the same domain and the links are always to the same domain. All of them have the same basic format. Other email from this IP address gets through no problem. Google's Postmaster tools give the IP address a 100% reputation. I tried switching to a new sending service, to no avail. Nothing seems to work – no matter what I do, Gmail marks my emails as phishing.

I've contacted Google's postmaster, to no avail.

Basically, Gmail has made it very difficult for people to read the content I send them. There seems to be no way to convince Gmail that my emails are not phishing.

The phishing for URLs in mails has little to do with the reputation of the sender, and everything to do with phishing reports for the links in question. It's quite likely that whatever site you are trying to link to is serving malware unbeknownst to you.

The GP should try this site on the URL in question: https://www.google.com/transparencyreport/safebrowsing/diagn...

> Current status: > Not dangerous

> Safe Browsing has not recently seen malicious content on

there are plenty of examples and at this point it seems that Google has a vested interest in making life difficult for independent or self hosted email providers

This has been my experience. I run a very small email setup for a few friends and a couple of tiny businesses. (Yes, we should just pack it in and move to Fastmail but I am hard-headed and have been around the Internet long enough to still be idealistic about these things.)

Every couple of months, Gmail starts depositing any email from my users into spam. No warning, no rejection, just a vague yellow banner on the recipient's view that says "this message has the characteristics of spam." The message could be in plain text with no links, still the same. I have SPF, DKIM, and rDNS all configured. I send from a single IPv4 and single IPv6 address with matching records. No RBL entries. Even the headers on a "spam" message say that everything passes and is fine.

Oh, and it isn't a domain reputation problem: only one of the domains I host even has a web site and all of them are at least five years old. Two of my domains predate Google itself. :P

The real kicker? I can't use their postmaster site, either. Why? We don't generate enough email to rank a report!

Meanwhile, no problems at all from any other receiving hosts. I have to log into several Gmail accounts and click "this is not spam" on some test messages and then it is fine for another 50 days. After that, back to the bit bucket.

Grr.

I don't think it is a problem with independent or self hosted email.

I have this problem sending through:

* An email service such as Mailchimp

* My own server with SPF / DKIM and good reputation / deliverability for other domains

* Amazon AWS Simple Email Service

I think it is something on the domain level – perhaps they've flagged the domain.

If you've implemented DKIM and SPF I'm surprised this is happening. Of course there are other factors but I'd be interested to see what the email looks like. Can you send an example to sean@catch.seanzieapples.com, please?

To be honest, I would rather some people have trouble emailing me than basically any spam getting through to my inbox.

My concerns are the opposite. I'd rather see a few Nigerian scams get through than completely miss an old friend's email.

Moved away from Gmail, to a big part because of that. Almost did not get my current job because the spamfilter blocked the mails of my future boss to me, after some month of email ping-pong to get all the documents sorted out. I thought they were not interested anymore. They thought I were not interested anymore, since I stopped replying.

so you 'ping-ponged' for some time and suddenly emails from your future boss became spam?

If the answer is yes, is it viable to sue Gmail of lost business/job?

p.s. if talking about the topic, I have also seen quite drastic measures on emails which are not Google's and pray to god if you CC bunch of GMail addresses. You can be flagged as spam, phishing or whatever pretty quick.

> so you 'ping-ponged' for some time and suddenly emails from your future boss became spam?

Exactly.

> If the answer is yes, is it viable to sue Gmail of lost business/job?

In my case definitely not, it is the job I currently have :) Everything worked out after I had an epiphany and checked the spam folder.

They probably have some licence boilerplate about how gmail comes with no warranty or fitness for any purpose.

You do have a point.

It sounds like the lion's share of their effort is going towards blocking attacks that prey on users' lack of knowledge. Infected email attachments and phishing are both examples of this that they brought up in the talk[1].

It worries me, though, that they're willing to accommodate novices to such a degree that they open up advanced users to targeted attacks. For example, Gmail bugs you quite a bit to set a "recovery phone number," but doesn't make it clear that this isn't like 2-factor auth. The phone number is actually a single factor that can be used to reset your password. It even works if you have "traditional" 2-factor enabled.

Thus, the attack looks like this:

1. Look up target's social security number. This is easy with certain online services that were meant for private investigators, but actually let anyone get an account.

2. Contact their cell phone provider. If you don't know which one, guess. There are only a handful of common providers and you'll hit on it eventually. Impersonate the target, say you're going on vacation and need your calls and texts forwarded, and give them the SSN from step 1 to verify.

3. Go to Gmail and say you forgot your password. Opt for the phone based reset, and wait for the text with a reset token to come in. Ideally, do this while the target is asleep to give yourself time to work.

High profile individuals have actually been hit this way, and I think Gmail should offer greater protection to sophisticated users who do everything right, don't fall for phishing, and would never forget their password, but can fall victim to highly targeted attacks.

[1] https://www.youtube.com/watch?v=nkV9kOsTyJU

Would a cell phone provider forward to any number on a verbal request over the phone? That seems pretty sketchy to me.

Customer service turns out to be the weakest link for many services[1][2]. I would not be surprised at all if this were the case.

[1]: http://krebsonsecurity.com/2015/12/2016-reality-lazy-authent...

[2]: https://medium.com/@espringe/amazon-s-customer-service-backd...

Check this out:

https://youtu.be/bjYhmX_OUQQ?t=2m30s

A phone provider's key goal is to continue to provide charged-for services to their customers, not to secure your bank account or dns registrar account... They'll do whatever's needed to allow paying customers to pay or pay more.

It makes sense : their business it not security. Companies use them for that, and never asked for permission, they don't have to honor it.

Alhough as a customer, I'd like my phones not to be redirect to anybody asking for it, but I understand they don't have the security measures a bank should have.

Social security number is step 1? Then you likely have bigger problems then your gmail. If your in a position that this actually is a big problem, get a private email server and gpg.

You can buy lists of Social Security numbers online. You give it to dozens of people and companies in your life, and thousands more have access to it afterward.

And with it you can steal an identity.

Because countless companies are stupid and treat it as authentication.

It's a password that you reuse everywhere and never change.

By the way, you _can_ change your SSN. Nobody ever bothers to do so, though.

Only under certain circumstances.

"The SSA may assign a new Social Security numberto you if you are being harassed, abused, or are in grave danger when using the original number, or if you can prove that someone has stolen your number and is using it."

You're right that many things are insecure. Of all the services people use on a daily basis, Gmail is probably among the best-defended. It still has security problems, though, and I want it to get better and not use the lack of quality competitors as an excuse to stop improving.

Unfortunately, maintaining and securing a private email server can be a big job. Google already does that job, and their data centers are much more secure than the VPS provider where you'll host your private email server.

All I'm asking for -- and I don't think this is unreasonable -- is to rely on their world-class technical security, while being able to disable all password recovery methods, which are vulnerable to social engineering.

Wouldn't your payment to such a service be tracable, if it was used for a nefarious purpose?

Yes, the private investigation service you used to lookup the SSN would have your payment trail, unless you used a prepaid visa. But how does that help?

All the target knows is, his phone has been forwarded and his Gmail password changed. Even assuming the police are willing to help (which is unlikely), all they'll discover is that your phone number is forwarding to a disposable cell phone which was bought with cash.

Wouldn't the police be able to see that a PI service looked up the person's social security number?

No. The services maintain their own databases, and don't disclose their access logs.

Would this attack work outside of the US?

If your telco authenticates phone support using your government ID number, it very well may work. People in some countries do not even try to keep their ID number private. Some companies ask you to enter it in various places for their own convenience. It was the same in the US some years ago...people did not view SSNs as very sensitive personal data. Some of those people are still alive today.

SSNs are not supposed to be used as ID numbers, even. Companies still use them.

Some of these are pretty good, but mostly just intuitive stuff. Really wish it was to more than just an infographic style png giving broad strokes.

Things like "You are only strong as your weakest link," "There is no silver bullet," and "Never stop improving" are essentially meaningless platitudes. I would, however, love to see data on headings like "Attacks come in bursts."

Is there a link to something that delves more into each topic?

edit: Saw the links to the slides and video talk. Looks much more comprehensive!

"Make it hard for attackers to understand your defenses"

> "Users are your best allies. Empower users to action through meaningful feedback."

Interesting how that works only one way. If you are in need for some support from Google, then good luck trying to contact them.

If you're on a paid account, then you get to contact them. It's not the best support ever, but you do get support.

You make an angry blog post about it, or contact a Googler..

actual talk: https://www.youtube.com/watch?v=nkV9kOsTyJU

slides: http://www.slideshare.net/elie-bursztein/lessons-learned-whi...

I'd like to learn more about their DDOS, network-level, and cross-datacenter protections. They have to deal with so much volume and so many integrations that there's probably some lessons they learned there on dealing with issues. I'd also be interested in how their filesystem or data-processing tech with built-in integrity and availability mechanisms factor into it. They might leverage it in interesting ways like they do with F1 RDBMS.

GMail cost me over $3K in rent.

I'm looking for a job, and after sending out resumes (and often a URL to my resume) for over a month I realized practically no one was getting my emails. I was being flagged as a spammer.

The only way around this was to signup for a GMail account.

I won't feel very protected losing the roof over my head, GMail. Please fix this.

Why not $100k instead of $3k, for the job you did not get because of GMail?

This happens vice-versa as well - I've found replies/invitations to interviews in my spam folder several times, when I've already had accepted another job.

Sweet infographic for ants - clearly it was done by a design pro, why upload an image with such tiny fonts?

Looks fine to me at ~3 feet on my 96 DPI screens and at ~2 feet on my ~200 DPI Nexus S screen. If I put on my dramatically too-old glasses, it still looks good from substantially further away.

shrug

This blog post has tons of good stuff. For one, George Hotz's intro to his timeless debugger QIRA and his 4 mins pitch to his autonomous driving company comma.ai was pretty awesome! Well worth that 20 minute than NSA TAO Chief's PR talk for half an hour.

Probably better. I wouldn't knock the TAO chief's talk as he basically gave a lot of good advice. He could've bullshited about just stacking a few security products like I've seen with RSA conference types. Instead, it was a thorough look at many ways they compromise systems and most of what needs to be considered. It was one of few positives I give to NSA as anyone listening might improve their stuff.

One of the comments points out that the URL shortener link at the bottom of the infographic is broken - the actual link is mixed-case, but the infographic is in trendy all caps. A lesson here for both devs and designers - don't make case-dependent urls, and don't force all caps on content that is case-sensitive...

> ...don't make case-dependent urls...

Ehh. If you're running a URL shortener, case-sensitive URLs let your URLs stay short longer. Ditto for things like YouTube's video IDs.

Don't make case-dependent URLs that humans have to type by hand.

Soft disagree. There's generally nothing wrong with case-sensitive URLs when they're rendered in a font that makes commonly confused characters easily distinguishable. The only problem I can see is when you have to speak such URLs.

eh cap bee cap cee dee eee eff gee is aBCdefg

Sure. It's not an insurmountable problem. ;)

Youtube and Imgur rely on case sensitive URLs

But you're right, you have do click on them, any transfer requiring typing is susceptible to losing case.

The icon for "Encrypt Everything" says "In transit & at rest" with a picture of the Gmail icon.

Does this mean they encrypt customer e-mails at rest on their server?

I would think it common-sense to do that. If nothing else, it makes hard-drive disposal much simpler, and accessing disk storage is slow enough that I can't imagine the overhead of (hardware-assisted) encryption is a problem. Most places I worked at require all desktops and laptops to use storage encryption too. Encryption at rest is very different from end-to-end encryption, though.

The overhead of at-rest encryption is negligible.

If you think about it, the files Gmail backends need to access are fairly small: even the upper limit is just couple of tens of megabytes. This is true regardless of whether you get to use hardware acceleration or not. (And in case of modern Xeon servers, you certainly do!)

Servers will spend more time waiting for disk seeks to complete than they do decrypting the data once it's read.

You can even test it yourself: just run "openssl speed aes". My puny laptop does 85MB/s at the most unoptimal settings (AES-256 with 16B blocks), and 92MB/s with conservative settings at same security level (AES-256 with 1kB blocks).

A decent server system can do multiples of that. And once you add hardware acceleration, we're talking about crypto throughput of several hundreds of MB/s. Google servers are connected to top-of-rack switches, and I can make an educated guess that the per-server bandwidth is 1Gb, or roughly ~120MB/s.

For hilarious comparison, even my RPi 2 can do 16MB/s.

So: if we're talking about on-disk storage, crypto will never be your performance bottleneck.

protip: run `openssl speed -evp AES256` and be amazed by the performance of AESNI!

>only as strong as weakest link

>diverse team

Adding diversity makes the weakest link weaker (and the strongest link stronger). The point of diversity is to increase variance in multiple areas so that a team's "strongest" member in any area is strong. Does not make sense as a solution to weak links.

I think the "only as strong as weakest link" was referring to the technological measures, not to the team. It's not the team that's under attack (normally - if it's a phishing attack against the Gmail team as a way of attacking Gmail, then yes, diversity on the team may be a weakness).

If each member is responsible for a different part of security, then you'd still want each to be as strong as possible.

It just occurred to me that diverse could mean "skilled in different forms of security", and then it's just saying to hire domain experts in as many security domains as you can. That would make diversity a direct solution to the weakest link problem.

Expected an article about new kinds of attacks or unexpected attacks, instead got an infographic with a couple of common proverbs ('only as secure as the weakest link!', 'layered defenses!' aka defense in depth, etc.)

Has anyone found a link to a higher res version of that infographic? The orange bubble text is infuriatingly unreadable.

Anyone have any idea on the Gmail stack? I think it's GWT for the front end?

Reply on news.ycombinator.com