Cloudflare Attack Post Mortem; Apparent Google Apps/Gmail Vulnerability | Hacker News Reader

Cloudflare Attack Post Mortem; Apparent Google Apps/Gmail Vulnerability

blog.cloudflare.com

118 pointsdemonfly7763 comments

The headline of this submission is misleading.

tl;dr; Hacker used social engineering to add a secondary recovery email address to the victims Google account.

2-factor wasn't even enabled on the victims personal account but the hacker didn't need a 2-factor code to reset the password on the victims company account.

From the article:

Google reports that they discovered a "subtle flaw affecting not 2-step verification itself, but the account recovery flow for some accounts. We've now blocked that attack vector to prevent further abuse."

The title of the article doesn't even mention 2-factor: Post Mortem: Today's Attack; Apparent Google Apps/Gmail Vulnerability; and How to Protect Yourself

Don't commingle business and personal accounts.

It's not misleading. It should not be possible to recover your way past 2-factor auth, which was enabled for the company account.

Thank you. I was skeptical about that headline but didn't feel like reading that wall of text just to make sure its BS.

Yes, it qualifies as social engineering rather than cracking. The original title is more appropriate.

How is that social engineering? It doesn't seem like human behavior played any role in this at all. The hacker interacted only with automated systems.

This is a security bug that was exploited. It should not be possible to reset a password without the 2-factor auth code, period. If that is possible, then that the 2 factor auth is broken.

When people say 2-factor auth code is broken, it implies there was an issue with the 2-factor authentication code itself. That would mean things like RSA and other Token based authentication might be at risk. So no 2-factor auth is not broken. Authentication in Google is broken.

-disclaimer: I am the founder of Authy.com.

Well of course the concept of 2-factor auth is still sound. No inherent flaws in the math or the theory. But this particular implementation is quite broken.

"We have senior contacts at Google who we worked with in order to regain control of the Google Apps accounts"

Boy, its nice they get that privilege. I love gmail, but god help me the day I lose access to my account- I can only imagine the support whack-a-mole us mortals would have to go through to get any resolution.

Anyone remember the "thomas monopoly" story from a while back where gmail disabled the guy for weeks with no comment and the dude had to contact every person in the world to get someone to look at it? (And then they reinstated his account)

My first though too.

They got great support, because they have a name. Any normal Gmail user would receive five automated reply emails from Goog, at most.

I'd love to move my email somewhere else, but there is not really any alternative.

Here's a list of alternatives, and discussion.

http://www.emaildiscussions.com/showthread.php?t=64118

Google Apps accounts are billed and offer phone support, right? I don't think there's a mechanism to upgrade/migrate free Gmail accounts to Google Apps, though.

Cloudflare isn't special. Any business (or individual) who pays for google apps gets 24/7 email and phone support. I hate this perception that google has terrible customer support. If you want support, stop being a free user.

Google having 24/7 email and phone support for google apps doesn't seem to jive with this statement in the parent:

"We have senior contacts at Google who we worked with"

Saying "senior contacts" seems to indicate that they have specific high level people that they personally know who were able to escalate beyond what the response would be for a typical user.

I friend of mine got his Gmail deactivated (as you know, without any explanation or means to dispute it). With it, hundreds of purchased Android apps got lost as well. As there isn't much you can do, after a week, he created a new account and had to repurchase most of the Android apps and switch all his accounts to it. Some services require access to your old email, so, he had to recreate some of those as well.

A potential security hole is very different from someone being locked out of their account (as annoying as that is). Most web companies by now have sound procedures in place to quickly escalate those reports.

And yes, realistically, a larger service like CloudFlare is going to get more attention than any individual. It might seem unfair to the individual, but overall, it's in everyone's interest when a situation like this arises.

There are a lot of good reasons not to use gmail or google hosted apps (performance, especially, but apparently also security). Two factor auth as google implements it goes partway, but email really is the skeleton key to all accounts. I'm not sure how meaningful Google Authenticator is as two factor auth when you're actually using the iPhone or iPad to access mail much of the time. Google Authenticator protects against untargeted mass attacks, but if you are specifically a target, it's a pretty mediocre two factor solution. I think it's fair to consider administrators of any major SaaS app to be specific targets.

Unfortunately, there really isn't a great alternative. The next best thing I've found is to just run your own mail server, which is pretty obnoxious if you have any non-technical users to support. Kerio is the thing which most Mac shops I know use (since people want directory services and the other stuff you can't easily implement just from open source). Yet, there's better search on server on gmail than from iOS to IMAP (i.e. Kerio), so that's a sacrifice.

I kind of hate Google for being the Craigslist of email -- a crappy product at $50/yr which sets the maximum price people are willing to pay below the point where anyone can offer a less crappy product. More like $250-$500/yr/mailbox would be a supportable price for real email.

Please define "real email" and what justifies it to cost $250-$500/yr/mailbox in your opinion.

If you're questioning the grandparent's notion that Gmail is "crappy" compared to expensive email products, I'd agree with you.

For me, the answer to what justifies a cost for email is deliverability. Of course, having recipients receive your email promptly, and receiving theirs promptly. But more importantly, receiving next to zero spam in my actual inbox, and misclassifying absolutely zero ham in my spam folder.

The only solution I've found that accomplishes this at any price is GMail / Google Apps.

Not only does it easily keep 12 - 18 year old email addresses that have been on usenet spam free, it happily handled over 3000 spam per second per box while letting through valid emails for sex.com. The owners could actually use their addresses on Google Apps. They hadn't been able to on email systems costing 4 and even 5 figures. Refreshing the spam label while seeing the inbox stay empty, day after day and month after month, was astonishing.

Gmail does better than most alternatives for antispam. It's pretty good for deliverability (although it's actually kind of bad for receiving mail from semi-sketchy places; it doesn't even go into my spam box).

It's a slow UI (I often see 20-30 second page loading times on a fast network, over SSL), the support on the free tier is horrible (i.e. non-existent), the $50/user/yr support is below what I'd want for a company to rely on it (it's ok as second or third tier support for your in-house people, I guess, but not great frontline support for non-technical users when traveling).

The real killer, though, is lack of the enhanced features Exchange offers for directory, scheduling, etc. I always hated people who depended on those kind of things, but once I started using them, it made life annoying to not have them. Google Calendar and Google Contacts are feature-incomplete and often buggy (syncing to devices, other than maybe Android phones), especially when you have multiple calendars, sharing, etc.

It's all fine for individual users, but for a group of 5-25+ people (startup, pe fund, etc.), using google apps often sucks.

Look at Exchange and Zimbra

Exchange is a pain to run, but is quite featureful. Zimbra is actually on my list; Kerio for OSX for self-hosted. Nothing is perfect.

The biggest weakness's to 2 factor auth are:

1: You can remove 2 factor auth without having to enter the auth code. So if your computer gets a virus, someone can just remove 2 factor auth via your browser

2: The "app specific passwords" are full on backdoors. If one of your app specific passwords gets hijacked, someone can login to your gmail with it and then remove the 2 factor auth pass (See above)

* http://productforums.google.com/forum/#!category-topic/mobil...

Aren't app specific passwords simply single-factor auth tokens, thus defeating the purpose of using two factor auth? Wouldn't a sufficiently complex pass-phrase be just as secure?

No, because each app has a different token and you can revoke them individually.

Google does want people to use OAuth instead of app-specific passwords as much as possible, but sometimes that's not possible or just isn't done. Then you at least get some security from not posting the same password to a million places.

Google's own products don't even conform to this. My Galaxy Nexus and Chrome browser requires app-specific passwords.

The latest version of Chrome dev (21.0.1155.2 dev-m) does not require an app-specific password - it works with the OTP if you have it enabled, but at least for me the sync is broken at this point.

If 2-factor is used simply to log in, then any attacker who targets your phone/tablet to get your totp secrets from google authenticator could compromise rsa token protected logins as well. They'd only have to wait until you log in legitimately and compromise your account at that point.

Real security against compromise of the device you're using to access a service, whether the 2nd factor is a hardware token or generated from a seed stored on an android/ios device, would require making every significant action, not just logins, re-prompt for a token. I haven't seen many systems that do that, because it harms usability.

I'm not disputing that there are cases where RSA tokens or other hardware solutions make sense, but the physical burden of more than a few hardware tokens would be too much for normal people, even if cost were no object. 2-factor totp/hotp oath auth is far better than 1-factor auth, and presents minimal annoyance to people who already have their smartphone with them 24/7. It can be deployed on every saas website with minimal additional burden to users.

A compromised android/ios device that you use to login to services is, as outlined above, pretty close to game over even if you have a separate hardware token. An attacker may not be able to maintain access, but the account information and various settings can be compromised in an instant. 2-factor of any sort pretty much solves the password reuse problem, and also the problem with individual service passwords getting compromised somehow in a way other than an ongoing compromise of the user's machine(s).

If I understand it correctly, the attacker managed to reset the password by adding a secondary, fraudulent email account as a recovery option, and Google just logged them in after the password was reset, not requiring a token from the 2-factor system.

That's quite the attack.

Just an information - the unnamed customer was 4chan.org, and, according to the group ... I think twitter, it was because they don't like My Little Pony fans, that have a board over there.

Stupid internet drama indeed.

This is really important news. It was mostly social engineering hacking, but it seems to have been very clever. It's important to notice that it wasn't a flaw in two factor authentication itself, but on the recovery options.

I really hate the recovery options. I don't know what they are there. If I forgot my password, my recovery e-mail, I don't want to answer questions to be sure it was me. Who knows how this exactly worked, but it sure seems to be a problem with that logic.

So it seems that there was a bug in the two factor authentication affecting "some accounts" during the recovery procedure (see the update to the article). Slightly worrying that the two factor authentication can be bypassed when resetting a password.

But even after resetting the password, two factor authentication is needed to log in right? How was that bypassed?

The blog author didn't actually have two factor authentication enabled. The headline is wrong.

You are incorrect, quote from the article:

all CloudFlare.com accounts use two-factor authentication. We are still working with Google to understand how the hacker was able to reset the password without providing a valid two-factor authentication token.

He didn't have it for his personal account. But he did have it for cloudfare.com accounts.

Seems to me that there were two flaws evident here. One was Google's reset flow, which they have since (hopefully) corrected.

The other issue, that seemed to be glossed over, was that they were BCCing password reset emails to a compromisable account for support/debugging purposes. I understand the desire to be aware of fradulent submissions early, but we, as an industry, should be treating password reset emails (and tokens) like passwords. Do not store them in plain text.

I appreciate CloudFlare's transparency on this, but feel they missed an opportunity to address a flaw that probably exists in a large number of other apps.

Copying the emails stood out to me as a ridiculously stupid thing to do. It was probably a quick and dirty solution to save coding up a way to monitor resets on the backend.

They say that they have disabled that now, so that's something.

The most interesting part to me is that somebody managed to get phone support for GMail.

Google Apps for Business (a paid solution) offers phone support: http://www.google.com/apps/intl/en/business/upgrade.html#utm...

Since they were using their own domain name, that's likely what they were on.

as stated on Softpedia, UGNazi got access to all Cloudflare customers' IPs and IDs, and they will sell these informations http://news.softpedia.com/news/CloudFlare-Reports-Breach-UGN...

So on one side Cloudflare is saying that they just compromised a single account and on the other we have the hacker claiming that all data was compromised... I guess at that point the question becomes not whom is telling the truth, but what any Cloudflare users (myself included) should be doing right about now?

Presumably after the hacker compromised the CEOs email account he had access to any account he wanted, as I interpret the blog post he was only able to use that access to compromise a single account before being locked out.

Nobody is necessarily lying.

The funny part - cloudflare still hosts DNS for UGNazi

   Domain Name: UGNAZI.COM
   Registrar: ENOM, INC.
   Whois Server: whois.enom.com
   Referral URL: http://www.enom.com
   Name Server: LEE.NS.CLOUDFLARE.COM
   Name Server: RUTH.NS.CLOUDFLARE.COM
   Status: clientTransferProhibited
   Updated Date: 29-may-2012
   Creation Date: 22-jan-2012
   Expiration Date: 22-jan-2013

Cloudflare supports hacking group by providing them services -> gets hacked by them. Reminds me very much of what happens in every episode of 24, the terrorists kill their accomplices after they're finished with them, it never works out for anyone involved.

Which "hacking group" was supported by Cloudflare?

lulzsec

Lulzsec was the most prominent, they used Cloudflare to keep their sites online, Cloudflare used it as a marketing tool, here's a blog post about it: http://blog.cloudflare.com/58611873

Cloudflare doesn't necessarily "support" hacker groups- although their CEO did a great talk at Defcon last year about keeping Lulsec up (which I consider more of an activist group than a hacker group, personally). However, they do very little to keep their network safe- they have no dedicated abuse team (or, if they do, they're impossible to contact directly), are slow to response to abuse requests (if they respond at all), and even when they do respond take only the most basic steps to filter the issues (which takes literally minutes for their account holders to work around, and they don't actually disable the accounts).

Dozens, if not hundreds, of hacker and malware related sites use Cloudflare to mask their origin and essentially use the other shared sites as hostages. If you block one of the malicious sites then you also end up blocking legitimate sites, and since Cloudflare goes out of their way to not take down sites (even those hosting malicious programs- including the drive by exploits that install them) it makes it very difficult to trace the malware back to it's source or actually take it down and stop people from getting infected.

I'm hoping that with the coming of IPv6 many of these issues are going to go away. At that point Cloudflare could give every customer their own /128 that is static across the world (they route their /64 or /48 however they please using anycast) and forward traffic as they have always done with caching.

Then it becomes trivial to block a single IP, since it is assigned to a single customer you won't have any collateral damage!

CloudFlare does in fact have a dedicated abuse team. It's extremely easy to contact us actually using our reporting page: https://cloudflare.com/abuse

If you do happen to have a report then please feel free to actually send it our way with specific details.

In fact, we respond to every single report that we receive.

Lulzsec and the people that did this are not necessarily the same.

I find it a bit strange to lump a hosting provider together with terrorists accomplices.

News report about CloudFare and Lulzsec: http://www.it-networks.org/2012/03/01/how-cloudflare-kept-lu...

I know the groups are different, my observation was that Cloudflare supports these groups by providing them with services and in return they themselves become a target of such groups. My terrorist comment wasn't comparing them to terrorists, I guess a better point would have been "no honor among thieves". Cloudflare support these sort of groups and in return they are targeted.

Why would they not be targeted if they didn't support them? I see no correlation, the point seems very strained.

There was no correlation, I was just making an observation, maybe it wasn't appropriate for HN.

Actually i think your right, your "observation" wasn't very appropriate.

Because your saying that you "observe" that a company who supported a certain group of hackers, got hacked back by someone we don't even know (maybe UGNazi, but who knows for sure). And then you look at it as if it was "funny" saying something like "see, no honor between thieves".

But... there's no point. Yeah because they got hacked like any other company could have been to get access to some of their clients data. And it's not like if what they did for LulzSec was to protect themselves of those sort of attacks. It's totally unrealistic to think that just by "helping" a group of hacker they would be totally immune against attacks from other groups. Imagine if Google also had helped a pair of hackers guys, would you except that the others hackers, decide, for honor, to not attack Gmail anymore ?

this is the tweet where CosmoTheGod states that UGNazi hacked Cloudflare and 4chan https://twitter.com/CosmoTheGod/status/208656791708508160

WHERE IS YOUR GOD NOW, JEFF? [1]

[1]: http://www.codinghorror.com/blog/2012/04/make-your-email-hac...

If only that admin had followed this suggestion for his personal account...

Have you read the article ? the 2 factor authentication wasn't compromised, it was mostly a social engineering attack. Even if it was the 2 factor authentication that was compromised, how would this be Jeff's fault ?

There was a flaw in the system that Google uses to allow the transfer of control from an account. Two-factor authentication wasn't compromised itself, but the attacker was able to bypass it to access it. That flaw, they tell us, has since been patched on their end.

Can you explain this? After the attacker reset the password of cloudfare.com email address what exactly did they do? How exactly did they login?

Reply on news.ycombinator.com