Hacker News Reader

If Google hadn't noticed and saved them from this bug, who can say how long they might have continued spraying private data into the world's caches? Apparently, this had been going on for months prior. Heartbleed was exposed for years. This could have been too. Worse: malicious attackers could have discovered and quietly exploited it. For years.

The very last people in the universe Cloudflare should be criticizing right now are the Google security team.

The comment from eastdakota seems to be political in nature; some on here have suggested he has an axe to grind with Google.

Why do humans politicize so much? That's one thing I'll never understand, and one of the reasons why I refuse to become a manager.

Too bad for those who have to suffer under the unjust policies of others that you must surely know exist.

I summarized The Stranger a long time ago, with a remark I admit was highly paradoxical: 'In our society any man who does not weep at his mother's funeral runs the risk of being sentenced to death.' I only meant that the hero of my book is condemned because he does not play the game.

Indeed. The life I live.

Any talk that is about Google's systemic influence or role is political by definition, nothing wrong with political. People and companies have political ideologies and it's good to discuss them.

Discuss them, sure.

But what does it really matter?

Well, it's easy to ignore politics until you're personally affected by it.

'politicize' is just a fancy word for 'dispute'. humans have emotions and we still have mostly reptilian brains.

Strongly agreed - Google's Project Zero helped them immensely and without them it would have continued to grow worse.

The length of time this went for was already disastrous. From the Cloudflare blog post, "The earliest date memory could have leaked is 2016-09-22".

As an example of how destructive it might have been beyond leaked information, note that Internet Archive have spent considerable time archiving the election and its aftermath.

donaldjtrump.com is served via Cloudflare.

Chances are it wasn't a domain that was leaking information - though we don't know as there's no publicly accessible list and no way to tell if they had the buffer overrun features active.

If it was however the Internet Archive now have a horrible choice on their hands - wipe data from a domain that will be of historical interest for posterity or try to sanitize the data by removing leaked Cloudflare PII details.

History will already look back at the early digital age with despair - most content will be lost or locked in arcane digital formats. Imagine having to explain that historical context was made worse as humanity deleted it after accidentally mixing PII into random web pages on the internet -_-

Can't they archive the content (but not expose), and then figure out how to filter the exfiltrated data? No history need be lost.

To some degree that's what happens. Internet Archive and Common Crawl use WARC files and deleting an individual entry or set of entries from them can be markedly difficult. It's easiest to mark it with some manner of tombstone for later handling.

The complication comes with the edge cases they'd need to face and @eastdakota's call to "get the crawl team to prioritize the clearing of their caches"[1]. This is also work they now need to do due to Cloudflare's blunder.

For Internet Archive and Common Crawl, these aren't caches, they're historical information. You can't just blow that away - but you also can't serve it if it has PII in it. Either they need to find all the needles and filter/tombstone them - which we'd expect to be very difficult given leaked information is still sitting in Google and Bing's cache now - or wipe/prevent access to the affected domains.

Wiping the donaldjtrump.com would be historically painful but even temporarily blocking access to the domain would be problematic.

Finally, and most importantly, the fact that non-profit projects need to worry about how to exfiltrate such information is ridiculous anyway. Exfiltrating the information is non-trivial as well and may well be destructive with even a minor bug in processing. Having looked at much of the web, it can be hard to tell what's rubbish and what isn't :)

(Fun example: I had an otherwise high quality Java library for HTML processing crash during a MapReduce job as someone used a phone number as a port (i.e. url:port). The internet is weird. The fact it works at all a mystery ;))

[1]: https://news.ycombinator.com/item?id=13721644

One way cloudflare could mitigate the damage they caused is by coughing up some money to facilitate non-profits in their cleanup activities on behalf of cloudflare.

Polluter pays.

It seems like google and bing should be able to sue cloudflare for the hours they're spending cleaning up their mess.

They decided to cache the internet, as soon as they decide to copy it to their servers, it's their problem, imho.

robots.txt

it's not my fault, robots.txt made me do it!

Cloudflare seems to be of the opinion they deserve priority treatment. Personally I think Google, Bing and others are mainly doing this to protect the users of Cloudflare powered websites, not Cloudflare.

Well ya, nothing to do with helping out Cloudflare, but however it got there random creds are being served to the world from their product. No specific criticism on their efforts to fix it, just that they bear some responsibility to make it inaccessible from their servers as fast as possible, even if its still cached in a million other spots. Shitty task but it comes with the territory.

To be fair, access to the archives of donaldjtrump.com could be blocked anyway if the owners at some point decide to add a robots.txt blocking the Wayback Machine.

>>While I am thankful to the Project Zero team for their informing us of the issue quickly, I'm troubled that they went ahead with disclosure before Google crawl team could complete the refresh of their own cache.

Why the help was not 100% appreciated...

One of us misunderstands what happened.

> Chances are it wasn't a domain that was leaking information - though we don't know as there's no publicly accessible list and no way to tell if they had the buffer overrun features active.

As far as I my understanding goes domains that had the features on were leaking the data of other domains. So it's near impossible to tell who was affected.

You understand it right, but I think you get the conclusions wrong. If a site didn't have said features active, information from it could have leaked via other sites that had them on, and could be in archives of those other sites (= so it still should rotate secrets where necessary). But the archive of the page itself can be kept safely, since without the buggy features no leaked data from other sites has been included in it.

Ah, I get it now, thanks for explaining.

The key bit is that your page needs to have been in memory in that process for it to leak. So malformed pages that use the affected services would leak data from other pages that use those services.

Wouldn't this be hard to exploit from an attackers poitn of view? I mean there is no way to know what data was currently in RAM, it is at best a blind attack.

But it's a blind attack that is much more likely to hit big players.

And seeing as cloudflare has some of the biggest pipes on the net, it would be easy to saturate a pipe to gather as much data as possible from any sites you can find that are affected.

If I understand correctly, you could literally keep smashing f5 on an affected page and get a different chunk of memory every time. An attacker could have potentially collected a lot very quickly with a simple script.

That's, from what I understand, completely correct. Not only that, but because of the nature of the flaw, it's not clear to me that the attackers would be generating any real anomalies by doing so. They're just fetching a particular pattern of otherwise non-notable web pages.

Cloudflare is physically incapable of posting a blog that doesn't make them out to be saviors of the internet; the only way they could spin this to make them seem like the hero here is to throw Google under the bus.

Cloudflare: "We've just saved you from Google's incompetence. You're welcome."

Sure. But remember when CloudFlare breached an embargo to sell their services on Heartbleed?

They seem to care a lot more about marketing than anything else.

(On edit: which also means they have a fucking cheek whining about disclosure timings.)

I don't remember. Can you point me to an article?

That is utter crap. No such thing happened.

After seeing how Cloudflare have handled this and tried to downplay and even try to imply Google is the problem here, I suspect if this bug wasn't discovered by a third party and they themselves found it out at some point, they pretty much would have buried this issue as just another "memory leak".

I still can't believe they tried to call this a memory leak. They're obviously trying to downplay the severity of the bug by referring to it that way, but anyone who would know what a memory leak is would also know this isn't one.

Sort of funny that one of CloudFlare products is Cloud Web Application Firewall. This 'enterprise-class' solution somehow for months failed to detect all sorts of invalid and suspicious content being returned in index.htmls. It even has detection of 'Sensitive data exposure' listed as a feature. https://www.cloudflare.com/waf/

Funny but totally expected, this kind of features is always snake oil with dead simple heuristics

The way it's described makes it sound like the WAF looks solely at the request payload, and not the response.

They mention using the default mod_security rules. I know mod_security can look at the response body, but it is not a default setting.

I think that's a lazy argument. Just because Google does one good thing (finding the vulnerability) doesn't make all their actions good. The affected party (in this case Cloudflare) has incentive to handle the situation in a way that's limit the damage to their customers and their customers customers (which are the ones affected). The 'security researchers' (in this case Google) doesn't have that incentive. In many ways they even have the opposite incentive, to disclose while it's still active so they can get larger recognition. The release rules aren't made to limit harm to the world, but to limit harm to Google.

Tavis Ormandy linking to leaked data to settle some personal groll is a pretty good example how much they really care about data leaks. I guess when you've already leaked all your customers data to intelligence agencies around the world while moving it between data centers it's hard to keep any form of standards.

Want to actually argue your idea with any sort of competition I suggest going outside HN.

> The affected party (in this case Cloudflare) has incentive to handle the situation in a way that's limit the damage to their customers and their customers customers (which are the ones affected).

Cloudflare has a much stronger incentive to handle the situation in a way that limits the damage to their reputation. As you note, researchers also have an incentive to maximize the benefit to their reputation. But the incentives for P0 are much more closely aligned to the interests of the public here than are the incentives for Cloudflare. The information revealed by the bug is indeed really bad, and there's likely no possible way to tell which of Cloudflare's customers were affected, much less those customers' users. No company wants to acknowledge "yeah, a bunch of our customers' data was compromised, and we have no possible way to tell whether your data was among the data compromised". Their incentives push them to downplay the impact; to accurately describe the potential impact to their customers would probably be disastrous for them as a business. In contrast, downplaying the impact would be contrary to P0's incentive to benefit their own reputation.

I'd say it shouldn't be their responsibility to fix it at all - this is on CloudFlare. CloudFlare shouldn't be downplaying the issue to their customers.

They should be telling them to reset passwords and get users to do similar. And I say this as a generally very happy CloudFlare customer.

A few Bitcoin exchanges have realized the seriousness and have already contacted me and told me to just go ahead and enable 2fa, which I did even on empty accounts. One of which is going so far as to revoke and change their SSL certificates even though there's no reason to believe they were at risk.

> They should be telling them to reset passwords and get users to do similar.

Someone correct me if I'm wrong, but:

This isn't just about passwords. Any memory from those proxy servers could have been dumped into these responses, meaning plaintext content of all kinds. The data needs to be purged in addition to telling customers that everything they did in these months has a chance of being out there in plaintext in the hands of random people.

Yes - definitely.

I was just saying passwords because they're something that could have leaked which could be used to actively compromise accounts and result in further attacks now. And probably the only thing most sites can do anything about at this point.

Of course if you were sending private keys, api keys, etc over an SSL connection - those all now need to be treated as exposed.

Some sites may also have special concerns - btc-e for example, a fairly large bitcoin exchange allows users to withdraw to a "btc-e code" which can be deposited by another user of the site. It'd be a serious mess if any of that kind of thing leaked.

Cloudflare f*cked up big time. It's not the responsible for other third parties to clean up the mess. Especially it won't happen 100% world wide anyway, there are too many parties who have indexes (public or private).

In the end Cloudflare should be held responsible. It also puts the whole CDN service business in a bad light.

I never liked CDN's because I don't like monocultures and the whole 'we decrypt your traffic but we can be trusted' bit defeated the purpose of HTTPS to begin with.

It's convenient for content publisher and reduce infrastructure costs a lot. Just have the authenticated/editing part served directly and the published content trough cdn.

So, retrospectively speaking: Was it worth it?

to us, yes. a large customer signed up and we had a 300x traffic spike overnight, all eaten up by cdn. the first impression for a startup matters and them testing the service unannounced could have gone wrong in so much many ways.. cloudflare literally saved the day.

the publishing part needs to be designed to leverage caching properly tho. however it's much easier to setup cloudflare+custom rules than a cdn push-on-change service

Except it is the responsibility of the third parties to clean up the mess. Cloudflare has no control over their infrastructure.

Forget the blame game for a minute: there is personally identifiable information of the search engines users sprayed all over their cache. At this point it doesn't matter who put it there, there's a moral responsibility to clean this up, for the greater good.

Yes. And once it is cleaned up, send Cloudflare the bill. And once they go bankrupt, send the executive officers and the shareholders the bill. And once they go bankrupt, make sure they are still held responsible for the damage inflicted and that nobody that incompetent tries to run a major business damaging millions of people’s lives ever again.

I guess we will have to agree to disagree on this one. If nginx does the same, or apache would you send them a bill also? This is a really really shitty mistake. CF hasn't handled it well. But it could have happened to anyone.

You are of course responsible for the tools you use, so if I use nginx and it has a bug, I won’t send them the bill.

A better analogy would be to crash your client’s car into the building of a competitor and then complain if the competitor doesn’t repair that car immediately because it makes you look bad and it is for the greater good for your client to have a working car. This is what happened! Cloudflare took their clients’ cars (their data), crashed them into random buildings all over town (Google, Bing etc.) and now complains that Google and Bing don’t get their act together to repair those cars (remove the private data from the public internet).

If Amazon Prime took your car to do your grocery shopping and then crashed it into Walmart, would you really complain that Walmart can’t send the bill to Amazon Prime for the clean-up?

And once you have done all of that you will have to nationalize all but the smallest businesses because no one in their right mind will invest in anything any more.

As a result, tax payers will have to foot the bill for all further mistakes.

>Cloudflare has no control over their infrastructure.

It has via robots.txt.

I agree. Also, I have to admit to liking this because of the way Cloudflare has treated Tor users. Feels like karma.

I suspect Google (and others) must have some mechanism to comply with EU 'right to be forgotten' law.

https://www.theguardian.com/technology/2016/feb/11/google-ex...

And it doesn't stop there. I bet the bad guys (or governments) are already crawling and storing the leaked data to their own databases for later analysis.

I bet NSA didn't even have to make any network requests on this one. They either got backdoors in most cache servers or have already duplicated their contents.

I would be shocked if the NSA hasn't been running their own crawlers for years. Because, you know, terrorists may be chatting on pages not allowed by robots.txt. And just because other crawlers respect it, doesn't mean we need to...

Also they don't seem to realize how many people have caches.

Crawling the web is something that a lot of people do these days for everything from SEO (Seomoz and every other major SEO tool probably has a crawl) to web data companies (Email Hunter, SimilarWeb, etc.) to national security agencies.

I'd be surprised if there weren't thousands of cached crawls out there that managed to grab some of this data.

Pretty obnoxious of cloudfare to act this way in the wake of their own mistake.

This is a burden that Cloudflare has placed on the internet community.

Nitpick: you're referring to the web (i.e. HTTP(S)) rather than the internet (UDP, TCP, IPv4, IPv6).

Is not the web part of the internet?

If I place a brick on the cushion on my couch it is correct to say I have placed a brick on my couch.

I really enjoyed this analogy. Is this a common phrase or did you just come up with it? Why a brick?

A spoon, or a book, would work just as well IMO

It's just the first random object that came into my mind.

It just came to me. I'm glad you liked it. :-)