Quantifying the Impact of “Cloudbleed” | Hacker News Reader

Quantifying the Impact of “Cloudbleed”

blog.cloudflare.com

287 pointseastdakota128 comments

The wording of the post seems to be tilted towards limiting Cloudflare's potential liability as opposed to objectively quantifying the impact. For example, the author mentions that the 100th largest website handles less than 10B requests per month - which positions near the median of the table of likelihood anticipated leaks.

I'm not surprised at this approach as the CEO has a background in law. I think that, combined with his hubris, shaped the tone of this post from an objective post mortem to a goal of minimizing the damage to Cloudflare itself.

I met Mr. Prince once at a tech conference. I mentioned that I followed him on Twitter, to which he freaked out a little bit and said "Oh I hate that when I run across people who tell me that". When I heard him tell his name to someone else there, he said his last name was "Prince, like King". I suspect that his attitude is prevalent at Cloudflare, and may have shaped not only the source of this issue, but the response to it as well, since CEO's generally influence the culture.

> his attitude

I don't quite follow from your description of his reaction. What sort of attitude were you attempting to describe/how'd you perceive his attitude to be?

I think his attitude, as described, was that he manipulates for his own perceived self importantness and perceived gain. If you mention you follow him on twitter the response is "oh that's so beneath me and I hate it", if you don't know him, he is "prince, like a king, and an important one of that so remember me" kind of attitude. Like a person you wouldn't want to meet in a bar.

That is how I understood it anyways.

Who would say "Prince, like King" when you can say "Prince, like 'the artist formerly known as'"?

> 3) after a review of tens of thousands of pages of leaked data from search engine caches, we have found a large number of instances of leaked internal Cloudflare headers and customer cookies, but we have not found any instances of passwords, credit card numbers, or health records; and

Downplaying how bad it was that customer cookie's were leaked is what I found particularly egregious in the original response. A session cookie is almost as bad as a password leak. Usually it'll let you everything except change your password (as most sites require your current one as an extra validation).

Session cookies can be reset much more easily than asking a user to reset their password. Session cookies also aren't used on multiple sites. It's still a big deal, but leaking a session cookie is much less dangerous than leaking a password.

Lots of sites use session cookies. Many don't automatically expire them or rely on browsers to expire them (ex: signed cookies).

Plus rotating the master session key (to force the issue of all users being reset) requires knowing you should do it. By downplaying the issue, CloudFlare is sending the message that customers don't have to do anything.

>Session cookies can be reset much more easily than asking a user to reset their password.

This statement is making an assertion about the behavior of vastly different software backends and the skill-set of the people administering the systems. Things that may appear trivial to you may not be for the sysadmin in charge of servers running healthcare applications that have an opaque internal state.

Was talking with colleagues about cloudbleed and S3 problems yesterday.

I don't feel like many people are actually concerned of the implication of having an internet that isn't an internet anymore, but merely a handful of big companies hosting everyone.

Or maybe it's me who don't understand.

It's almost as if self-reinforcing feedback loops are all over society these days. Being at the top makes it that much easier to stay at the top and crush everything else. Internet companies are just one small slice of this trend.

antitrust is all bark with no bite these days. Most industries having 3-4 "competitors" that don't actually compete is the new norm. I wouldn't be amazed if a link is shown between this situation and the increasing wealth inequality

That's because you're viewing this situation entirely from within your prior beliefs about large corporations, which somehow have convinced you that only 3 or 4 hosting companies exist and everyone must use them.

Exactly how do you figure this? There are lots of alternatives to S3:

1. All the cloud stuff, which is the new fangled hotness (3 to 4 companies)

2. Non-cloud hosting providers (hundreds of them, from shared hosting to VPSs and above, and usually cheaper if you don't have high traffic)

3. Hosting your own stuff on your own hardware, which is the cheapest if you really need it, and can be done in a data center just about anywhere in the world with good internet.

So, what planet is this that you live on where 3 to 4 companies handle all hosting and we have to use them?

Beyond simple anti-trust, in many cases, we need to reform legal structures so that big players can't use them to block out startups.

It's practically impossible to defend yourself well in a lawsuit against any respectably-sized company without a few million lying around, first of all. That affects everything, and big companies use that fact to bully upstarts and other small innovators into shutting down all the time.

My own business was effectively shuttered (had to stop selling our primary product) by a C&D from a Fortune 100. It would've been 5-10 years and ~$5 million to see that case all the way through, and under current precedent, it's very likely I would have lost.

Aside from that, industries frequently get laws and rules put in place with ostensibly-reasonable rationale, while the actual intent is to make it virtually impossible for disruptive competitors to enter the marketplace.

The CFAA is the piece of legislation that primarily enshrines entrenched players in the online space. We also need to reform copyright law and clarify some matters regarding the applicability of EULAs, especially with regard to clickwrap and browsewrap.

Once that's done, the flood of innovators that have been held back by big companies dispatching their law firms will finally be able to contribute, and the internet's competitve landscape will truly be back in the hands of the users. It will shift from "Who has my data? I have to go with them" to "I can use any interface I want to access that data", effectively resolving the chicken-and-egg effect that imperils any potential competing social network (not even Google could compete with Facebook on this!).

> It's practically impossible to defend yourself well in a lawsuit against any respectably-sized company without a few million lying around

Large corporations keeping a bunch of lawyers on their payroll are the equivalent of large nation states stockpiling nuclear weapons.

Antitrust laws were remarkably different in the recent past. Here is an excerpt from http://washingtonmonthly.com/magazine/novdec-2015/bloom-and-....

"To get a flavor of how thoroughly the federal government managed competition throughout the economy in the 1960s, consider the case of Brown Shoe Co., Inc. v. United States, in which the Supreme Court blocked a merger that would have given a single distributor a mere 2 percent share of the national shoe market."

What are you proposing? We should all serve off of our DSL lines?

Collocation, dedicated servers, even VPS. There's plenty of a middle ground between serving off your home server and AWS, Cloudflare et al.

And to be blunt, I'd even claim that the middle ground is superior for most people than AWS, Cloudfare et al.

I have limited experience with AWS, but they always seem to be having problems that ruin everything that they don't acknowledge. That's what I get from people at the office who have to use it to support a particular customer. Nothing we do ever seems to run properly on it, even though our software is fine everywhere else. We're trying to push that customer off to using their own hardware, and they agreed with us that it's necessary but for different reasons.

Cloudfare, I don't even understand why anyone would use it. I understand the benefits it claims to provide, but you can get those without having an external MITM proxy that can spew information all over the Internet with one bug.

that would be cool

I wish. Too bad there isn't any meaningful competition in the ISP space for most people.

Ah! Maybe that would encourage people not to make 15mb pages doing dozens of requests!

http://urbit.org/

20 years ago, sites got "slashdotted" left, right and centre. Cloudflare and friends put an end to that, forever. Sites went offline for days if not weeks because of hardware failure -- some never came back because they didn't back up correctly. S3 went offline for five hours, but didn't lose a single bit of data.

You're looking at the past through rose-tinted glasses. We learn all the time, and we will probably also learn to build some resilience into our systems against these issues. But as someone who's been through a thing or two (see above) on the Internet of yesterday, I like the one we have today.

Some website being Slashdotted is not in the same class of issues as users' (possibly encrypted) page content being spread all over the web. Also, the notion that you either need to use some super-centralized platform or be susceptible to Slashdot effect or loosing backups is a false dichotomy.

Cloudflare had nothing to do with ending slashdotting. This stopped being a problem years before they existed.

Slashdotting was mostly a problem caused by Apache's incredibly inefficient design. It consumed huge amounts of memory per connection at a time when most of us had very slow connections. A link from Slashdot was, in effect, a Slowloris attack on your server.

The big change was moving from a fork/thread-based webserver (Apache) to an event-based webserver (nginx), which was made even more efficient by kernel features like epoll.

This sounds plausible, but is there any way to validate this with some kind of empirical evidence?

> This stopped being a problem years before they existed.

I still see this issue happening daily here on HN.

And every time it happens 10 people jump to comment about how the site is written poorly and could probably be an entirely static site.

Sorry to be blunt, but this is just incorrect.

The problem with "Slashdotting" was the number of concurrent connections. Heck a fair portion of the time it was the database that keeled over first, not Apache.

Slowloris attacks send purposefully incomplete requests and hold them open with additional headers. Even with dial-up modems, connections were never slow enough for this to be a problem with actual requests, which are lightweight.

Responses are heavy and can tie up slow connections, especially if they have to go get stuff out of the database. But in that case it's no longer a Slowloris type attack. It's just too many concurrent connections.

The Slashdot effect was solved with static HTML caching, simply because caches are faster and don't touch the DB. Cloudflare is a simple, free example of such a cache, although certainly not the only one.

Bluntness is okay, but aren't you wrong about me being wrong here?

I didn't say it was a Slowloris attack. I said Slashdotting was "in effect" the same thing. Which it is, both problems are one of exhausting limited concurrency.

> The problem with "Slashdotting" was the number of concurrent connections.

Exactly the same problem a Slowloris attack exploits.

> Responses are heavy and can tie up slow connections...

Yes, responses tie up the limited number of available httpd processes.

The problem was that Apache couldn't even serve static files to many clients because of its heavy weight httpd processes and the fact that clients were so slow.

If your web server can only handle 200 concurrent connections, and you want to serve a 500 KB screenshot of your 1337 Linux desktop to clients that download at 3.5 kbyte/s, you can handle like ~1.4 req/s. Doesn't take much to get Slashdotted.

Whereas, event-based webservers could handle at least 10x more connections on the same hardware even before epoll existed.

I had this problem in 1998 and fixed it with select/poll based servers, and then eventually other epoll-based servers before nginx existed.

If we go back to 1998, maybe network throughput was the limiting factor that drove up concurrent connections and killed servers. But I also don't think we can say nginx solved that, since nginx didn't start seeing wide usage until about 10 years later.

I guess I wrote what I did because the comparison to Slowloris seemed to over-emphasize the importance of handling high numbers of concurrent connections, since that is the only mitigation for Slowloris.

But, for a flood of real traffic, concurrent connections and throughput are related. The faster your web server can serve responses, the fewer concurrent connections it will need to handle. And as the percentage of dynamic DB-backed sites has increased over time, so has the value of caching. Basic page caching can speed up a Wordpress blog by hundreds of times for unauth'd users, for example. For most little sites, implementing caching will get them more than installing nginx.

And really, what good are valid concurrent connections if the throughput isn't there? For most users, a site that waits 5 minutes on a blank page is no better than a server that's down.

One of the main causes for that was a one line fix in the apache config:

 	KeepAlive Off

Yup!

Wait. How does Clouldflare prevent a site from going offline for weeks because of hardware failure?

S3 is the reference here, not Cloudflare.

Nonetheless, CloudFlare caches sites and may serve a cached version of you site when it's down, assuming it's "static" enough to be readable from cache.

>>Sites went offline for days if not weeks because of hardware failure -- some never came back because they didn't back up correctly. S3 went offline for five hours, but didn't lose a single bit of data.

S3 going offline for five hours probably caused orders of magnitude more damage. The reason is simple: the stakes are higher today than they were 20 years ago. Back then the overwhelming majority of the business in most companies was conducted on paper and in person. Losing a website for a few days, or even a few weeks, wasn't a big deal. Today though? There are so many companies that host business-critical operations and infrastructure in the cloud that it's hard to fathom how they are coping with being taken offline for several hours in the middle of the week.

I'm sorry, but I disagree that we have an internet that isn't an internet.

Even though two particular problems affected a great many sites, they didn't affect every site or even a majority of sites. That's because we don't actually have a "handful of companies hosting everyone."

Maybe I'm not a typical user, but the only site that was affected by S3 that I use was HN. The only three sites affected by Cloudbleed that I use were HN, Discord, and another smaller site elsewhere. The rest of the internet worked perfectly fine for me.

I'm pretty sure 90% of the sites typical users were unaffected also. I mean, facebook didn’t go down, did it? People could still search on The Google. I don't know of anyone whose job was affected by this aside from people who run websites. Neither of these were as bad as the Dyn outage, which wasn't the apocalypse either.

I agree that over-centralization of the Internet to be concerned about actually creating, and it's definitely something you should consider before choosing services like Cloudfare and S3. But I don't think it's so bad today that the Internet isn't the Internet anymore.

> I don't feel like many people are actually concerned

Judging by how frequently I read this sort of comment on HN, there are definitely many people who are concerned here.

Man, back in my day the internet was even LESS of an internet than it is now. Very few companies had websites, and very few hobbyists could afford to run a website (there were few hosting companies, and it cost $100 to register a domain name with the only registrar around, Network Solutions). Running an internet presence was EXPENSIVE.

Sure, there has been consolidation in the industry, but it is much easier to exist on the internet now than ever before.

Not everything Cloudflare has written about this breach has been good, but this is solid. In particular, I'm satisfied that commercially reasonable efforts have been made to ensure there wasn't deliberate exploitation of the bug.

What remains now is an accounting of how some of the most sensitive C code on the Internet was tested prior to the discovery of this bug (by Cloudflare's own accounting, the underlying error seems to have been present long before it became symptomatic), and what they're doing about that now.

>I'm satisfied that commercially reasonable efforts have been made to ensure there wasn't deliberate exploitation of the bug.

How would you classify people using all of the crawled data though? It seems sort of hand-wavy to claim nobody deliberately exploited the bug if they used leaked session tokens in crawled data to get access to user accounts.

One aspect I haven't seen mentioned yet is that there is a geographic dimension to what ends up in search engine caches; their crawlers end up talking to the nearest Cloudflare point-of-presence so any leaked data is also going to be from requests served in that same general area.

For example, my Cloudflare-using site is in the 1B to 10B requests a month bracket, meaning 112–1,118 anticipated leaks per Cloudflare's calculations. It's enough that you'd expect to possibly find some in the search engines, but we haven't.

One potential explanation is that our usage is heavily concentrated within a geographic area (the Nordics), and this just isn't where the big search engines are doing their web crawling from.

Yes, that's correct and not something our analysis has taken into account. Search crawlers are more likely to hit certain PoPs. If your content was less likely to be in one of those PoPs then you're less likely to have leaked data to one of them.

We also didn't take into account that the distribution of the ~6,500 sites that triggered the bug weren't even across our infrastructure. We distribute load for any site across some fraction of all the servers in any given PoP. Based on whether you're on a cluster of servers with more or fewer vulnerable sites you'd have been more or less likely to leak data.

For the general case the stats are directionally correct and, if anything, conservative (we believe, if anything, overstate the risk). But a particular customer's situation could deviate from the expected probabilities for a number of reasons.

The problem is that CloudFlare encourages a fundamentally insecure architecture. CDN should be used to host static files on a separate, cookie-less subdomain. With such architecture consequences of any data leak from a shared CDN infrastructure are limited.

With CloudFlare all the requests are passed through the CDN. With dynamic, authenticated requests the benefits of this are dubious. The responses can't be cached anyway, an additional HTTP(S)-level hop is required for each request (and many additional IP-level hops). The drawbacks are now obvious: sensitive, unencrypted data of thousands of different sites resides in the same, not isolated, shared process memory. With such architecture we will always be one C pointer manipulation bug away from another leak.

The reason CloudFlare encourages this is because it isn't meant for use as a traditional CDN - they're not a CDN at all really. You don't upload content to them. They're meant as a cache and anti-DDoS proxy more than anything else.

Look at what happens when anyone gets hit with a large scale DDoS attack these days, the first thing people will tell them, myself included is "CloudFlare". How do you handle a massive spike in load on a tiny, usually empty server? CloudFlare is the go-to tool for this purpose, especially for people who can't afford to pay Akamai, BlackLotus, Amazon or others. And they're impressively cheap for small businesses too. You can't hide from a DDoS attack or save your tiny DigitalOcean box if your web server isn't hidden away behind a service like this.

I have nothing against CloudFlare, they've proven themselves extremely trustworthy in many ways and even in handling of past security issues they've been very responsible. But this time that's not the case - the downplaying of the issue is really damaging their reputation in my eyes. I'm a paying customer and I'm quite disappointed to see this.

> They're meant as a cache and anti-DDoS proxy more than anything else

You can't cache responses to authenticated requests, so in case of DDoS attack against an HTTP end point that requires authentication the best CloudFlare can do to safe the back-end is to drop requests, which makes the attack successful. I really can't see benefits of passing authenticated traffic through CloudFlare.

That assumes all users who are performing the attack are authenticated - an extremely unlikely scenario. All said users would also have to be able to pass a captcha which gets engaged in the case that a site is under active attack.

> the best CloudFlare can do to safe the back-end is to drop requests, which makes the attack successful

It can also drop the attack requests, which is generally not from authenticated users, and pass the real traffic to the backend.

Only in the case of real user, authenticated traffic is CloudFlare not a solution. In cases of high unauthenticated user or high fake user traffic or in cases of attacks not operating on HTTP at all, CloudFlare will solve the problem perfectly. Some of the nastiest attacks, like reflection attacks, don't even touch HTTP at all but will quickly knock most servers offline - even lead many server providers to nullroute you to protect their other customers. It'll also get you out in cases of high real user, unauthenticated traffic - like being posted on HN, reddit, etc.

You can cache authed requests. I'm not sure cloudflare supports it, but it's very common.

Even without caching, it's useful to run traffic through frontend servers that are close to users. Terminating SSL as close to visitors as possible speeds things up. Getting visitors off their ISP network and on to quality backhaul helps too.

You aren't typically DDoSed by your authenticated users. Cloudflare can validate authentication tokens at the edge; they also have a feature, Railgun, that can potentially cache portions of those authenticated responses. (No affiliation to Cloudflare, btw)

It was my understanding that people were finding numerous examples of leaked data that included Authorization headers, which don't always leak passwords but which still leak session keys. Maybe if espadrine from this linked comment reads this, he could explain more?

https://news.ycombinator.com/item?id=13719455

FWIW, as an example, Grindr uses CloudFlare: do a Google search for "authorization: grindr3" and you will find a URL which (no longer cached but you can still get snippets) contained an authenticated grindr request, which would be enough to have had temporary access to that person's account.

https://costumla.com/wild-west-costumes-for-men.html

"... U Edge Certificate Authority1 0 U San Francisco1 0 U California�1p ��U� N��U � � �f"�0! %��U@ @GET /v3/profiles/[REDACTED] HTTP/1.1 CF-RAY: 33282514b8d957d7 FL-Server: 15f76 Host: grindr.mobi X-Real-IP: [REDACTED] Accept-Encoding: gzip Client-Accept-Encoding: gzip X-Forwarded-Proto: https Connect-Via-Https : on Connect-Via-Port: 443 Connect-Via-IP: 104.16.85.62 Connect-Via-Host: grindr.mobi CF-Visitor: {"scheme":"https"} CF-Host-Origin-IP: [REDACTED] Zone-ID: 22252132 Owner-ID: 2607399 CF-Int-Brand-ID: 100 Zone-Name: grindr.mobi Connection: Keep-Alive X-SSL-Protocol: TLSv1.2 X-SSL-Cipher: ECDHE-RSA-AES128-GCM-SHA256 X-SSL-Server-Name: grindr.mobi X-SSL-Session-Reused: . SSL-Server-IP: 104.16.85.62 X-SSL-Connection-ID: 15d1b8de6025864d-DFW X-SPDY-Protocol: 3.1 authorization: Grindr3 ... accept: application/json user-agent: grindr3/3.0.13.16790;16790;Free;Android 6.0 CF-Use-OB: 0 Set-Expires-TTL: 14400 CF-Cache-Max-File-Size: 512m Set-SSL-Name: grindr.mobi CF-Cache-Level: byc CF-Unbuffered-Upload: 0 Set-SSL-Client-Cert: 0 Set-Limit-Conn-Cache-Host: 50000 CF-WAN-RG5: 0 CF-Brand-Name: cloudflare CF-Age-Header-Enabled: 0 CF-Respect-Strong-Etag: 0 Set-Proxy-Read-Timeout: 100 Set-Proxy-Send-Timeout: 30 CF-Connecting-IP: [REDACTED] Set-Proxy-Connect-Timeout: 90 Set-Cache-Bypass: 0 Set-SSL-Verify: 0 CF-Force-Miss-TS: 0 Set-Buffering: 0 CF-Pref-OB: 1 Set-Keepalive: 1 CF-Pref-Geoloc: 1 CF-Use-BYC: 0 CF-IPCountry: [REDACTED] CF-IPType ..."

edit: I have spent the last fifteen minutes pulling the search snippet (edit: and now an hour; but all on my phone, so this is harder than it should be, and I also am distracted by other stuff). The way you do this is by walking through the parts you can see to get nearby context. (I do this a lot to pull content purged from or inaccessible to or simply updated in Google's cache.)

In so doing, while I haven't been able to still have access to the session key (likely too long and unique for a snippet), I have pulled the X-Real-IP address field of this Grindr user and a profile identifier they were checking out (both of which I redacted above, but you could trivially get yourself now using that context).

CloudFlare: if you think there isn't private data that was leaked, OR EVEN PRIVATE DATA STILL ACCESSIBLE, you are a bunch of fucking idiots. 1) Clearing the cache isn't sufficient, as for anything built from short plain text words we can pull the snippet. 2) IP addresses and session keys count as "private data". 3) GET requests actually are often sensitive information :/.

I am the second responder in that thread.

I personally saw an oAuth bearer token for a user of Fitbit. It was there clear as day.

Maybe I saw the only one that leaked, but that seems...unlikely.

OAuth2 does indeed send a secret bearer as an "Authorization: Bearer" header, but OAuth1 does not.

The FitBit stuff I recall seeing other day was OAuth1. It would look like oauth_token=XXX and oauth_consumer_key=YYY, along with oauth_signature=ZZZ. If that's what you're seeing, it's not a bearer token and that data isn't secret. Those values are essentially record locators.

OAuth1 has an undisclosed "consumer secret" (embedded in the app) and "token secret" (acquired at login) that are used to sign the request. So these requests don't leak any usable tokens. If you also see oauth_nonce and oauth_timestamp values, then the signature is also protected from a replay attack.

The response body of the request that originally acquired the token/secret would leak these secrets.

I'm sure there are bearer tokens out there for other services, but if fitbit is using OAuth1, you'd have to snag that original token acquisition to call the api as that user.

Edit - I hope this doesn't come off as argumentative. I just want to point out the good design choices made by the guys that weren't actually compromised by this. OAuth2 is really easy to implement, but it kinda feels like a step backwards security-wise.

> The FitBit stuff I recall seeing other day was OAuth1.

Fitbit uses oAuth 2

https://dev.fitbit.com/docs/oauth2/

"OAuth2 is really easy to implement, but it kinda feels like a step backwards security-wise."

No, no it isn't. There are about three proper ways to implement it. Almost nobody whose site I've come across that uses OAuth2 has it properly implemented.

Token revocation and authentication is hard, folks.

I don't see anywhere in the post where they claim private data wasn't leaked. In fact, they explicitly state that authorization cookies were leaked. What they said they didn't find was passwords, CC info, health records, SSNs, and customer encryption keys.

They even have a paragraph talking specifically about cookies in GET requests:

> This is not to downplay the seriousness of the bug. For instance, depending on how a Cloudflare customer’s systems are implemented, cookie data, which would be present in GET requests, could be used to impersonate another user’s session. We’ve seen approximately 150 Cloudflare customers’ data in the more than 80,000 cached pages we’ve purged from search engine caches. When data for a customer is present, we’ve reached out to the customer proactively to share the data that we’ve discovered and help them work to mitigate any impact. Generally, if customer data was exposed, invalidating session cookies and rolling any internal authorization tokens is the best advice to mitigate the largest potential risk based on our investigation so far.

It's still weird to enumerate things not found, but no mention of examples like dating site messages that were found. They're obscuring that by focusing on internal headers, etc. Why does the table with "0 health records" not have an entry for "X private correspondence"?

My biggest issue is that they didn't release their data-set. With something this major, it's standard to either have a third party investigate or publicly release your data so it can be validated.

For all we know they cherry picked the responses they tested from a single site that doesn't handle anything sensitive.

> For all we know they cherry picked the responses they tested from a single site that doesn't handle anything sensitive.

I don't think you understand how Cloudbleed works. It doesn't matter what site they picked; every single vulnerable site can leak the exact same info. It's literally impossible to cherry-pick that data.

I don't think you understand how the internet works. Some websites only serve static content and don't deal with any sensitive information. Without seeing Cloudflare's data set there is no way to verify that the responses they picked are a representative sample.

Ok you really don't know how Cloudbleed works. Go read up on it. Every single vulnerable site can and did leak the same information. The only way to "cherry-pick" it would be to literally throw away the responses that you saw and didn't like, or in other words, by lying.

I think he was trying to explain to you that, for this particular leak, what was leaked was the private memory of the CloudFlare servers. So that memory doesn't have all a single site in it. It doesn't matter what site triggered the data to be output, the data that was output can still come from any CloudFlare customer even if they had no pages with the condition that triggered the issue.

There's 3 classes of information:

1. Information where improper disclosure is illegal. For example, health records. Or credit card details, which would presumably be a violation of PCI DSS.

2. Information that can be actively exploited, but can also be fixed so the previous disclosure is harmless. This means passwords, authentication tokens, etc.

3. Information that is merely private in nature.

Cloudflare is focusing on the first two items. The third one is hard to quantify; what one person may consider private, another person might not care about. And there's not much that can be done about this kind of disclosure (beyond scrubbing caches which they're doing anyway). Also, it's difficult to automatically identify this type of content (whereas cookies, passwords, credit card numbers, etc are pretty easy to detect) and Cloudflare probably doesn't want to have their employees spending their time reading through all of the cached data they can find looking to see if there's private info (both because it's a lot of work and a huge waste of time since it won't affect anything, and because it's private info; chances are nobody's going to see it normally, and having employees reading your private messages doesn't help anyone and is a violation of your privacy).

Plus a 4th class: public information.

I don't know how they could determine how many health records were leaked without looking at and classifying the data, so they've already gone ahead and done that. They presumably know how many steamy Grindr messages were in there.

Additionally, there are laws about messages as well. Email laws generally don't specify smtp only.

Health records probably have some sort of health record ID, and there's only so many formats that can take. Detecting strings that match certain ID patterns is easy.

That's one hell of a busted test. So a page fragment could leak my name and "hurts when I pee" but if the universal standard ID was cutoff it's not a health record?

2. Information that can be actively exploited, but can also be fixed so the previous disclosure is harmless. This means passwords, authentication tokens, etc.

I wouldn't call the disclosure harmless. It's unknown if anyone made use of the leaked information before Cloudflare knew, so accounts should be treated as compromised unless it's shown otherwise.

Also, leaking user credentials to any system that handles payments and health info would also breach PCI/HIPAA . This broadens the scope of systems effectively breaking the law.

Another thing to keep in mind is that many(most?) token based authentication systems don't invalidate tokens. So any tokens captured will be valid until they expire, and they can't be "changed" without invalidating every outstanding token (changing the server key)

No I mean after it's fixed, the previously-disclosed information becomes harmless. Obviously anyone who exploited it before you reset your password/tokens may have caused you harm.

> Another thing to keep in mind is that many(most?) token based authentication systems don't invalidate tokens.

In my experience, changing your password generally invalidates all outstanding tokens. And yes, this does mean invalidating all of them instead of just the leaked one, but that's not usually a big deal.

> 3. Information that is merely private in nature. > Cloudflare is focusing on the first two items. The third one is hard to quantify; ...there's not much that can be done about this kind of disclosure (beyond scrubbing caches which they're doing anyway). Also, it's difficult to automatically identify this type of content...

I have some pretty fundamental moral issues with this position. Cloudflare is measuring the risk that you were affected by something they can fix, not measuring the risk that you were affected by something you consider important. They are absolutely downplaying the importance of all of this private information: saying "this isn't to downplay" is like adding "we have no affiliation with" at the bottom of a massive trademark violation: it means nothing. That something is difficult to measure and might be impossible to fix it does not somehow make it unimportant.

Let's translate this: we find that some company has been dumping a bunch of random chemicals in our water supply. They respond with an attempt to make us not be concerned, but they concentrate their analysis on a handful of toxins that can be "corrected" with an antidote or a chelation or some other form of direct mitigation in the water itself, while giving lip service to something which merely increases your cancer risk and carefully not mentioning the stuff that will just make you violently ill for a couple days, as it is difficult to know what will cause that, it is a dosage mediated effect, and they can't fix it.

Meanwhile, people keep reporting that they are measuring the water coming out of their tap and keep finding junk in it, and some of the stuff they are finding would make people sick if they drank it. Are you seriously telling me that you think this kind of statement is legitimate?

I am going to maintain: these Cloudflare headers are themselves private information. Even if you ignore Authorization headers, just knowing what URLs people are browsing is something that we would normally consider to be a serious problem... I mean, all BEAST leaked was the size of the files you were downloading, and people take that seriously: the fact that there are even worse potential problems shouldn't distract us from the base issue.

> 3. Information that is merely private in nature.

Whether information is merely private is a judgment call that we may be ill equipped to make since we lack important context.

Being outed as gay may range from merely inconvenient for my coworker where friends and family already know and folks from the company have much evidence to assume so to life-threatening for a person that's living in a state where LGBT* people are actively prosecuted by the state and live under the threat of death penalty for merely living their life. Anything in between is possible.

Cloudflare mentioned they only went through about 2000 leaked responses out of over a million. That combined with random people on hackernews STILL finding private data after the purge points towards this leak being a lot worse than they're letting on.

Thankfully CloudFlare spent a week cleaning up the leak in search engines and caches before publicly announcing the issue, so a lot of the evidence is gone.

It seems to me like the point of this post was to clarify whether the vulnerability was being actively exploited prior to the announcement, and I think it does that well.

Just think of how easy it would have been to exploit this bug: buy a load of random domains, host some plausible looking but malformed content on it then setup a few thousand bots to hit those sites and harvest the responses. Keep that plugging away for a few months and you'd easily collect a lot of supposed-to-be-private info.

That would have been an order of magnitude worse than having to sift through caches for scraps. Not to say that what's happened is 'good', but it could have been a whole lot worse.

Disappointing that their response is "damage control" and downplaying impact.

Why? As long as they come along with facts and data, then that's a good thing.

they only sampled a few thousand leaked responses out of over a million. The margin of error is 2.5% on conclusions because they didn't use enough data, not even close.

But also, we gotta start writing stuff like this parser in safer languages where this class of bug simply can't happen, right? Every time one of these breaches occurs, it's basically the same thing. "Data wasn't what we expected, we read a bunch of extra junk that had nothing to do with the input." It's kind of crazy it's acceptable at all.

At this point, these kind of problems are on us as a community that we keep using unsafe tools. Every time we choose one of these languages we are implicitly trading security for performance (a.k.a. money).

If things aren't written in a "safe language" then they should at least be tested. Valgrind should have picked this issue up.

The lesson I really hope Cloudflare learns from this is to do thorough fuzz testing before deploying anything (and to write their parsers in a memory-safe language like Rust).

I've linked to this post from the top of the pirate/sites-using-cloudflare README. I'm glad that it's a refreshingly level-headed response that isn't placing the blame on Google or other caches. It's still down-playing the potential impact slightly, but at least it's a more thorough response than some of the HN comments we've been seeing from CF employees.

My advice to friends and family currently is to use their "log out all active sessions" buttons, and to reset only crucial passwords. As Techcrunch mentioned in their coverage, many sites will likely pay for identity theft insurance to cover the change of a leak, rather than force password resets and lose their users trust. It's a risk analysis that each site needs to make individually, and it may be the right call to eat the losses of ~10 possibly compromised users instead of forcing millions to reset their passwords.

In total, between 22 September 2016 and 18 February 2017 we now estimate based on our logs the bug was triggered 1,242,071 times.

Wow, so just as bad as we thought.

We did not find any passwords, credit cards, health records, social security numbers, or customer encryption keys in the sample set.

BUT WAIT, THERE'S MORE

The sample included thousands of pages and was statistically significant to a confidence level of 99% with a margin of error of 2.5%.

Oh, so it could actually be as a high as 2.5% leaking encryption credentials. And if none of the data was found to leak anything sensitive where the fuck is the dataset? I've been around way too long to take a "study" like this at face value without third party verification.

I also enjoy the straight up lie at the end:

We are continuing to work with third party caches to expunge leaked data and will not let up until every bit has been removed.

That sounds great right? Well, its too bad that a lot of 'third parties' are a box sitting on the corporate network edge that hasn't been touched in 5 years. Deleting all of this data from third party caches is not physically possible. In fact it might actually make things worse because it's destroying evidence of which credentials were leaked.

1.2 million page views over 5 months is almost nothing for the amount of traffic going through Cloudflare.

And 1.2 million people dying is nothing compared to the number of people on the earth.

IMO a leak this bad should be enough to sink cloudflare. A provider of SSL was randomly spitting out private data onto public websites. OVER A MILLION TIMES. Entire CA's have been shut down for leaking a couple hundred certificates. This has leaked private data over a million times, cloudflare is a joke

> And 1.2 million people dying is nothing compared to the number of people on the earth.

A != B

It seems to me that the absolute number is what's relevant, not how it compares to the total amount of traffic. That's 1.2 million potential data leaks. That's it "out of a bazillion" doesn't change it.

Correction: It's 1.2 million definitive leaks.

The only question is what is in these leaks exactly.

For reference, two years ago Cloudflare was serving more than three million requests per second[0]. I can only imagine this figure has gone up a lot.

0: https://youtu.be/LA-gNoxSLCE?t=2m33s

If you are this concerned, change your credentials.

So every website that uses cloudflare should ask their users to change all of their passwords, credit card numbers, and SSN's?

This leak is being downplayed by webmasters because it's so incredibly bad that there's no way of handling it. The credentials of practically any internet user could have been leaked. The only "safe" way to handle this is to give everyone in the US new credit cards and SSN's and to reset accounts and security questions for every user on a site with cloudflare

No but judging by how much you are freaking out in the comments, I was recommending you to change yours. Im not really sure what you want anyone else to do? It seems like you are just screaming at your monitor over something that while a significant bug isn't a huge deal. This is just basic risk management.

Credit cards and SSNs are regularly compromised. The real problem is that they are used as an authentication mechanism. That's what we should be concerned about.

This issue is a drop in the bucket when it comes to the amount of sensitive data leaked.

Password would not help if session cookie is leaked. In many instances you cannot do anything towards that, as many services do not have any "logout all"-feature.

And this is why I changed the key I use for cookies on the application I had that was behind Cloudflare. This triggered all users to be logged out and invalidated any session cookie out there.

So, yes, responsible websites can mitigate session cookies being leaked.

That said, I am not impressed by Cloudflare's transparency which in this case consists of downplaying things, blaming Google and Taviso and not really taking responsibility.

It's good practice to destroy all sessions (besides the current one) when a password is changed, since a password change suggests that the old password may have been compromised. Not sure how many websites do that in practice, though.

On sites I write, I hash the hash of the current password into the session key. That way if you change your password all sessions are invalid, even if you change your password to itself.

1.3m pages served with "Extra" data is miniscule considering the number of actual pages served. Ideally you'd have no pages served but when you've got a bugproof technology the world will be your oyster.

> We are continuing to work with third party caches

One of the caches they worked with was Baidu, which has direct ties to Chinese intelligence. Just because it isn't publicly available doesn't mean people aren't still pouring over it looking for useful data.

Also, a lot of web spiders are not benign. All sorts of bots troll the internet looking specifically for data leaks like publicly visible email addresses and SSN's. I'm sure they're having a field day.

Just consider all the private crawlers whose caches weren't purged.

Anything short of invalidating any and all tokens, cookies and sessions is irresponsible and CloudFlare should communicate is as such.

These exact looking numbers are based on extrapolating a look at 1% of all requests, for the sites they think were leaking, for a 10 day period.

They don't get into the full detail of how they knew which sites to look for. For example, if I created a site with these settings, used it for a week, and then deleted it...is my site counted? What if I changed the settings back to normal? Does it slide under their radar? Did they really look for every site that had those settings at any time, or just any site that had those settings at the specific time they looked?

To me, it still feels like there's a window where a bad actor could have created a site that triggers this behavior, and then pumped it with a scraper, over and over.

Yeah, exactly - this "look at a 10 day period and extrapolate" idea is based on the assumption that the rate of hits to the triggering pages was constant over the whole time, but in the case of looking for a malicious actor that's essentially begging the question. If there was unnatural traffic in there you can't assume its rate would have stayed constant over the whole period.

I'm glad to see an article that isn't all puppies and kittens, this wasn't a fun thing to deal with and certainly wasn't anticipated. The CloudFlare team did a great job responding to it IMHO.

"Of the 1,242,071 requests that triggered the bug, we estimate more than half came from search engine crawlers."

This is very important to sort out, most people don't think about security much less the storing of credential or identifying data by search engines, this is a huge part of the incident response.

I think that what we should take away from this is that even though the bug existed it was responded to in a reasonable manner.

tl;dr excerpted:

In total, between 22 September 2016 and 18 February 2017 we now estimate based on our logs the bug was triggered 1,242,071 times.

1) we have found no evidence based on our logs that the bug was maliciously exploited before it was patched;

2) the vast majority of Cloudflare customers had no data leaked;

3) after a review of tens of thousands of pages of leaked data from search engine caches, we have found a large number of instances of leaked internal Cloudflare headers and customer cookies, but we have not found any instances of passwords, credit card numbers, or health records;

and 4) our review is ongoing.

>we now estimate based on our logs the bug was triggered 1,242,071 times

Based on looking at 1% of their log entries for 10 days of the total ~150 day period.

>In total, between 22 September 2016 and 18 February 2017 we now estimate based on our logs the bug was triggered 1,242,071 times.

So uh, Cloudflare has logs of all page loads since september at the very least? And I guess with response sizes, since that seems like the most reasonable way for them to come up with such a number.

Awesome.

Doubt that they have any log, they have explained before that they only keep 4hours of logs because it's too much volume of data. FYI: Traffic is over 4M requests per second.

Guess they only kept a counter of requests. A bit of maths and you get an estimate for the amount of leaks.

Their messaging is a bit strange. They write:

> For the last twelve days we've been reviewing our logs to see if there's any evidence to indicate that a hacker was exploiting the bug before it was patched. We’ve found nothing so far to indicate that was the case.

right after they explain:

> Cloudbleed is different. It's more akin to learning that a stranger may have listened in on two employees at your company talking over lunch. ... you can't know exactly what the stranger may have heard, including potentially sensitive information about your company.

So they say they've found no evidence after explaining that the situation is analogous to a case where no evidence can be found even if information were stolen.

I'm not going to comment on the methodology of CloudFlare's analysis, but just the fact that they are logging millions of unencrypted HTTP requests (cookies, headers and all) is just scary! Wasn't the point of SSL/TLS that intermediate proxies cannot peek into requests?

We're not logging "cookies, headers and all".

My bad! Just re-read the article and it did say information about requests and responses. Thanks for the clarification.

A theoretical question.

There seems to be some question about whether any passwords were actually leaked, but in principle some password data could have gotten out.

What would actually have been leaked, a cleartext password or the encrypted/hashed password stored on an affected server (or both)?

I have accounts on a number of sites that were affected by Cloudbleed, as indicated by http://www.doesitusecloudflare.com/, and I've already changed my passwords on those sites. But the passwords I use are randomly generated and should be practically unguessable given a hashed version. Would I have been at risk if I had left my passwords alone?

I forced user password resets on a forum I run. It is powered by myBB.

This is towards the edge of my knowledge circle but...

It doesn't appear that any hashing takes place client side[0] with myBB before the password is transmitted to the server, but I may have missed it when rechecking the code.

Presumably that would mean in my case users who logged in potentially had their cleartext passwords pass through Cloudflare's network when they submitted their login details.

Despite being behind https, when https is terminated at Cloudflare's edge and re-encrypted to my origin server (I use full/strict crypto setting), there was the possibility for decrypted data to be held in memory and then the possibility for it to be injected into the html pages of other sites when the bug was triggered.

It would be a very very small probability of actually happening, but not zero.

No details from the server should have leaked because only the content of transmitted data that passed over Cloudflare could have been leaked and the comparison of the provided password to the hashed/salted one in the database would have occurred on the server and not passed over the network.

So the answer is it would depend on the way the sites you use handled password transmission and comparison. Personally I'd change them, and actually have done so for sites I use.

[0] https://community.mybb.com/archive/index.php?thread-88668.ht...

> What would actually have been leaked, a cleartext password or the encrypted/hashed password stored on an affected server (or both)?

Depends on the app. The apps I write will do a PBKDF2 w/ 20K rounds on a password before sending to the server, which then hashes that hash again before storing/comparing. I suspect most apps just sent the username/password plaintext over SSL, meaning the plaintext passwords would have been leaked if the app used Cloudflare to terminate SSL.

That said, if someone got the PBKDF2ed password hash from the SSL payload, they could just as easily use that to log into an account, but they'd have to at least do a little bit of work to break the hash if they wanted to actually crack the password.

In other words, you were absolutely right to change all your passwords.

Passwords are in HTTPS requests, and should never be in HTTPS responses (especially not cacheable responses).

Only GET responses should be cached (not requests), so it seems unlikely a password would leak.

If users were discerning based on the inner workings of this organization, they should be switching their nameservers away from Cloudflare. Quantifying the number of domainnames CF has in their zones before and after this reported incident would be interesting. Sadly, I doubt there will be much change.

TL;DR: "Since this is just a sample, it is not correct to conclude that no passwords, credit cards, health records, social security numbers, or customer encryption keys were ever exposed."

My website gets 75,000,000 total requests per month through CloudFlare, so they estimate 6-11 leaks.

I am happy with the way CloudFlare have responded to this.

Tagging along on the top reply here, but does anyone else notice some serious HN gaming going on in this thread?

Accounts commenting that have made 10 comments in 5 years, people defending CF that, after going through their profiles, are clearly linked to CloudFlare or maybe even employees. Top relies suddenly being near the bottom and replaced by posts supporting CloudFlare with statements that don't match what was actually in the blog post.

Just seems really really fishy to me

I doubt anything more nefarious is happening than a flood of CF employees hitting the thread. Is that gaming?

It's, at the very least, disingenuous.

It's not gaming for a CF employee to offer the own insights, but it would be if they're doing it as part of a coordinated effort. Also, they should be upfront about the affiliation.

I'm defending them because they're an incredible value for the $200/month we pay them, they've handled it responsibly, and the real impact looks vanishingly small.

HN is very clearly gamed and I see more and more of this, not only in this thread.

We take this problem seriously and have years of experience with it. In our experience, "very clearly gamed" nearly always reduces to "seems to me, based on personal biases and incomplete information". When abuse really is going on, that's extremely important, but it is certainly fewer than 10%, probably fewer than 5%, and maybe fewer than 1% of the times we hear this. So we actually have two problems, unfortunately intertwined: (1) abuse and (2) this tendency to imagine it.

For this reason, please don't post unsubstantive allegations to HN. If you have evidence, or significant suspicious, email them to hn@ycombinator.com instead.

This is rich coming from an anonymous account created 69 days ago with 17 total comments (and 7 of them in this post).

Hey at least mine is an obvious throwaway. And I don't think 17 comments in two months is that bad compared to ten in 5 years

Let's not assume that accounts are thownaway just because they are made up of random characters ;)

> Tagging along on the top reply here

First, no, please don't do that. It's one of the worst patterns in active discussions.

Second, if anyone thinks they see gaming or abuse, they should let us know right away at hn@ycombinator.com so we can investigate. Please don't post about it in comments, for a couple reasons: (1) if there really is abuse going on, we need to know, and we don't see most comments; (2) most of us internet readers are orders of magnitude too quick to interpret our own cognitive biases as abuse by others (e.g. X seems obvious to me so anyone arguing ~X must be a shill). This places the threshold for useless, nasty arguments ('you're a shill. no, you're the shill') dangerously low. Combine that with the evil catnip power of all things meta and you get the most malignant strain of offtopicness there is, so we all have to be careful with it. And yes, genuine abuse does also exist. It's complicated that way.

We detached this subthread from https://news.ycombinator.com/item?id=13767122 and marked it off-topic.

:( My bad.

Reply on news.ycombinator.com