jgrahamc: can you list which public caches you worked with to attempt to address this? It does not inspire confidence when even google is still showing obvious results
I agree it's troubling that Google is taking so long. We were working with them to coordinate disclosure after their caches were cleared. While I am thankful to the Project Zero team for their informing us of the issue quickly, I'm troubled that they went ahead with disclosure before Google crawl team could complete the refresh of their own cache. We have continued to escalate this within Google to get the crawl team to prioritize the clearing of their caches as that is the highest priority remaining remediation step.
view-source:http://cc.bingj.com/cache.aspx?q=&d=4857656909960944&w=rj9cg...
view-source:http://cc.bingj.com/cache.aspx?q=&d=4901023173710126&w=n3mEZ...
view-source:http://cc.bingj.com/cache.aspx?q=&d=4558611265887320&w=urwoW...
view-source:http://cc.bingj.com/cache.aspx?q=&d=4592983872701813&w=Ghwdd...
view-source:http://cc.bingj.com/cache.aspx?q=&d=4997243316273666&w=wdpFH...
Not as simple as you thought?
Thousands of years from now, when biological life on this planet is all but extinct and superintelligent AI evolving at incomprehensible rates roam the planet, taviso will still be finding 0-days impacting billions of machines on an hourly basis.
Be glad that Google is employing him and not some random intelligence agency.
However, I am always wondering: are they really globally unique in their work and skill? So that they are really the ones finding all the security holes before anyone else does because they are just so much better (and/or with better infrastructure) than anyone else? Or is it more likely that on a global scale there are other teams who at least come close regarding skill and resources, but who are employed by actors less willing to share what they found?
I really do hope Tavis is a once-in-a-lifetime genius when it comes to vulnerability research!
If I were just casually googling two weeks ago and came across a leaked cloudflare session in the middle of my search results I think I would have vomited all over my desk immediately. Dude must have been sweating bullets and trembling as he reached out on twitter for a contact, not knowing yet how bad this was or for just how long it's been going on.
http://208.71.46.190/search/srpcache?p=yOGHqpbGWiXrRAIqLM87w...
http://cc.bingj.com/cache.aspx?q=%22X-SSL-Server-IP+104.16.5...
http://news.bbc.co.uk/2/hi/business/8174763.stm
I know the search I performed now on Yahoo states "Powered by Bing™" at the bottom.
<!-- fe072.syc.search.gq1.yahoo.com Sat Feb 25 03:58:27 UTC 2017 -->
Given they are identical results it's pretty clear it must be a shared index I suppose, that or the leaked memory was cached.
Yahoo was never really a search company (even its founding, it was a "directory", not a "search"). Sure, they pretended fairly well from 2004ish (following their move off Google results) to 2009 (when they did the Bing deal), but the company never really nailed search or more importantly search monetization despite acquiring one of the first great search engines (Altavista) and the actual inventor of the tech Google stole for its cash cow Adwords (Overture).
And some sketchy internal variables: `log_only_china`, `http_not_in_china`, `baidu_dns_test`, and `better_tor`.
And monoculture is the elephant in the room most pretend not to see. The current engineering ideology (it is ideology, not technology) of sycophancy towards big and rich companies, and popular software stacks, is sickening.
I've never seen anyone suggest it, I suppose It cannot or should not be done for some reason?
http://todayilearned.co.uk/2012/12/04/what-the-british-say-v...
The way I see it, time given by GZero was sufficient to close the loophole, it was not meant to give them chance to clear caches world-wide. They have a PR disaster on their hands, but blaming Google won't help with it.
20 hours since this post and these entries are still up ...
You can also click on "parent", and repeat as necessary.
HTTP/1.1
Host gateway.discord.ggHe discovered one of the worst private information leaks in the history of the internet, and for that, he won the highest reward in their bug bounty: a Cloudflare t-shirt.
They also tried to delay disclosure and wouldn't send him drafts of their disclosure blog post, which, when finally published, significantly downplayed the impact of the leak.
Now, here's the CEO of Cloudflare making it sound like Google was somehow being uncooperative, and also claiming that there's no more leaked private information in the Bing caches.
Wrong and wrong. I'd be annoyed, too.
--
Read the full timeline here: https://bugs.chromium.org/p/project-zero/issues/detail?id=11...
I can see a whole team at Cloudflare panicking, trying to solve the issue, trying to communicate with big crawlers trying to evict all of the bad cache they have while trying to craft a blogpost that would save them from a PR catastrophe.
All the while Taviso is just becoming more and more aggressive to get the story out there. 6 freaking days.
short timeline for disclosures are not fun.
Two questions came to mind: "how do we clean up search engine caches?" (Tavis helped with Google), and "has anyone actively exploited this in the past?"
Internally, I prioritized clean up because we knew that this would become public at some point and I felt we had a duty of care to clean up the mess to protect people.
Has this question been answered yet?
Wouldn't your team now even have to decide how to deal with this even after some specific well known caches have been cleared? I mean there's no guarantee that someone may not have collected all this data and use it to target those cloudflare customer sites. Are you planning to ask all your customers to reset all their access credentials and other secrets?
There are very good reasons to enforce clear rules like this.
Cloudbleed obviously falls into the second category.
Legally, there's nothing stopping researchers from simply publishing a vulnerability as soon as they find it. The fact that they give the vendor a heads-up at all is a courtesy to the vendor and to their clients.
It is the norm, and it is called responsible disclosure. You're trying to do the less harm, and the less harm is a combination between giving some time to the developers to develop a fix and getting the news out there for customers and customers of customers to be aware of the issue.
I would also advise you notify your cloud-based services' customers how they might be affected (yes really), trust erosion tends to be contagious.
I think you have misunderstood the issue. Just because YOU did not use those services does not mean your data was not leaked. It means that other peoples data was not leaked on YOUR site, but YOUR data could be leaked on other sites that were using these services.
If this part is true, they're not vulnerable. Only data that was sent to CloudFlare's nginx proxy could have leaked, so if they only proxy their static content, then that's the only content that would leak.
The rest of their comment gives the wrong impression though, yeah.
The way it worked, the bug also leaked data sent by the visitors of the these "static sites": IP addresses, cookies, visited pages etc.
Don't use CF, and after seeing behavior like this, don't think I will.
Before Let's Encrypt is available to public use (beta), CF provided "MITM" https for everyone: just use CF and they can issue you a certificate and server https for you. So I tried that with my personal website.
But then I found out that they replace a lot of my HTML, resulting mixed content on the https version they served. This is the support ticket I filed with them:
On wang.yuxuan.org, the css file is served as:
<link rel="stylesheet" title="Default" href="inc/style.css" type="text/css" />
Via cloudflare, it becomes:
<link rel="stylesheet" title="Default" href="http://wang.yuxuan.org/inc/A.style.css.pagespeed.cf.5Dzr782jVo.css" type="text/css"/>
This won't work with your free https, as it's mixed content.
Please change it from http:// to //. Thanks.
There should be more similar cases.
Luckily I have Let's Encrypt now and no longer need them.
This led to Cloudflare refusing to implement support for Google Authenticator for 4 years.
Also, the notion that the CEO of an internet company would have a "beef with Google" is pretty funny.
Bugs happen to us all; how you deal with this is what counts, and wilful, blatant lying in a transparent attempt to deflect blame from where it belongs (Cloudflare) onto the team that saved your bacon?
I've recommended Cloudflare in the past, and I was planning, with some reservations, to continue to do so even after disclosure of this issue. But seeing this comment? I don't see how I can continue.
(For the sake of maximum clarity: I take issue: 1) with the attempt at suggesting the main issue is in clearing caches, not on the leak itself. It doesn't matter how fast you close the barn door after the horse is gone and the barn has burned down. 2) With the blatantly false claim that non-Google caches have been cleared, or were faster to clear than Google's. Cloudflare should know, better than anyone, the massive scope of this leak, and the fact that NO search engine's cache has or could be cleared of this leak. If you find yourself in a situation so bad you feel like you need to misdirect attention to someone else, and it turns out no one else is actually doing anything so you have to like about that...maybe you should just shut up and stop digging?)
Google has absolutely no obligation to clean up after your mess.
You should be grateful for any help they and other search engines give you.
But I still find it troubling. Is it their mess? No. Does it affect a lot of people negatively - yes. I expect Google to clean this up because they're decent human beings. It's troubling because it's not just CloudFare's mess at this point.
It reminds me of the humorous response to "Am I my brother's keeper?", which is "You're your brother's brother"
I view leaving up the cached copy of leaked data as being a jerk move - not towards CloudFare, but to anyone whose data was leaked.
This is an opportunity for Google to show what they do with rather sensitive data leaks - do they leave them up or scrub them?
Had damage from the leak been aleady done (to those whose data it was)? Probably. Even taking that into account, I think the Google search comes off as a jerk in this situation.
This is not the case; it is not obvious, trivial, or easy to delete the leaked data. It is not simple to find it all. This is not like they are being given a URL and being asked to clear the cached version of it; they are being asked to search through millions of pages for possibly leaked content.
I will be migrating away from your service first thing Monday. I will not use you services again and will ensure that my clients and colleagues are informed of you horrific business practices now and in the future.
https://arxiv.org/pdf/1105.2576.pdf
(disclaimer: co-author)
It sounded like they (cf) were under a lot of pressure to disclose ASAP from project zero and their 7 day requirement...
If you are using the same attitude as you use in this comment, with their team, i'm pretty sure they will be thrilled to keep aside all their regular work and help you out cleaning up a enormous mess created by a bug in your service.
I'm no longer using CF for my own projects, but you've just cemented my decision that none of my clients will either.