Hacker News Reader

Comment by forgotAgain | Hacker News Reader

A different CloudFlare post on HN https://news.ycombinator.com/item?id=7572666 claimed that they had a fix for the HeartBleed bug 12 days ago.

At CloudFlare, we received early warning of the Heartbleed vulnerability and patched our systems 12 days ago.

I commented on that post that the date of discovery was 7 days ago http://www.vocativ.com/tech/hacking/behind-scenes-crazy-72-h...

For whatever reason that HN post was deleted and resubmitted so it now has no comments on it. https://news.ycombinator.com/item?id=7572796

Was the bug discovered 12 days ago or 7?

View on HN

The bug was discovered by whitehat researchers approximately 12 days ago. It was publicly announced 5 days ago. We got early word of it from the researchers who initially discovered it, allowing us to patch our systems and ensure all sites behind CloudFlare were not vulnerable. However, we have no way of knowing how long blackhats may have had it. It had been present in the OpenSSL software for the last 2+ years. Therefore we're trying to get an informed sense of the security risks. Hence our work attempting to use the vulnerability to recover SSL private keys and, as announced today, the CloudFlare Challenge.

I'm disgusted they chose to share it with you early and not the major Linux distros...

I have not seen any comment by the security researchers about why they choose to disclose to some organizations (such as cloud flare) earlier than the general public or the distributions. Perhaps, they felt the need to have some organizations with very large OpenSSL installations test the patch prior to make sure it worked and did not cause any unexpected problems?

That works up until it doesn't. Telling n people can work great, telling n+1 people may be complete failure, worse than telling none at all.

It seems (barring evidence saying otherwise) that CloudFlare managed to keep it secret while working on it, so they were proper to have in the n.

There's no way to share such a bug with the major Linux distros and let them deploy a fix to users without making it public at the same time. Even assuming that the distros commit to handle the fix submission and silently repackage openssl (which they don't always do, depending on their policy), the word would get out minutes after it's pushed to the update servers.

So telling major Linux distros == telling the public. At this point, you have to decide whether you want to forewarn big hosters handling millions of sites and billions of visits like Cloudflare, Google, AWS, etc., or not. I don't think there's a good universal answer to this.

Not so; companies like RHEL understand the importance of disclosure timelines and won't leak it early.

The importance of telling large distros doesn't lie in them immediately releasing a fix; it lies in them being able to prepare a package with the fix before the announcement, and then exactly when the announcement happens, they can publish the package (and possibly do something to make it propagate faster to their distribution servers)

As is, when Heartbleed was announced, many distros took an hour or significantly more to offer a fixed package. Proof-of-concept exploits were also made in that time. That was a dangerous situation.

I fully expect critical vulnerabilities that are "responsibly disclosed" to be reported to major distros so that packages can be prepared, but not released, in advance; furthermore, it allows people to be ready when it's announced at an agreed-upon date so that the packages can be pushed to live.

I'd actually be okay with a system where smaller distros which use similar packaging formats to larger ones are alerted with "There is an exploit. We will publish a fixed package that will likely be compatible with your distro on DATE. Be awake then to make sure these changes go live quickly, not when one key dude wakes up in 5 hours".

Sorry for the long-winded comment. What I really wanted to do is just explain "no way to share ... deploy a fix ... without making it public" is not the reason for sharing. The reason for sharing is so that the fix can be deployed more quickly when it is deployed.

It doesn't matter if they show "responsibly disclosed" all their repos are publicly available to see, it takes one person looking through commits then going wtf is this followed by a quick look at the code and a blog post to make this a wildfire.

I think the notion is the distro security team codes and tests a patch, but doesn't commit the code to public repos or release the patch publicly until an agreed-upon disclosure date.

Not necessarily that the distro security team codes the patch even. In most cases, upstream (e.g. openssl here) should have an official patch/commit that is private, but is given to these trusted distros. The security team only has to create a package with the upstream patch.

Other than that, yes, that's exactly the notion.

Every major linux distro has a procedure in place for discreetly preparing updates for pre-disclosure security flaws.

If this is really the case, there is no excuse.

It only takes one person on any of these teams to leak the details and the cat is out of the bag, and then the scramble is on.

Considering the circumstances, it appears that the process that was followed with regards to the release was as flawless as it could have been.

You would think that they could easily work on in advance of incidents who they can trust to with early information. I'd be amazed if Canonical (Ubuntu), Redhat (RHEL) and The Attachmate Group (SUSE) wouldn't show discreet discretion. I don't know about Debian and similar projects, but you would think they could determine this in advance.

I haven't been involved in distro security for a few years, but all this coordination used to happen via a mailing list. Organizations (distro maintainers, OS vendors, security people representing some of the larger/more security sensitive open source projects, etc) would need to apply to be on the list. They'd need to document who would have access to the sensitive materials posted, what their procedure would be for handling, etc. Impact assessment, disclosure timelines, CVE assignments from MITRE, attributions, etc etc would all be coordinated on this list. Fixes would not be pushed to public VCS systems or package repositories before the agreed-upon disclosure date.

AFAICT, none of this happened here. A very small number of organizations was told in advance, but nobody knows what the criteria were to get on this special advance notice list. Given how completely off-guard some really big organizations were caught (yahoo, for instance; all the linux distros, etc), this could have been handled a lot better.

There is just no easy way to handle this and someone had to make the decision on who got what.

The reality is that the open source community isn't vetted like an intelligence agency when it comes to holding secrets to their vest. It only takes one person in all those OSS communities to leak to the press about something of this magnitude and then the result could be even worse. The fact that this was kept under wraps for 12 days (that we know) is a testament to the folks who made the decision whom to inform.

The fix wasn't a massive one. Even without giving details they could have let the distros know that an emergency release of OpenSSL was coming.

No, it works like this: everyone gets their fix ready hush hush, and on an agreed date it's made public, and vendors hit the "publish" button more or less simlultaneously.

Except that it doesn't work like this. The major Linux vendors are too decentralized for this to work.

Except that it does work like this on a regular basis. It's not just something that sounds nice in theory. Distros and other major software vendors regularly coordinate disclosure. Have there been failures of the process? Sure, but that's the nature of secret keeping. The advantages of coordinated release far outweigh the risk of occasional mistakes, since the latter simply leaves people in the same position as they'd have been without any coordination (i.e. the exact same position that the distros were in with heartbleed).

I'm sure this bug was discovered at least 1 month ago. I would not trust my keys are safe even if using cloudflare.

From the related blog post: http://blog.cloudflare.com/answering-the-critical-question-c...

"While we believe it is unlikely that private key data was exposed, we are proceeding with an abundance of caution. We’ve begun the process of reissuing and revoking the keys CloudFlare manages on behalf of our customers. In order to ensure that we don’t overburden the certificate authority resources, we are staging this process. We expect that it will be complete by early next week."

If it was a month ago that was a very long period of time to sit on something so devastating.

Reply on news.ycombinator.com