Heartbleed disclosure timeline: who knew what and when
smh.com.au
smh.com.au
We probably need a stronger web security system people can be on. Also some of the blame falls to the big companies, banks, telcos etc themselves, I mean who wants to report security flaws to Telcos in a world where they then turn around and instead of giving you a $15k bounty, send you to jail like Weeve. And ok that's not the best comparison because he went well beyond discovering and did exploit as well. But none the less, some of the big corporate world need to clue in and get more friendly with tech rather than hostile, because right now they don't have a very good reputation and so then tech leaves them out in the cold. Did anyone think about notifying telcos and banks much? or just other big tech companies.
It wouldn't surprise me if plenty of people who would have previously reported a vulnerability to the government or local CERT are no longer willing to do so.
Once there is reason to believe that the benefits outweigh the costs, they will disclose it in a way that doesn't expose their knowing about it beforehand.
You seem to think "a government" is a unified entity. If the NSA (for example) discovered Heartbleed I'd expect you would be correct.
If someone in the IRS (for example) found it I'd expect they would behave very differently.
At which point (I imagine), decisions would be made along the same lines as I described, the key questions being "can we keep it secret?" and "who has this capability?".
I tried to be nation-agnostic, as I imagine there may be a bit of difference in how a given nations' intelligence agency weighs the value of their offensive capabilities and the public defensive capabilities in light of who knows about the bug.
A collection of rational actors may, as a whole, behave irrationally - as defined as 'against their own best interest'. Tragedy of the commons is but one example.
Organized systems may easily have structures that create irrational and destructive behavior, and those structures can be stable enough so that for each separate employee attempting to change the system would be risky, disadvantageous and irrational.
Citation needed. :)
There are two schemes that assure irrational governmental behavior -- dictatorship, and democracy. For different reasons, of course.
> Not really surprising but many people seem to assume the government will be either irrationally benevolent or irrationally evil.
There's plenty of historical support for those views -- you know, evidence?
High-ranking intelligence officials will push for more power, more capabilities, and will scrap things if they become risky.
If there's somebody sufficiently powerful whose career would be threatened from some risk of exploiting the bug, that risk will be taken seriously.
If there's somebody sufficiently powerful whose career would benefit from the government letting people know about that bug, it'll happen. This sounds unlikely, but this isn't my area of expertise and if somebody wants to correct me please do so.
Congress will largely ignore it because it has nothing to do with their constituency, they're mostly technically illiterate, and because they're being lied to anyways.
...and so on.
Daniel Kahneman says this has been proven false in his book Thinking Fast and Slow.
It think it is wrong on two levels. First individual's "perceived best interest" is often very far off of their real interest. Second, individual, and not only a few edge cases, are often not behaving in their best interest, even if we consider it to be their "perceived best interest".
Kahneman listed the three attributes of humans modelized as "econs" in the most common economic model: They are rational, selfish and their taste do not change over time. All three are obviously wrong.
In the theoretical sense, I agree with that, but in a world of social interconnections, we need a meaningful framework and vocabulary for judging these sorts of things. While I believe that true objectivity is hard (and maybe impossible), I think there's a lot of social value in coming up with widely accepted definitions of rational vs. irrational, at least on a case-by-case basis.
It has meaning. It means "we have insufficient information" or "something is going on here that we don't know about or understand."
In any case, that's a false dichotomy. Governments are not uniform entities, and there's no point talking about them as if they are. Governments, like people, do some rational and some irrational things.
Also, banks only deal with money, whereas in certain locations and situations, lives potentially depend on Gmail, Facebook and Twitter being secure. (oppressive governments and such)
However, even though spouting a conspiracy theory is a faux pas here, I can't help but wonder if the "he who smelt it, dealt it" rule applies here. Lets say your country were to setup a network interconnecting major research institutions, etc. After its use takes off and it is obvious that everyone is going to be communicating over this new medium in a short amount of time, you see the value in keeping tabs on people. You decide that it is in your best interest to put backdoors into encryption algorithms in enterprise communication software. So you see there are these guys that have become the place that everyone is starting to go to find what they are looking for. This is a good place to be. You eventually get your hands on this also. It's a waste of time and energy to constantly be decrypting everyone's messages, so what the hell- let's put a backdoor in that also. Everything is going well. Wait... ok, we should have thought of that. Another country now knows about this vulnerability and it hasn't been publicized, which means they will start using it to spy on contractors that work for us. We'd better leak this information so everyone fixes their hole. Let's tell Google. We're already on good terms with them.
But next, the feature went live on the servers of more or less everyone, including Google and Yahoo and Amazon. People who employ and, presumably, well-compensate many experts in security and SSL implementations. Still, the code marched on, unnoticed, undisabled, deployed. How did that happen?
This suggests that first the NSA discovered it, then Neel Mehta, and finally Codenomicon. As other commenters have rightly suggested, the government, or at least an agency there of, uses information asymmetry as a weapon of intelligence on its users (aka citizens). In the case of private corporations, they have a more contractual duty to their users to ensure that these bugs are patched so that those with malevolent intentions can't harm their users.
In my opinion, a bug that has been discovered but not shared is just as bad as a weaponized biological agent. Each rely on secrecy to exploit a group that is unaware.
I am not saying it's impossible the NSA knew. But people throwing around that article as evidence is absolutely laughable.
What's funny about that is that the nginx bug is functionally identical to Heartbleed (modulo that it only worked on nginx).
“The Baffling Popularity of Randomized Linear Network Coding
As I write this chapter, I’m attending a computer science conference in San Jose, California. Earlier today, something interesting happened. I attended a session in which four different professors from four different universities presented their latest research. Surprisingly, all four presentations tackled the same narrow problem—information dissemination in networks—using the same narrow technique—randomized linear network coding. It was as if my research community woke up one morning and collectively and spontaneously decided to tackle the same esoteric problem.
This example of joint discovery surprised me, but it would not have surprised the science writer Steven Johnson. In his engaging 2010 book, Where Good Ideas Come From, Johnson explains that such “multiples” are frequent in the history of science. Consider the discovery of sunspots in 1611: As Johnson notes, four scientists, from four different countries, all identified the phenomenon during that same year. The first electrical battery? Invented twice in the mid-eighteenth century. Oxygen? Isolated independently in 1772 and 1774. In one study cited by Johnson, researchers from Columbia University found just shy of 150 different examples of prominent scientific breakthroughs made by multiple researchers at near the same time.”
Excerpt From: Newport, Cal. “So Good They Can't Ignore You: Why Skills Trump Passion in the Quest for Work You Love.”
His thesis on this is basically that these discoveries depend on a lot of other things occurring first and that once those things have occurred, anyone looking in the right place will see it.
With all of that said, I remain skeptical, at least in the Heartbleed case (they're just SOO close together). tptacek has more experience in these things than me of course, so I'll defer to his thoughts.
I think that still leaves open the question of how likely it is, or perhaps the question is better phrased as whether it is due to chance or not.
It is extremely unlikely to be due to chance in this case. It was out for years, but discovered independently within days of each other.
One possibility is that it was discovered independently many times, but we only find out about it after someone actually goes public. At that point, those finding it earlier either never wanted to mention publicly that they found it (or they already would have), or even if they want to, they would be admitting to finding it earlier but not disclosing, so they'd look bad.
That leaves others finding it slightly after the group that goes public - soon enough that it wasn't public yet, so they have evidence that they actually did find it independently. That would lead to exactly the result we see here, in fact. But this does imply that there were, very likely, multiple other groups that found this earlier but never went public.
The only other option is that it wasn't random, but some event led to both discoveries. Perhaps a hint was around, a new method of finding vulnerabilities, or anything at all that could nudge people's minds in that direction. That option is much less worrying, but very hard to prove.
Sorry, there's probably nothing interesting happening here, except for coincidence.
What I disagree with is that one event happening means that another event very similar to it is likely in a statistical sense.
Of course, it is an argument that supports that to some extent. But it is fairly weak support, when on the other hand statistics strongly imply the opposite.
Why? It makes perfect sense to me that, when one type of vulnerability is discovered, many more of same type will be discovered very soon thereafter. You have to consider that vulnerability discoveries don't happen in a vacuum. There's a near-infinite number of attack routes that one could investigate, but which one you're looking at now is a product of the environment you operate in.
For example, let's say you're investigating a web server. Then, some security researcher demonstrates a flaw in an image codec where even using "safe" memory copy functions in C leads to a vulnerability if tainted values are passed in for the size parameters. You think, "Hmm...I'm not decoding images, but web servers do copy memory. I should check to see if any memory copy operations are using tainted values." Bam! You discover Heartbleed...but do you honestly think you'd be the only researcher working on web servers that saw the image codec demo and made that connection? Unlikely.
I'm not arguing this is a coincidence. Just that if it was totally random, it would be very unlikely. So the plausible possibilities are (1) what you suggested, some common cause, or (2) that the discovery happened randomly multiple times but was only disclosed once.
Well, two years, the minimum number of years for which you can claim it was out for years. In order to avoid tricking yourself and your audience I'd stick to "a little over two years" rather than "years".
There were 749 days between a bugged version of OpenSSL being released and the flaw being discovered. There were 13 days between the independent discovery. That is still a small window compared to 749 days but not that small, and during that 749 days more and more people put bugged versions of OpenSSL into their infrastructure, which, I would assume, increases the chances that somebody discovers this flaw.
I would agree with you that it's not totally random -- of course it isn't, it's probably related to the state of the world changing over time. People adopting bugged versions is one factor, progress on security tools may be another, other publicized TLS bugs causing people to look harder at this area may be other, etc
> finding vulnerabilities, or anything at all that could nudge people's minds in that direction.
As somebody who has witnessed several cases of simultaneous discovery of security vulnerabilities, this is exactly how it happens. Some vaguely related event happens which causes multiple researchers to all start looking in the same place.
http://en.wikipedia.org/wiki/Multiple_discovery
I remember years ago reading about how some researchers were investigating a specific variation of this phenomenon, in which a number of independent and geographically dispersed labs would be, for example, trying to grow a particular crystal. For a number of years none were successful. Then within a matter of weeks of each other, all of the labs would independently discover the same technique that made it possible. This happened so close together that pure coincidence was incredibly unlikely.
Apparently this same phenomenon was occurring far more often than it should in theory (based on the probability of pure coincidences), which is what prompted the research into it.
Unfortunately, I don't know what the final outcome was. It would be amazing if anyone else remembers this and can point out the original source.
Personally I am inclined to believe that it is simply a matter of prior events causing multiple individuals to be thinking about the same things. If you have lots of people searching in the same locality, there's a much higher chance that they will all find the same things around the same time than if they are all operating in a truly random search space.
A more sinister explanation would be that evidence of exploitation helped focus attention by each team once it touched them. However, a colleague of Neel Mehta implies that this was an audit-driven discovery without regard to active exploitation (https://news.ycombinator.com/item?id=7558015).
That could still leave the possibility that news of Mehta's discovery leaked, as either a vague hint or as enough info to create larger scans, which then helped tip off the Codenomicon group.
Despite all the reasons for secrecy, non-disclosure, and protection of proprietary methods, I hope each discoverer eventually says more about the steps leading up to discovery.
They didn't find the bug. They benefited from a heads-up on the bug. Then they promoted a mistaken assumption about the bug, along with a gamified challenge site that was in fact a poor vehicle for investigating nginx+OpenSSL (how about, for instance, any decent debugger instead?).
Meanwhile, ops teams at big companies were in heated debates with security about whether keys needed to roll.
There are good people working at Cloudflare and I'm not part of any outrage battalion. I'm just not a fan of how they handled this particular incident.
I did what Willem did: I instrumented a small OpenSSL driver program and snapshotted memory. I did not go through the effort Jeremi Gosney and Willem and Thai and Ben Murphey went through to trace things through the code, but it was immediately apparent that there was more going on than the blog post Cloudflare wrote suggested.
More importantly, the whole thesis of that blog post is that RSA private keys are loaded into memory once and never moved again. But that's obviously not true: intermediates based on private key components are created during Montgomery multiplication, for instance.
The bigger problem is not that Cloudflare got things wrong. It's that they (a) marketed the wrong conclusion, and (b) put the conclusion to trial in a way that spent smart people's time unnecessarily.
============
We think the stealing private keys on most NGINX servers is at least extremely hard and, likely, impossible. Even with Apache, which we think may be slightly more vulnerable, and we do not use at CloudFlare, we believe the likelihood of private SSL keys being revealed with the Heartbleed vulnerability is very low. That’s about the only good news of the last week.
We want others to test our results so we created the Heartbleed Challenge. Aristotle struggled with the problem of disproving the existence of something that doesn’t exist. You can’t prove the negative, so through experimental results we will never be absolutely sure there’s not a condition we haven’t tested. However, the more eyes we get on the problem, the more confident we will be that, in spite of a number of other ways the Heartbleed vulnerability was extremely bad, we may have gotten lucky and been spared the worst of the potential consequences.
That said, we’re proceeding assuming the worst. With respect to private keys held by CloudFlare, we patched the vulnerability before the public had knowledge of the vulnerability, making it unlikely that attackers were able to obtain private keys. Still, to be safe, as outlined at the beginning of this post, we are executing on a plan to reissue and revoke potentially affected certificates, including the cloudflare.com certificate.
Vulnerabilities like this one are challenging because people have imperfect information about the risks they pose. It is important that the community works together to identify the real risks and work towards a safer Internet. We’ll monitor the results on the Heartbleed Challenge and immediately publicize results that challenge any of the above.
============
Which is exactly what we did. To be clear, we were wrong. Our mistaken assumption was focusing on the private key itself and not focusing enough on the exponents that are used to generate the key -- which is what the researchers who solved the challenge were able to obtain. As the wishy-washy conclusion above hopefully makes clear, even when we said it was hard to get the private keys, we were very uncertain and uncomfortable with that conclusion. That's why we posted the challenge. What the challenge did was answer the question definitively: you can get private SSL keys. Knowing that has been valuable for us in deciding to accelerate reissue/revocation process for all the certs we manage on behalf of our customers. Remember: at our scale, revoking hundreds of thousands of certs risked breaking our CA partners' infrastructure, so it wasn't without a cost. Knowing the risk is higher than we originally thought accelerated our efforts which will be complete in the next 48 hours. And, beyond CloudFlare, our hope is knowing the risk is real and proven has benefited other organizations as well.
Your "Cloudflare Challenge" (that's what you called it) was not a particularly useful way to answer the question posed by Heartbleed. What you want to know is, "is there private key material in heap memory?", or, to make things even simpler, "are our assumptions about how key material hits heap memory accurate?". The correct way to answer this question is to instrument and analyze an OpenSSL/nginx runtime, not to create and market a treasure hunt for an undisclosed private key on a single site.
You employ smart people. You could have done better than this challenge. Instead, what seems to have happened is that your company got inserted into the middle of a story it had little to do with (correct me if I'm mistaken and unaware of something your team did to research Heartbleed), and, with that spotlight shining, actively marketed a harmful false conclusion about the bug while also bidding for the spare cycles of other people who might have been more effective doing something other than poking at your server. (For what it's worth, I don't think for a moment that you did either of those things intentionally).
I think if you did either of those two things differently --- either didn't publicly go out on a limb suggesting that you thought keys would be hard (or, as Bruce Schneier seems to have read from your blog post, "next to impossible") to recover keys, or didn't set up the game site while doing it --- I wouldn't be moved to comment.
Like I said, not after pelts. Just, if we're putting the Cloudflare response up for questioning, I have some issues to point out.
It seems like you can just look at the code and see how that's not the case. But I might be wrong, too.
It is a tricky thing, being in the center of a critical vulnerability disclosure story.
(If you really want to know the issue, google his name with "dns", but it's not really relevant to rehash his mistakes here, except for the fact that in his choice of words on this thread ("They didn't find the bug. They benefited from a heads-up on the bug. Then they promoted a mistaken assumption about the bug") are a screamingly obvious reference to his old mistakes.)
He's not being hypocritical; he's speaking from the voice of experience and having had things blow up in his face. He's trying to stop other people from making the same mistakes he did. When the guy with the big burn mark on his face talks to your chemistry class about the importance of lab safety, you should LISTEN.
"Experience is a dear teacher but fools will learn at no other." Sometimes your elders know what they are talking about.
(Now to re-bandage that wound. Another big piece of advice from another elder: people change and mature, yourself included. You should get over other people's mistakes.)
The challenge result instantly convinced a lot of people who still had doubts, because of the mixed messages elsewhere. The sideshow drama, much like the catchy name 'heartbleed' before it, worked perfectly for its intended purpose.
It really pissed me off because I developed the ability to get keys long before you even wrote the post. I commented about it here several days before you wrote the post[1]. After your blog post, I was accused of fabricating the entire story because you said keys were unobtainable. I cannot, legally, release code without opening myself up to legal ramifications for reasons I won't get into here. Then, after your post, people I've known for a long time accused me of making the entire thing up for a "shot at glory." Meanwhile, I had to explain that your blog post was not definitive to multiple people who were reassured by the false security.
I brought your company's marketing strategy up with you before on HN. Remember when your company jumped on nytimes.com getting owned at the registrar[0]? And you wrote a hurried postmortem of events (I'm assuming, from the typos) without even consulting the affected vendor, then went so far as to speculate on what happened at the affected vendor, and made sure your "postmortem" got on top of HN first?
> An e-mail obtained by Matther [sic] Key, an independant [sic] journalist, indicates that the hackers used a MelbourneIT domain reseller account as part of the attack. While we are only speculating at this point, it's possible that there was a vulnerability in Melbourne IT's reseller systems that allowed a privilege escalation.
You replied here on HN with "no good deed goes unpunished" after I expressed my displeasure with your company's behavior in that scenario, and you didn't really address my points. You basically pointed to the CTO of NYT's praise of your company as evidence enough that you did the right thing. You took advantage of the marketing opportunity (which is fine, I don't fault you) at the expense of allowing the affected vendor to even draft a postmortem or contact customers in a timely fashion (I fault you for that).
We get it. You want to position CloudFlare as a superhero company, capable of fixing the Internet problems that the rest of us cannot handle. However, your marketing strategy has alienated me from ever using your services, and I am not alone in that opinion. Please, rethink that strategy. Focus on the product instead of fixing what you perceive to be a broken Internet that only you can fix. That's the obvious tone I get from your marketing and choices of venue.
[0]: http://blog.cloudflare.com/details-behind-todays-internet-ha...
I'm not looking to collect pelts. I just think there was a better way to address the question of private key exposure than "The Cloudflare Challenge". In the future, maybe we can address serious questions with engineering instead of marketing stunts; how about the "let's all work to instrument OpenSSL" challenge?
I think we're in violent agreement here.
I wonder if Facebook found and fixed the bug locally beforehand? Makes me think of all the OSS libraries we use, without bothering to do much more in the way of static-analysis, etc besides simply making sure they compile.
I worry that Amazon wasn't given prior notice according to the timeline.
Given that Adam Langley works with the chrome folks, i suspect it did not end up needing it in the end :)
It would mitigate the possibility of it leaking and getting exploited by someone.
Many vendors will not just simply compile a new version of a library from upstream source and just throw it on their machines. They depend on a tested release from their distribution maintainer, or something along those lines.
Also many vendors aren't prepared to do a simple upgrade: some may have customization they need to forward-port and test. Or perhaps they'd prefer to backport the fix to their older version.
So basically, your "everyone gets info at once" means that blackhats can get the information and exploit it almost immediately, while the good guys scramble to -- much more slowly -- patch their systems.
2. Ask to speak to a representative of the people developing the thing on the telephone.
3. Between the two of you, figure out who to tell next.
4. Realize that other people may find the bug, too. You have some time, but not infinite time.
Now, if you want a bug bounty, you have to file a report and wait a certain amount of time before you are allowed to disclose.
But hackers overtaking banking for example might be a real risk.
This is an overly optimistic account of who knew what when.
And, how can we protect? Patch? Yes, that's necessary, but not sufficient. We have so many other protocols we rely on that are fragile & intercepted. Self-censorship sucks, and can't possible protect everything either. What to do?
For most people reading this actual article, I think they will come away with the impression that it's a complete account.
>Ever since the "Heartbleed" flaw in encryption protocol OpenSSL was made public on April 7 in the US there have been various questions about who knew what and when.
>Fairfax Media has spoken to various people and groups involved and has compiled the below timeline.
>If you have further information or corrections - especially information about what occurred prior to March 21 at Google - please email the author: bgrubb@fairfaxmedia.com.au. Click here for his PGP key.
I mean, when the NSA & other actors do NOT submit their data to this guy, they can say, it's complete now?
It just stinks to me like it's complacency. Just change your passwords & patch, & then don't worry, share everything again, it's private. In 2014, I don't think we're safe & private anymore at all. I can take the downvotes. I don't like it either.