Hacker News Reader

Comment by tptacek | Hacker News Reader

It happens routinely. Allow me to illustrate: on a pentest a couple years ago, Vitaly McLain from our Chicago team found that if you accidentally slipped a NUL into a header that nginx reverse-proxied, nginx would short-copy (it used strncpy) and reveal server memory. We reported to our client, and in the time it took for them to OK our upstream report --- a matter of hours --- someone else reported the same bug.

What's funny about that is that the nginx bug is functionally identical to Heartbleed (modulo that it only worked on nginx).

View on HN

I've been reading So Good They Can't Ignore You and there's an interesting section on this phenomenon:

“The Baffling Popularity of Randomized Linear Network Coding

As I write this chapter, I’m attending a computer science conference in San Jose, California. Earlier today, something interesting happened. I attended a session in which four different professors from four different universities presented their latest research. Surprisingly, all four presentations tackled the same narrow problem—information dissemination in networks—using the same narrow technique—randomized linear network coding. It was as if my research community woke up one morning and collectively and spontaneously decided to tackle the same esoteric problem.

This example of joint discovery surprised me, but it would not have surprised the science writer Steven Johnson. In his engaging 2010 book, Where Good Ideas Come From, Johnson explains that such “multiples” are frequent in the history of science. Consider the discovery of sunspots in 1611: As Johnson notes, four scientists, from four different countries, all identified the phenomenon during that same year. The first electrical battery? Invented twice in the mid-eighteenth century. Oxygen? Isolated independently in 1772 and 1774. In one study cited by Johnson, researchers from Columbia University found just shy of 150 different examples of prominent scientific breakthroughs made by multiple researchers at near the same time.”

Excerpt From: Newport, Cal. “So Good They Can't Ignore You: Why Skills Trump Passion in the Quest for Work You Love.”

His thesis on this is basically that these discoveries depend on a lot of other things occurring first and that once those things have occurred, anyone looking in the right place will see it.

With all of that said, I remain skeptical, at least in the Heartbleed case (they're just SOO close together). tptacek has more experience in these things than me of course, so I'll defer to his thoughts.

> It happens routinely.

I think that still leaves open the question of how likely it is, or perhaps the question is better phrased as whether it is due to chance or not.

It is extremely unlikely to be due to chance in this case. It was out for years, but discovered independently within days of each other.

One possibility is that it was discovered independently many times, but we only find out about it after someone actually goes public. At that point, those finding it earlier either never wanted to mention publicly that they found it (or they already would have), or even if they want to, they would be admitting to finding it earlier but not disclosing, so they'd look bad.

That leaves others finding it slightly after the group that goes public - soon enough that it wasn't public yet, so they have evidence that they actually did find it independently. That would lead to exactly the result we see here, in fact. But this does imply that there were, very likely, multiple other groups that found this earlier but never went public.

The only other option is that it wasn't random, but some event led to both discoveries. Perhaps a hint was around, a new method of finding vulnerabilities, or anything at all that could nudge people's minds in that direction. That option is much less worrying, but very hard to prove.

I think if you reread my comment closely, you'll see that it describes a scenario that is spookily similar to Heartbleed: same impact, same latent bug (presumably in the code for many years, maybe even longer than TLS heartbeats), and co-discovery not in a matter of days but hours.

Sorry, there's probably nothing interesting happening here, except for coincidence.

Yes, it is spookily similar, I agree 100% (I did read it carefully ;)

What I disagree with is that one event happening means that another event very similar to it is likely in a statistical sense.

Of course, it is an argument that supports that to some extent. But it is fairly weak support, when on the other hand statistics strongly imply the opposite.

> What I disagree with is that one event happening means that another event very similar to it is likely in a statistical sense.

Why? It makes perfect sense to me that, when one type of vulnerability is discovered, many more of same type will be discovered very soon thereafter. You have to consider that vulnerability discoveries don't happen in a vacuum. There's a near-infinite number of attack routes that one could investigate, but which one you're looking at now is a product of the environment you operate in.

For example, let's say you're investigating a web server. Then, some security researcher demonstrates a flaw in an image codec where even using "safe" memory copy functions in C leads to a vulnerability if tainted values are passed in for the size parameters. You think, "Hmm...I'm not decoding images, but web servers do copy memory. I should check to see if any memory copy operations are using tainted values." Bam! You discover Heartbleed...but do you honestly think you'd be the only researcher working on web servers that saw the image codec demo and made that connection? Unlikely.

Certainly, yes - that would mean that this is not a coincidence.

I'm not arguing this is a coincidence. Just that if it was totally random, it would be very unlikely. So the plausible possibilities are (1) what you suggested, some common cause, or (2) that the discovery happened randomly multiple times but was only disclosed once.

> It was out for years, but discovered independently within days of each other.

Well, two years, the minimum number of years for which you can claim it was out for years. In order to avoid tricking yourself and your audience I'd stick to "a little over two years" rather than "years".

There were 749 days between a bugged version of OpenSSL being released and the flaw being discovered. There were 13 days between the independent discovery. That is still a small window compared to 749 days but not that small, and during that 749 days more and more people put bugged versions of OpenSSL into their infrastructure, which, I would assume, increases the chances that somebody discovers this flaw.

I would agree with you that it's not totally random -- of course it isn't, it's probably related to the state of the world changing over time. People adopting bugged versions is one factor, progress on security tools may be another, other publicized TLS bugs causing people to look harder at this area may be other, etc

> The only other option is that it wasn't random, but some event led to both discoveries. Perhaps a hint was around, a new method of

> finding vulnerabilities, or anything at all that could nudge people's minds in that direction.

As somebody who has witnessed several cases of simultaneous discovery of security vulnerabilities, this is exactly how it happens. Some vaguely related event happens which causes multiple researchers to all start looking in the same place.

An aside - was it really strncpy? That function has such baffling behavior (it creates unterminated strings on overflow — contrary to its str* name) that I'm surprised it isn't banned by every static checker in existence.

Reply on news.ycombinator.com