Lessons from the Debian/OpenSSL Fiasco (2008) | Hacker News Reader

Lessons from the Debian/OpenSSL Fiasco (2008)

research.swtch.com

59 pointspgn67425 comments

I recently spend some time looking into this bug again, because I develop a tool to detect known cryptographic vulnerabilities in public keys called badkeys - https://badkeys.info/ if you're interested.

Some notable things I learned:

* This affects both OpenSSL and OpenSSH, but the keys are different. I.e. you have a set of vulnerable OpenSSH keys and a set of vulnerable OpenSSL keys. But the key format is the same, yet most of the tools to detect just look for either of these. I found a TLS certificate created with a vulnerable key generated by OpenSSH.

* It was "conventional wisdom" that ECDSA was unaffected because some sources said that OpenSSL version did not support ECDSA. However that was wrong, you can generate ECDSA keys with that old version.

Generally it seems a lot of the detection tools are incomplete. E.g. github seems to block some vulnerable keys, but only a subset.

You also wrote a really great article about it! In German: https://www.golem.de/news/sicherheitsluecke-in-druckern-uebe...

Every time this gets brought up, people forget that the patch had been sent to the openssl mailing list and someone said that it looked fine.

But here we have all the proponents of "distributions should never do any patch (and thus leave all the security issues open)". But they live in a fantasy world where all upstream authors reply within 3 minutes, fix issues within 30 minutes and of course backport the fix.

the doubly weird thing is that OpenSSL already had a ifdef-- -DPURIFY -- that did what the packager desired.

Did it have the same security problem?

Nope, it is the thing the debian developer intended, without introducing any problems-- and was at the time. It's a bit opaquely named (it's named after an earlier tool, 'purify' that protested some of the same things valgrind did) which is presumably why the debian developer was unaware of it.

I believe most distros ship with it on these days because without it you can't really use valgrind on programs that use openssl. (Suppressions don't really work because the uninitialized data taints all downstream users of openssl randomness)

Also, after it blew up people said "why did you mail to -dev? That's where users post to and nobody has time with that, we have a special wizards list where the devs hang out".

Which says a lot. With better communication, from everyone involved, this wouldn't have happened the way it did.

Discussed at the time:

Lessons from the Debian/OpenSSL Fiasco - https://news.ycombinator.com/item?id=196035 - May 2008 (2 comments)

> Try not to write clever code. Try to write well-organized code.

Decades ago someone wrote an empty loop to do "something" and it looped for a fixed number of times. No one knew why. But seemed that loop depended upon the frequency of the CPU. It was kind of a sleep (I forgot most of the details) that was needed for some reason. When the system was upgraded, things stated breaking.

That statement should be a tattoo on everyone's hand :)

Busy loops for delays used to be common. That is why some old PCs had a "turbo mode" switch. When on, the system ran at full speed. When off, it slowed down to match timing of old games etc.

On a DOS PC, a better way to delay would be to count timer interrupts.

I looked at the list of take-aways and one big important take away was missing. Testing. If a SSL depends on generating millions of unique keys then there should be an existing test somewhere the ensures that SSL does this before the release into production. The test would have caught the initialization and send the code back for re-work, clarification.

It is extremely difficult to tell a working and subtly broken RNG apart through testing. Imagine an RNG that repeatedly ran this code:

  date +%s.%N | sha256sum

The output will never recur no matter how many times you run the test, so long as it takes at least a nanosecond. It'll pass all statistical tests of randomness.

And it's completely insecure - just guess the time and you know the output.

True, but that wasn't the case here. If there were only "32,767 possible SSH keys" you wouldn't need to test very many before you'd get a collision. Such a test would have detected this issue and any similar issue that produces key in such a small set.

I don't quite understand why would Debian keep their own forks of software and don't even attempt to upstream the changes. Is it just security backports to ensure "stability" by not actually updating the packages, or do they have other reasons for the changes?

Where do you get “don’t even attempt to upstream the changes” from? They absolutely try to in the common case, from what I’ve seen. Every delta from upstream carries a cost, and no one knows that better than package maintainers.

Wonder if any of the lessons got learned? We, as an industry, are absolutely terrible at this, but hope springs eternal...

> Wonder if any of the lessons got learned?

Debian patches too much and can't be trusted is the lesson I learnt.

They haven't changed their patching policy since so I refuse to use any Debian derivative and advise everyone to do so and stick to distributions which stay as close to vanilla as possible.

Similarly I also think that anything pretending to provide LTS while freezing software versions is misguided and mostly lying to you and you are one incorrectly backported patch away from distaster.

> Debian patches too much

It does. This is partly because of The Debian Guarantee: I think that's why they delivered a castrated version of FFMPEG.

If upstream is deemed non-free, then they should just put the package in non-free, and delete it from main. Mangling upstream so they can squeeze it into main is worse than pointless.

> anything pretending to provide LTS while freezing software versions is misguided

How would you define LTS?

LTS is making only the breaking changes that I want, of course. /s

It's the complete opposite.

The lesson that OpenSSL code is terrible was learnt much harder way in 2014 with Heartbleed, and lead to LibreSSL existence.

No, generally there is no respect for the complexity of libraries. There is a reason why library authors themselves are reluctant to do many or broad changes.

But first the sysadmin culture (which I generally like, but not in this aspect) taught us that anyone can modify anything to make it work for him.

Now the equity culture teaches us that anyone is equal and has a right to do any modifications. Correctness does not matter, and if you insist, you are a class traitor.

Projects that deviate from upstream with this attitude have frequent issues.

Your tangent into right wing politics aside, everybody absolutely has a right to modify openssl and run it modified, it's in the license. Doesn't make every instance of it a good idea, but that right exists.

Short answer: no.

The reality of a distribution like Debian is that it needs to make patches to code to make software work in a holistic distribution, so a strict "no-patches-allowed" policy just isn't workable. The best you can hope to do is have a policy that requires at least some level of approval from upstream developers on the correctness of the patch. Which is what happened here; the only real failure that could have been avoided on the Debian contributor's was to send the entire patch for review. For all intents and purposes, it does look as if the upstream developers approved the change, and at best you're hoping that a more rigorous, formal process would have caught the fact that the people approving the change were not in fact upstream developers.

From the OpenSSL side, the faults are more legion. On the technical side, the code relied on questionable voodoo magic working correctly (when it was known that it didn't work correctly in at least some circumstances!), and the code was intrinsically confusing. On the social side, there was no clear way for a would-be contributor to identify how to get a patch approved by upstream developers. The postmortem on the OpenSSL side, however, was to point and laugh at the Debian contributor for being an ignorant n00b (it's not linked in the article), denying any fault on their part.

Perhaps it should be no surprise that a couple of years later, OpenSSL would be slammed with Heartbleed, which surfaced much of the same technical issues that the Debian bug did but without any ability to blame anybody else for making the bug. Heartbleed was severe enough to force people to learn some of the lessons they should have learned from the Debian/OpenSSL fiasco.

Reply on news.ycombinator.com