That fuzzing is tricky was not the problem here. The problem is the culture that allowed ossl_a2ulabel to exist without unit tests. And before some weird nerd jumps in to say that openssl is so old we can't apply modern standards of project health, please note that the vulnerable function was committed from scratch in August 2020. Without unit tests.
https://en.wikipedia.org/wiki/Core_Infrastructure_Initiative...
If the current OpenSSL maintainers closed the project, given its importance, there would be a rush to follow up maintenance. Chances are, it'd be better funded; even in worst case, it'll hardly be assigned less than two devs.
This a case of the general dynamic where a barely-sufficient-but-arguably-insufficient solution prevents actors from finding and executing a proper one.
Blaming the OpenSSL developers for writing bad C is just a "no true scotsman" at this point, since there is no large, popular C codebase in existence that I'm aware of that avoids running into vulnerabilities like this; vulnerabilities that just about every other language (mainly excluding C++) would have prevented from becoming an RCE, and likely prevented from even being a DoS. Memory safe languages obviously can't prevent all vulnerabilities, since the developer can still intentionally or unintentionally write code that simply does the wrong thing, but memory safe languages can prevent a lot of dumb vulnerabilities, including this one.
No feasible amount of funding would have prevented this, since it continues to happen to much better funded projects also written in C.
On the other hand, I guess we could blame the OpenSSL developers for writing C at all, being unwilling to start writing new code in a memory safe language of some kind, and ideally rewriting particularly risks code paths like parsers as well. We've learned this lesson the hard way a thousand times. C isn't going away any time soon (unfortunately), but that doesn't mean we have to continue writing new vulnerabilities like this one, which was written in the last two years.
I didn't suggest a complete rewrite of the project. However, they could choose to only write new code in something else, and they could rewrite certain critical paths too. The bulk of the code would continue to be a liability written in C.
I agree that it would be nearly impossible to rewrite OpenSSL as-is. It would take huge amounts of funding and time. In general, people with that much funding are probably better off starting from scratch and focusing on only the most commonly used functionality, as well as designing the public interface to be more ergonomic / harder to misuse.
No, this whole thing is about the lack of testing. Adding a parser without matching tests is just absurd regardless of the language it's implemented with. If only for basic correctness check, you want a test.
Not all vulnerabilities or bugs are memory-related, vulnerabilities are bound to surface in any language with that kind of organizational culture.
Most "safe" languages like Python are written in C and are full of segfaults, mostly due to the high churn rate and the attitude of the developers.
I haven't tried Rust yet, so I won't comment on that.
It is simply difficult to reconcile these facts with the idea that it is a very good team doing very good work.
it's not realistic to enforce unit test coverage % with a project at the scale of OpenSSL, right?
I don't know what the deal is with their testing culture but in year 27 of the project they demonstrably haven't learned this lesson. It's nice that they added integration tests (testing given encoded certs) but as the article points out that was insufficient.
Writing unit tests for c/c++ is trivial. There are perfectly fine test frameworks, used by developers every day, integrated in any major IDE or runnable as one-liner from the command line.
This is absolutely a cultural problem.
It was part of our data structures and algorithms project, failure to execute the automatic tests meant no admission to the final exam.
We had three sets of tests, those provided initially at the begin of the term, those that we were expected to write ourselves, and a surprise set on the integration week at the end of the semester.
There is no substitute for reviewers who really understand the code in question. The problem is they are the ones writing the code and so are biased and not able to give a good review.
Why not?
You can enforce that all new files should be covered (at the very least line-covered). It requires some setup effort (collecting code coverage and either sending it to a tool which perform the correlation or correlating yourself), but once that's done... it does its thing.
Then you can work on increasing coverage for existing files, and ratcheting requirements.
Otherwise, I agree.
IMO to prove that code is correct requires a proof; a unit test can only provide evidence suggestive of correctness.
An exhaustive test is just one type of a machine verified proof.
Not entirely sure I agree with this. A proof by construction is a very different beast to empirical unit tests that only cover a subset of inputs. The equivalent would be units tests that cover every single possible input.
That's what "exhaustive" means.
Formal proofs also have limits. Donald Knuth once famously wrote "beware of bugs in the above code, I proved it correct but never ran it". Which is why I think we should write tests for code as well as formally prove it. (On the later I've never figured out how to prove my code - writing C++ I'm not sure if it is possible but I'd like to)
Even running through all 4 billion some cases a single 32 bit number can result in your test taking a significant amount of time - enough that you wouldn't want to run it very often. One value of real world tests is often that they can detect that you broke what you thought was a completely unrelated area of code.
Enums, bools, etc.
>can result in your test taking a significant amount of time - enough that you wouldn't want to run it very often.
It is irrelevant in theoretical discussions like this
You can have coverage on code that divides - it won't tell you if you ever divide by zero.
You can have coverage on code that follows a pointer - it won't tell you if you ever pass a bad pointer.