Why CVE-2022-3602 was not detected by fuzz testing
allsoftwaresucks.blogspot.com
allsoftwaresucks.blogspot.com
Well, everyone would want that, but it's not possible. Formal verification comes nowhere close to promising that, especially not on a large project. I'm pretty sure OpenSSL is larger than any formally verified software to date (perhaps CompCert is larger?).
Donald Knuth
(I guess that semantics can also be seen as a formally verified property)
Compare that to a language designed well enough that reflection isn't necessary for good APIs, for instance.
That's not really where you'd expect RCE-problems.
Like, the log4j thing came from (among other design errors) choosing to use reflection to look up filters for processing data during logging. Why would log4j's developers possibly think reflection is an appropriate tool for making filters available? Because it's the easy option in Java. Because it's the easy option, people are already comfortable with it in other libraries. Because it's easy and comfortable, it's what gets done.
Some languages make reflection much more difficult (or nearly impossible) and other APIs much easier. It's far more difficult to make that class of error in languages like that.
whistles in python
pickle._getattribute(__import__(package), path)
But more broadly the thing is that eliminating these low level language footguns would allow people to focus on the logic and design errors.
Yes, a safer language is not enough, but it is a huge leap forward, so I'll take it.
NSA's Software Memory Safety recommendation:
https://media.defense.gov/2022/Nov/10/2003112742/-1/-1/1/CSI... (pdf)
So better approach is to use container security or something like that.
"The threat of accidental vulnerabilities in local code is almost impossible to address with the Security Manager. Many of the claims that the Security Manager is widely used to secure local code do not stand up to scrutiny; it is used far less in production than many people assume. There are many reasons for its lack of use: [...]"
Would be interesting to know if there were other cases besides ElasticSearch that were protected from log4j by JSM.
Which is a pity, but unfortunely capability based systems still seem to have a problem for the common developer to properly configure them.
Bad design is a universal orthogonal problem.
See how eg optimized tail calls essentially give you back all the power of goto without the old downsides.
I think interfaces in Botan, to give an example, are way easier to use.
It looks to me like a minefield the OpenSSL API.
That fuzzing is tricky was not the problem here. The problem is the culture that allowed ossl_a2ulabel to exist without unit tests. And before some weird nerd jumps in to say that openssl is so old we can't apply modern standards of project health, please note that the vulnerable function was committed from scratch in August 2020. Without unit tests.
it's not realistic to enforce unit test coverage % with a project at the scale of OpenSSL, right?
I don't know what the deal is with their testing culture but in year 27 of the project they demonstrably haven't learned this lesson. It's nice that they added integration tests (testing given encoded certs) but as the article points out that was insufficient.
Writing unit tests for c/c++ is trivial. There are perfectly fine test frameworks, used by developers every day, integrated in any major IDE or runnable as one-liner from the command line.
This is absolutely a cultural problem.
It was part of our data structures and algorithms project, failure to execute the automatic tests meant no admission to the final exam.
We had three sets of tests, those provided initially at the begin of the term, those that we were expected to write ourselves, and a surprise set on the integration week at the end of the semester.
There is no substitute for reviewers who really understand the code in question. The problem is they are the ones writing the code and so are biased and not able to give a good review.
Why not?
You can enforce that all new files should be covered (at the very least line-covered). It requires some setup effort (collecting code coverage and either sending it to a tool which perform the correlation or correlating yourself), but once that's done... it does its thing.
Then you can work on increasing coverage for existing files, and ratcheting requirements.
Otherwise, I agree.
IMO to prove that code is correct requires a proof; a unit test can only provide evidence suggestive of correctness.
An exhaustive test is just one type of a machine verified proof.
Not entirely sure I agree with this. A proof by construction is a very different beast to empirical unit tests that only cover a subset of inputs. The equivalent would be units tests that cover every single possible input.
That's what "exhaustive" means.
Formal proofs also have limits. Donald Knuth once famously wrote "beware of bugs in the above code, I proved it correct but never ran it". Which is why I think we should write tests for code as well as formally prove it. (On the later I've never figured out how to prove my code - writing C++ I'm not sure if it is possible but I'd like to)
Even running through all 4 billion some cases a single 32 bit number can result in your test taking a significant amount of time - enough that you wouldn't want to run it very often. One value of real world tests is often that they can detect that you broke what you thought was a completely unrelated area of code.
Enums, bools, etc.
>can result in your test taking a significant amount of time - enough that you wouldn't want to run it very often.
It is irrelevant in theoretical discussions like this
You can have coverage on code that divides - it won't tell you if you ever divide by zero.
You can have coverage on code that follows a pointer - it won't tell you if you ever pass a bad pointer.
https://en.wikipedia.org/wiki/Core_Infrastructure_Initiative...
If the current OpenSSL maintainers closed the project, given its importance, there would be a rush to follow up maintenance. Chances are, it'd be better funded; even in worst case, it'll hardly be assigned less than two devs.
This a case of the general dynamic where a barely-sufficient-but-arguably-insufficient solution prevents actors from finding and executing a proper one.
Blaming the OpenSSL developers for writing bad C is just a "no true scotsman" at this point, since there is no large, popular C codebase in existence that I'm aware of that avoids running into vulnerabilities like this; vulnerabilities that just about every other language (mainly excluding C++) would have prevented from becoming an RCE, and likely prevented from even being a DoS. Memory safe languages obviously can't prevent all vulnerabilities, since the developer can still intentionally or unintentionally write code that simply does the wrong thing, but memory safe languages can prevent a lot of dumb vulnerabilities, including this one.
No feasible amount of funding would have prevented this, since it continues to happen to much better funded projects also written in C.
On the other hand, I guess we could blame the OpenSSL developers for writing C at all, being unwilling to start writing new code in a memory safe language of some kind, and ideally rewriting particularly risks code paths like parsers as well. We've learned this lesson the hard way a thousand times. C isn't going away any time soon (unfortunately), but that doesn't mean we have to continue writing new vulnerabilities like this one, which was written in the last two years.
I didn't suggest a complete rewrite of the project. However, they could choose to only write new code in something else, and they could rewrite certain critical paths too. The bulk of the code would continue to be a liability written in C.
I agree that it would be nearly impossible to rewrite OpenSSL as-is. It would take huge amounts of funding and time. In general, people with that much funding are probably better off starting from scratch and focusing on only the most commonly used functionality, as well as designing the public interface to be more ergonomic / harder to misuse.
No, this whole thing is about the lack of testing. Adding a parser without matching tests is just absurd regardless of the language it's implemented with. If only for basic correctness check, you want a test.
Not all vulnerabilities or bugs are memory-related, vulnerabilities are bound to surface in any language with that kind of organizational culture.
Most "safe" languages like Python are written in C and are full of segfaults, mostly due to the high churn rate and the attitude of the developers.
I haven't tried Rust yet, so I won't comment on that.
It is simply difficult to reconcile these facts with the idea that it is a very good team doing very good work.
Is anyone able to explain this to me?
In short, the issue is that you forgot a check, and your code effectively "trusted" that the input would close all its strings. If you never make mistakes like that, you can validate input in C just like in any other language. But the consequences of making that mistake in C are really nasty.
Strings specifically are often enclosed enclosed in a `while(c != '\0')` loop (assume c is the character being examined) or something to that effect, which means you'll exit at the end of the string (non-string arrays don't have this).
The CVE in question seems to be the exact opposite of this. It's that someone didn't check the bounds on a write instead of a read.
Entirely possible, especially if the attacker is local. But when we're dealing with something coming in over the network, I think even the old arpa headers get you a null byte at the end, regardless of if one was sent.
Unless we aren't dealing with tcp/ip, in which case I'm way out of my depth.
You aren’t wrong that even downloading untrusted data is less secure than not downloading it. But to actually exploit a machine that is actively sanitizing unsafe data, you need either (A) an attack vector for executing code at an arbitrary location in memory, or (B) a known OOB bug in the code that you can exploit to read your malicious data, by ensuring your data is right after the data affected by the OOB bug.
Sure, but memory isn't normally executed.
One of the more common problems was not checking length. Many C functions assume sanitized data and so they don't check. You have functions to get that data that don't check length - thus if someone supplies more data than you have more room for (gets is most famous, but there are others) the rest of the data will just keep going off the end - and it turns out in many cases you and predict where that off the end is, and then craft that data to be something the computer will run.
One common variation: C assumes that many strings end with a null character. There are a number of ways to get a string to not end with that null, and if the user can force that those functions will read/write past the end of data which is sometimes something you can exploit.
So long as your C code carefully checks the length of everything you are fine. One common variation of this is checking length but miss counting by one character. It is very hard to get this right every single time, and mess it up just once and you are open to something unknown in the future.
(Note, there are also memory issues with malloc that I didn't cover, but that is something else C makes hard to get right).
It's being handled how it should be. This happened, let's handle it, and how can we work to better address future problems.
I fail to see what is problematic about giving the control over the entire flow of the program to the developer. Quite the contrary, I am more concerned about the paradigm shift towards higher level system programming languages that hide more and more control from the developer while putting more burden on the perfectness of optimizer.
Why are people still using parsers for untrusted input in C? That is the real flaw here, not how the fuzzing was done.
>parsers for untrusted input in C
If it hands it to a C program, that C program needs to parse (in some form!) those values!
How is a C program expected to ever do anything if it can’t safely handle input?
Probably not that important for `ls`, probably worth it for OpenSSL.
No matter what the parser itself is written in, if you're writing in C you'll be using the parser in C.
2. That’s still less of a problem as the C will then be handling trusted data validated by the safe langauge.
I can answer that one. The parser is more dangerous because a parser, essentially by definition, takes untrusted input.
Nothing the parser does is any more dangerous than the rest of the code; it's all about the parser's position in the data flow.
Plain pointer access in high-level code (say when parsing a particular syntactic element by hand in a recursive descent parser) is a violation of the principle of separation of concerns IMO.
In any case I still don't see what's special about parsers. Most vulnerabilities I suspect to be in the higher levels, like validating parsed numbers and references, for a trivial example. In general, those are checks that are likely to be implemented much closer at the core of the application.
What I see (especially in libraries like OpenSSL) is the core logic often receives a lot of scrutiny and testing, and thus it is silly mistakes with offsets and bounds checks that make up the majority of bugs.
It’s also worth considering the severity of different kinds of bug. A bug in high level logic might allow an attacker to do something they shouldn’t be able to do, but it doesn’t give them code execution.
The worst bit is, an attacker can often gain code execution through a part of the code that otherwise wouldn’t be security critical (where a logic mistake would be low impact). So writing code in a language that allows for these vulnerabilities greatly increases your attack surface.
The kernel is written in C.
So that pretty much means all parsers written in C and every other language should consider all input untrustworthy, no?
> Linux is probably the most carefully constructed C codebase in existence and still falls in to C pitfalls semi regularly.
My guess is that it would actually be OpenBSD, but I'm not sure either way.
Using parsers for untrusted input in C is a legacy of when this was written. Requiring the parsing portion (or any version of OpenSSL) to be rewritten in Rust or whatever new language is a massive change given the length of time the OpenSSL project has been around.
Shipping a CVE in critical infrastructure because of a trivial memory safety bug is borderline negligence in 2022. This is why people get upset over new code being written in C. The cost of writing new portions of the software with memory safety in mind dwarfs the cost of writing in C because it's more convenient for the build tooling.
The bigger question is why hasn't OpenSSL bitten the bullet and adopted some memory safety guarantees in their tooling, given the knowledge of the sources of these bugs and prevalent literature and tools in avoiding them!
This does not actually exist, as far as I'm aware. There are certain things people propose doing in C++ that eliminate a small number of issues, but I haven't seen anyone clearly define and propose a subset of C++ that is reasonably described as memory safe. Even if such a subset existed, you would still need some way to statically enforce that people only use that.
Even just writing the parsers in Lua should be a safer choice than writing it in C, but I think now is as good of a time as any to start writing critical code paths in Rust. If the Linux kernel is beginning to allow Rust for kernel modules, then it is high time that OpenSSL looked more seriously at Rust too
As others have pointed out, parser generators could be a useful intermediate option for some of this.
My point is that there isn't a compelling reason to write new code in C for something where safety is critical.
Hence why the ongoing efforts to improve static analysis tooling in regards to mechanical enforcement of C++ Core Guidelines across all major C++ compilers, IDEs and commercial static analysers.
It is perfect? Don't let perfect be the enemy of good.
In any case, anyone that cares about secure code shouldn't be touching any language that is copy-paste compatible with C, unless they can't avoid it.
As for the rest, I was quite clear where I stand on my last sentence.
> What to do with leaks out of temporaries? : p = (s1 + s2).c_str();
> pointer/iterator invalidation leading to dangling pointers
I feel like those are relatively common memory safety pitfalls, not obscure corner cases. The Core Guidelines have been in the works for about 7 years, I think? It's not clear if/when these will ever be addressed if they haven't been addressed by now.
https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines...
There are other things mentioned in the list that look suspicious, but they're less clear. So, unless I'm misreading this, then I stand behind my original assertion that there is no safe subset of C++, but the Core Guidelines are certainly better than nothing... assuming they're actually used/enforced in real world applications. Other people are welcome to have their own opinions.
The idea of not writing parsers directly was well established by the time OpenSSL started in the late 90s.
The main drawback is that it is difficult to get good error messages when a parse fails.
EDIT: An important note to newbs: The Halting Problem is correct. However, a problem which maps to the halting problem can still be solved often enough in practice to make it worthwhile. In fact, entire industries have been born of heuristic solutions to such problems.
Valgrind and fuzzing are useful tools, but there is no general method. Being semi-decidable, with enumerable inputs. This means that fuzzing can be useful both with random walks and constrained random walks. But it doesn't come close to generating 'provably correct code'.
The 'Post correspondence problem' is maybe an easier way to see how this applies at the compiler level.
But tools that help are opportunistic, but that doesn't change the undecidability of generalizations.
While crossing domains, but because it is commonly used, considering the VC dimensionality of the problem also helps, or pure math problems like the Collatz conjecture that generalize to the same halting problem.
Generating 'provably correct code' in the general case is simply not possible without major advances in math. There is room to improve with many paths to do so, just not going down this particular path.
Note that parsing a context free grammar maps to a stack machine. The Halting Problem uses a Turing Machine. I don't think it applies! (But if you do still want to show off your knowledge, please elucidate. I've forgotten about half of the automata theory I covered as an undergrad and in grad school.) For parsing tasks corresponding to computational models which are less complex, like regular expressions, the problem is well under the threshold of "semi-decidable, with enumerable inputs" in your words.
Generating 'provably correct code' in the general case is simply not possible without major advances in math.
Note that I was responding specifically to the problem domain of parsing and compiler compilers, and that most parsing problems involve models of computation which are akin to a stack machine or less powerful.
People generally throwing out The Halting Problem as an objection without carefully considering the particulars/context is one of my chief pet peeves.
C++ templates, Haskel templates, Lisp macros, etc... are all examples of metaprogramming facilities that would be considered TC, and thus subject to Rice's Theorem (To avoid HP).
But last I saw the code that was generated was the problem, not the primitives. As the languages are TC, a parser not being so doesn't remove the issue with this CVE.
Good! So, if what we want the compiler compiler to do is, say, just to produce an output isomorphic to the input, then this also reduces the complexity of what we're asking to do. I think this falls well inside what we could automate with some kind of guarantee of correctness.
A good mental habit for programmers is to constantly ask, "Can the stated problem be reduced in scope, such that we satisfy the goal?"
Parser generators feel like an academic dream
https://github.com/openssl/openssl/blob/openssl-3.0.6/crypto...
* Have a stream-like abstraction for getting or peeking at the next symbol (and pushing back, if necessary). Make it impervious to abuse; under no circumstances will it access memory beyond the end of a string or whatever.
* Have some safe primitives for producing whatever output the parser produces.
* Work only with the primitives, and check all the cases of their return values.
One solution to this problem would be to write an LLVM backend that outputs C. Maybe such a thing already exists.
Which is great news for those of us who approach such research by gaining a deep understanding of the code and the systems it exists in, and figuring out vulnerabilities from that perspective. An overreliance on fuzzing keeps us employed.
It's far more doable than you are suggesting: fuzzing automatically covers most branches anyway, so you just need to manually deal with the exceptions (which are easy to locate from the code coverage).
I used fuzzing to test an implementation of Raft, and with only a little help, the fuzzer was able to execute every major code path, including dynamic cluster membership changes, network failure and delays. The Raft safety invariants are checked after each step. Does this guarantee that there are no bugs? Of course not. It did however find some very difficult to reproduce issues that would never have been caught during manual testing. And this is with a project not even particularly well suited to fuzzing! A parser is the dream scenario for a fuzzer, you just have to actually run it...
Coverage might have helped here (or not), but it doesn't fix the general problem of fuzzing being stochastic and only testing some behaviours of the covered code.
I feel like there is this trend of assuming any harsh criticism is bad faith. Asking why industry standard $SECURITY_CONTROL didn't work immediately after an issue happened that should have been caught by $SECURITY_CONTROL is hardly a bad faith question.
Someone pushing hard on legitimate criticisms with the intent of attacking a project or members thereof is acting in bad-faith, while someone ignorant with a totally bogus criticism could be acting in good-faith. Many bad-faith actors hide behind a veneer of legitimacy by disguising or shifting the gaze away from their motivations.
Bad/good faith is about whether you are being misleading or dishonest in asking the question.
You can intend to go attack a project in good faith as long as you are not being misleading in your intentions.
For example, a movie critic who pans a film is not acting in bad faith since they aren't being misleading in their intentions.
We might actually be in agreement with each other because a critic who leads with "The director slept with my wife, so I'm only going to say all the bad things about the film and you should probably ignore this review" would have significantly blunted their attack by leading with it, and are arguably not acting in bad-faith.
All this bickering over language misses the real problem.
The actual solution is that open source, widely used code is a target for hackers.
By using one library used everywhere for everything, you’re painting a target on your own back.
The real solution is we need the software ecosystem to have more competition and decentralization.
Use alternative crypto libraries.
If you want a drop in replacement, use LibreSSL which was forked and cleaned up by the OpenBSD guys due to HeartBleed.
But the long term solution, is more competition by using smaller, more specialized libraries, or even writing your own.
The long term solution is likely using languages which are A. Memory safe, and B. make formal verification viable. Being widely used and open source isn't an issue if there are no exploitable bugs in the code.
What is your primary language?
Mac OS was originally written in Object Pascal + Assembly, just to cite one example from several.
OpenSMPTD had its fair share of exploits. sudo had its fair share of exploits.