Compiler fuzzing, part 1
vegardno.net
vegardno.net
inline namespace {
namespace {}
}
namespace {}
I fuzzed early clang for segfaults, just for fun. I found dozens of bugs by just taking a large preprocessed C++ file, deleting a few lines, and then when finding a segfault, reducing by deleting lines and tokens until I reached a minimal example of the same error (make sure you are crashing in the same place, or you will probably find you keep rediscovering the same bug).
Reported some great fun minimal crashes, like:
:)
:{}
namespace
I also fuzz in my day job. There we want to fuzz valid inputs -- we have a bunch of ways of generating two inputs which should produce the same output -- generate a pair of inputs, test outputs, repeat. It has found hundreds of bugs, and almost no bugs (famous last words) have been reported in releases.
(In my case I fuzzed a simple virtual machine and the display-mode of a console-based email-client.)
It is disappointing how often it is trivial to find bugs via fuzzing, which I guess just underlines the point that we SHOULD be fuzzing more often. Even on programs/libraries that are "old".
> I found dozens of bugs by just taking a large preprocessed C++ file, deleting a few lines, and then when finding a segfault, reducing by deleting lines and tokens until I reached a minimal example of the same error
To run with this idea: this process could be automated, and applied to any number of different compilers.
Out of curiosity, did these bugs tend to be things like buffer-overflows? How many of the bugs you reported do you think would have been avoided had the compiler been written in a 'safe' language rather than C++?
I'd say that's the best kind of bug. Reflects well on the codebase.
C++ parsing is an infamously untidy problem.
For C(-like) code, there's venerable Delta (http://delta.tigris.org/) and more powerful C-Reduce (https://embed.cs.utah.edu/creduce/).
For binary formats, there's afl-tmin, part of AFL (http://lcamtuf.coredump.cx/afl/).
Kudos for the AFL creator http://lcamtuf.coredump.cx/afl/
I would love if it was easier to do "external checks" (for example network fuzzing). Something like a dissociation of the "submit payload" and the "tortured piece of software" would be really really nice.
That's a really cool setup. Precompiled binaries to quickly git bisect a change.
How often did you run into not being able to use git bisect because you encountered problems due to code changes that were unrelated to the bug you were hunting?
Quickly being able to reproduce and find the commit for a bug is tremendously useful.
But apart from the direct value, there's a huge value in keeping high paid staff motivated and sharpening their skills of course, on top of the good will it creates among potential hires.
Then, the computer started swapping like hell and became unresponsive, which didn't settle for another 10 minutes after I had shot down the process. Looking at the case that radamsa had generated, it had found a billion laughs attack vector. Macros in my language can be defined recursively, and the code is stored in an array that gets reallocated and grown when the code no longer fits, unboundedly. Radamsa had created an initial macro and then redefined it over and over such that it always referred to the previous definition twice.
I was optimistic about fuzzing, but I never really had any expectations of it finding anything other than stack smashing and segfaults.
In such a case where there's a much better original source, could you please post that instead? This is in the site guidelines: https://news.ycombinator.com/newsguidelines.html.