OSS-Fuzz: Five months later, and rewarding projects | Hacker News Reader

OSS-Fuzz: Five months later, and rewarding projects

opensource.googleblog.com

132 pointstanin10 comments

Interesting that they managed to find 8 security bugs in SQLite, which is renowned for having a test suite with ~100% code coverage - yet more evidence for the Dijkstra dictum that testing cannot prove the absence of bugs.

https://sqlite.org/testing.html

That is quite fascinating. I do somewhat wonder if some of that can be lessened by using more "property based testing"[1]. Property based testing can be thought of writing tests that describe the functionality you are trying to cover. It then interpolates (not too unlike fuzzing actually) to come up with every way that might be non-compliant and then checks to ensure your hypothesis holds up. Amusingly enough, the most active python property based testing library is named hypothesis[2], which describes quite well how you write unit tests using it.

[1] http://blog.jessitron.com/2013/04/property-based-testing-wha...

[2] https://hypothesis.readthedocs.io/en/latest/

Here's one of those for C:

https://github.com/silentbicycle/theft

Slides on automated tool below. They use "model-based testing" which is another phrase to get interesting search results.

https://www.slideshare.net/dganesan11/ganesan-automated-test...

Also combinational testing which showed promise at catching in high 90's of defect percentage with test case minimization as a bonus. See summary and slides.

http://csrc.nist.gov/groups/SNS/acts/index.html

I had pretty good results fuzzing SQLite back in the day:

https://lcamtuf.blogspot.com/2015/04/finding-bugs-in-sqlite-...

I think they eventually incorporated AFL into their continuous testing and squashed several dozen bugs. OSS Fuzz scales it up, but yup - the bottom line is that you might think you have 100% test coverage, but you really still need to fuzz =)

It's probably worth pointing out that some categories of bugs are almost invariably caused by the absence of a branch, hence 100% branch coverage doesn't show up such failings. (e.g., null pointer dereferences are typically a failure to branch on a null pointer, malloc failures being unhandled are almost always a failure to branch on the return value of malloc, etc.)

Rather than their code coverage, I'd say SQLite is known for having literally millions of test cases in their test suite. The number of lines of code in their test suite is some absurd multiple of the number of lines of code in SQLite itself.

100% test coverage is an ambiguous term. Is it line coverage, branch coverage, and do the branches include arithmetic nonlinearity branches and exception flags? Are all possibly syscall error states tested?

One hundred percent line coverage is admirable, but it's just a start! (Edit: I see that they had far more than just 100% line coverage, but as you can see even this is not enough to find all cases.)

> 100% test coverage is an ambiguous term. Is it line coverage, branch coverage

The linked document answers that both repeatedly and at length:

> The SQLite core, including the unix VFS, has 100% branch test coverage under TH3 in its default configuration as measured by gcov.

[follow half a dozen paragraph explaining what they mean precisely by branch coverage]

And my edit denotes that I saw that and acknowledged it! :)

Even with 100% coverage, there might be the following code (just a one of numerous examples):

char buffer[15];

.....

memcpy(buffer, source, strlen(source));

Depending on the `source`, the bug can be triggered or not, but the coverage is the same for different `source` values.

Reply on news.ycombinator.com