RamFuzz, a fuzzer for individual method parameters in unit tests | Hacker News Reader

RamFuzz, a fuzzer for individual method parameters in unit tests

github.com

13 pointsingve8 comments

You are looking for QuickCheck.

Unfortunate that this project is only a partial reimplementation of QuickCheck. I was hoping that this would use fuzzing coverage.

Coverage guided QuickCheck has been on my mind for a while, I believe it would greatly improve fuzzing experience and allow to easily and thoughroughly check/fuzz things that aren't parsers/decoders/deserializers or functions that operate on strings in general.

As proposed by Dan Luu: https://danluu.com/testing/

As I explain in the RamFuzz overview paper [0], Dan Luu's idea is unlikely to work because fuzzing uses mutation techniques developed for file input, not individual parameter values. What does seem to work is frameworks like EvoSuite and eToc, which mutate method-invocation sequences to try and maximize coverage.

And yes, QuickCheck (or RapidCheck[1], for C++) is what you need if you enjoy manually specifying your test inputs. RamFuzz is a tool to support research into automating that and, eventually, the entire process of writing unit tests.

[0] https://github.com/dekimir/RamFuzz/blob/master/sci/ramfuzz.m...

[1] https://github.com/emil-e/rapidcheck

It doesn't sound like you're talking about the same thing. Dan Luu's idea sounded more about adding coverage guidance to QuickCheck, which presently lacks it, not using fuzzing-like mutators.

But how exactly does Dan propose to add coverage "guidance"? Here's what he says:

"Since Zalewski has already done the work of figuring out, empirically, what heuristics are likely to exercise more code paths, it seems like a waste to ignore that and just generate totally random values."

Dan does, in fact, explicitly suggest "using fuzzing-like mutators", as you put it, in QuickCheck's random value generator. But those mutators work on continuous file contents, not on a bunch of disparate parameter values generated at distinct moments in the program. An early version of RamFuzz tried to apply AFL to the log file recording all generated random values, but all that yielded was invalid logs that couldn't be replayed. I don't know how else one would leverage Zalewski's heuristics in a random-value generator, though I'm obviously open to suggestions.

This applies to C++, maybe the title should be updated.

The previous submission's title does mention C++ ;)

https://news.ycombinator.com/item?id=15789033

Reply on news.ycombinator.com