Hacker News Reader

Comment by KTibow | Hacker News Reader

Testing tests if it responds with the correct output for a known input. Fuzzing can't replace testing because a. you still want to test that it works correctly given known inputs b. not all functions need to accept badly formatted input

View on HN

Paradigm shift: fuzzing is testing if you write fuzzers that test properties. For instance, if you have a class that can be parsed and serialized, you may write something like this in a fuzzer:

    if (serialize(parse(data, size)) != data) {
        CRASH();
    }

More directly addressing your point about not all functions accepting malformed input, you can also use a fuzzer to exercise sequences of methods on an object. If your class has any invariants, you can test them along the way.

Is this any different from property based testing?

Indeed, what bqmjjx0kac proposes, is orthogonal to fuzzing (which is an algorithmic technique to drive test-coverage), and an instance of defining, and dynamically checking a property.

Apropos: https://blog.regehr.org/archives/1687

I consider coverage-guided fuzzing with property checks to be a degenerate form of property-based testing à la Haskell's QuickCheck.

Defining custom mutators (e.g libprotobuf-mutator) is sort of like implementing the Arbitrary typeclass if you squint.

I admit, this might be where the similarities end :)

If it is a common task like decoding base64 then you can compare the results of your function with the results of a well established existing implementation and abort when they are not equal.

Reply on news.ycombinator.com