Hacker News Reader

Comment by platz | Hacker News Reader

referencing fuzzing so heavily in the definition places a lot of faith in understanding that there are those subtle varieties of fuzzers that can generate structured data/inputs.

I didn't know fuzzers did that, so I'd guess that others would miss the nuance there too; and it's a rather large part of property based testing

This is a definition that aims to be more "categorically correct" without providing a lot of intuitive explanatory power. (E.g. A monad is a monoid in the category of endofunctors). Maybe that's ok in that was the aim and PBT can't be reduced to a single sentence like a lot of abstract concepts

It also assumes a well-defined definition of a fuzzer

I was also surprised to not see the word 'invariants' mentioned once. Maybe that's very close to "property", but property has other connotations so it still seems useful

View on HN

To me property based testing is just checking that a function obeys a certain law by throwing lots of random data at it.

That means you have to define the law in terms of the input and output. Rather than fix the input like you'd do on a regular test and just assert the output.

The hard part is trying to figure out what laws you should expect from your functions.

Maybe you should write a function that does the right thing, and then use it to tell you the correct output :/

You may be joking, but this can be a good technique for cases where you have a simple but inefficient solution and a complex optimized solution. Ditto for anything that has a special case that can be easily answered.

Also known as an 'oracle'

Half-joking. It can be helpful to verify an optimized implementation with a naive one. That's a small minority of testing though.

Special casing is effectively traditional testing: input is 1, output is 2.

I did that for high-assurance systems as an equivalence check to ensure optimizing compiler didn't break things. It's also standard in hardware verification where they equivalence check each step from highest-level form to lowest-level form.

Not totally silly. Just last Friday I was writing a Python hypothesis-based test using constrained-random stimulus generation. The target function under test is rather complicated because it is coded to be space and time efficient. It is not so easy to read and reason about, surprise, surprise. But there is a straight-forward, easy to ready, while-loops-and-simple-if's implementation that is slow and inefficient, but it's not too hard to convince yourself it produces the correct answer. So I use that in my test driver to generated expected results. It's certainly possible for both implementations to contain bugs, but extremely unlikely for them both to report the same wrong answer for the same stimulus.

Here's an example I wrote where I test the fast API against the (probably correct) brute-force version: https://github.com/llimllib/ckmeans/blob/master/ckmeans.py#L...

Just nitpicking about "extremely unlikely for them both to report the same wrong answer for the same stimulus".

This assumption is what are "blind implementation voting systems" were based on, and it was proved to not survive that well in practice. It may totally work in your case, but you may be interested in looking at the work on Nancy Leveson at the MIT (all links are from her website)

http://sunnyday.mit.edu/papers/nver-tse.pdf

http://sunnyday.mit.edu/critics.pdf

http://sunnyday.mit.edu/papers/nver2.pdf

http://sunnyday.mit.edu/papers/consistent-comp.pdf

This body of work launched quite a controversy, but well, it is still here...

I look at property-based testing the same way I look at Design by Contract. It can be used to find errors in existing systems, but it's also valuable when you are designing.

Just asking yourself what laws a function should adhere to as you are considering writing it function puts you in a different frame of mind and you can end up with simpler software.

Yeah, fuzzing is just throwing random data. Property-based is supposed to be more like DbC where it's guided to test the righg things.

Reply on news.ycombinator.com