A new approach to property based testing | Hacker News Reader

A new approach to property based testing

drmaciver.com

35 pointsluu9 comments

Interesting read and a very clever idea to minimize test cases.

I've been doing similar property based testing on some math/physics heavy code, with a somewhat similar idea but without minimization. I have test cases that take a number of floating point arguments in range 0..1, which are generated from a 64 bit integer "seed" value picked from a special sequence that is ordered to produce failures as early as possible (failures are usually at 0, 1 or 1/2). Any failure can be reproduced by re-running the test case with the same seed value.

In my case, minimization doesn't make sense since each test case takes a certain number of floats as input. I use about a minute of CPU time per test case (running with 8M different seeds).

Example test case (it's about orbital mechanics): https://github.com/rikusalminen/twobody/blob/master/test/two...

By all means, please deliver; this sounds really nice! If there's one area in software development, it's more approachable and extensive testing rigs that work end-to-end (i.e. that don't rely on mocks, but work with actual data input and output).

I love Hypothesis. I can't find many practical uses for it (it's more useful in more "algorithmic" code, rather than for interactive apps), but it's fantastic in what it does.

Have you seen the talk I did about using it with Django? http://www.drmaciver.com/2015/06/hypothesis-for-django/

We're all still figuring out the details, but this sort of testing definitely works for testing your applications as well as your algorithms.

ETA: Oh, and thanks for the kind words. :-)

I didn't, but I will and get back to you. There's certainly a need for randomness in unit tests, since I can't be trusted to write tests for the code I wrote (if I knew everything that would break before writing the tests, I'd fix it in the first place).

I've been using hypothesis for testing a decent sized flask project at work and it's been great.

Not sure if it's directly applicable to Hypothesis, but one way to use QuickCheck for interactive code is to define a type representing actions to take, eg.

  data Action = DoFoo | DoBar | DoSequence [Action] | DoParallel [Action]

We can then write an "run" function which executes these action, and use it in our properties.

It's pretty easy to do this sort of thing in Hypothesis. You can also use [the support for stateful testing directly](http://hypothesis.readthedocs.org/en/latest/stateful.html). I find there's still a place for more granular tests that look more like your classic property based tests with a bit more imperative state floating around though.

This is quite cool, and opened up a different way of looking at property based testing more like Metropolis or other random walk optimizers.

Reply on news.ycombinator.com