Comment by eru | Hacker News Reader

Comment by eru | Hacker News Reader

Parent Full thread

You are right that shrinking is a problem for QuickCheck inspired property based testing libraries. Please do have a look at Python's Hypothesis.

Hypothesis is perhaps the most advanced property based testing library. Or at least: miles ahead of QuickCheck. Hypothesis 'solved' shrinking by integrating it with generation. See https://hypothesis.works/articles/integrated-shrinking/ for an overview. And https://hypothesis.works/articles/compositional-shrinking/ for some more.

I am glad that you found a way to (manually) generate useful test cases. I would like to learn more, and I wonder how we can perhaps automate your technique to make property based testing better!

(David R. MacIver, the author of Hypothesis, has lots of interesting ideas. He says that property based testing is such a great concept that QuickCheck, despite making all the wrong design choices whenever they had any choice, was still a revolutionary library.)

> Or at least: miles ahead of QuickCheck

I'm not so sure about this. The blog post saying that it's better to tie size to generation is dated 2016.

It looks like QuickGen was tying size to generation in 2008: https://hackage.haskell.org/package/QuickCheck-2.1.0.1/docs/...

EDIT: I take it back, it looks like there is shrinking in modern QuickCheck https://hackage.haskell.org/package/QuickCheck-2.14.3/docs/T...

Hypothesis approach looks interesting. I’ll look into it.

As for my approach to generation, there’s not much to it. Instead of generating arbitrarily complex values and then trying to shrink them to simpler ones, I start with simple values and, if they work, proceed to more complex ones. In every test run I use the same values. It’s basically a brute-force, which omits values which are sufficiently similar to values already tested.

The omission is achieved by having a base set of simple values where each is judged by me to be sufficiently different from the others (I basically copy e.g. tricky numbers/unicode values from quickcheck implementations, which are used inside generating code), and the base set is ordered from simplest to less simple. Then I run a loop which goes over all elements of the base set, run the test, then take a function which based on the base set produces more complex values, run the test on that etc. So the earliest fail is going to be the simplest value it can generate.

Reply on news.ycombinator.com