Original author here, I'll be interested in feedback about the post and use cases you have for property-based testing.
FYI my personal blog lives at https://mathspp.com/blog.
FYI my personal blog lives at https://mathspp.com/blog.
https://en.wikipedia.org/wiki/Equivalence_partitioning
Then I talk about randomly sampling from a partition as a way to eventually detect "hidden" partitions you didn't expect. Like Hypothesis does.
When dealing with many parameters, each with several partitions, the variations multiply quickly. To deal with combinatoric explosion, I talk about all pairs testing, also known as "pairwise" testing.
For the longest time I was convinced I should have “perfect” oracles (unit test style) because that’s what simple examples usually show off, but outside of trivial examples it’s really only possible for limited cases unless you have a complete and perfect oracle (usually an existing known-good implementation).
Instead, I would write multiple tests, mostly just asserting one thing each. For example: `test_gcd_always_positive(m, n)`, `test_gcd_divides_arguments(m, n)`, `test_gcd_is_largest_divisor(m, n)`, etc.
I also think in terms of a 'threat model' for our colleagues or past/future selves:
- They're probably not trying to write broken code, so we don't need to catch all incorrect implementations. In other words, they're not adversarial.
- They're probably lazy or mistaken, so we should try to catch incorrect implementations which are simple to write, make obvious mistakes, etc.
For example, having `my_sort = lambda _: []` is a really lazy way to implement the right type signature. Checking that the lengths and set of elements are the same are good tests to catch that, so it seems worth doing. Likewise, checking that the elements are sorted is a good test to avoid the lazy approach of `my_sort = lambda l: l`.
However, the implementation which sorts the set of elements and pads with the largest is pretty convoluted, so a lazy developer will likely just write a basic sort algorithm at that point (e.g. `my_sort = lambda l: [x for y in set(l) for x in l if x == y]`). Also, the test is a bit tricky; if that counting function didn't already exist, I probably wouldn't bother writing that test at all (there are equivalents which might be simpler to write, e.g. for each input element: assert it appears in the output, then remove its first occurrence; then also assert the output ends up empty)