Teaching and Assessing Property-Based Testing

blog.brownplt.org

43 pointsluu3 comments

In my opinion therer is a bigger problem of mutation testing being even less known and less used, even though it's more widely applicable and easier.

How do you personally use mutation testing? In my experience it gives noisy results, so it's most valuable when I have a lot of time to personally examine the output -- contrasted with tools like property testing which can be trivially integrated into an automated pipeline.

As one class of examples, a ton of numeric code works by iteratively finding better solutions till some stopping condition is met, and since such algorithms are by design robust to certain kinds of perturbations a mutation testing framework will find a ton of mutations which don't meaningfully affect the final output. Interpreting such a report requires spending time to figure out if any particular perturbation should or should not have been detected by your tests.

I don't see why the output of the property based testing should be any easier to work with? It sounds more like you're arguing that PBT finds less actionable stuff, which is my argument :)

I use mutation testing for critical code. I don't run it in CI due to it being too slow and bad at incremental work. This is my own mutation tester so that's on me :)

Reply on news.ycombinator.com