Mutmut – Python Mutation Tester

github.com

20 points·e-topy··3 comments

I can’t speak for this specific library, I’ve only use PiTest for Java projects, but mutation testing is a pretty awesome idea.

One hard part is deciding which code is worth doing mutation testing on. A lot of “business” code probably shouldn’t be tested this way. The effort in writing the tests outweighs the return on investment. But if you’re making shared library code then mutation testing is great. It really gives you confidence that the code is actually working as expected.

A PiTest blog post [1] mentions an interesting article [2] on how mutation testing is (or was) used by Google.

If I remember correctly, instead of overwhelming the developer with the full report, CI only presented a hand full of randomly chosen failed mutation tests. The developer then could decide to ignore the result (mutation testing will usually produce some false positives) or extend the tests to kill the mutation.

I used PiTest recently the first time and found it quite useful. As you said, a pretty awesome idea. But definitely not a tool for people who always strive for 100% test coverage.

[1] https://blog.pitest.org/dont-let-your-code-dry/

[2] https://homes.cs.washington.edu/~rjust/publ/mutation_testing...

Check out our project Mutahunter - we made use of LLMs to do the mutations at the AST level - https://github.com/codeintegrity-ai/mutahunter

Reply on news.ycombinator.com