Mutation Testing: A Tale of Two Suites | Hacker News Reader

Mutation Testing: A Tale of Two Suites

codeascraft.com

16 pointskiyanwang7 comments

I've tried to get mutation testing going on my last few projects. Amusing how little faith/interest there is in this from most of my colleagues. Would be interested in hearing how folks keep this up in a project.

It's probably most important in language ecosystems like Ruby which seriously depend on test suites for catching bugs.

One way to view the problem that Mutation Testing solves is, there's no test suite for the test suite. And so there's no safety net to enable refactoring the test code. It's test suites all the way down.

Mutation testing solves this infinite recursion problem by automating the test-suite-test-suite creation.

Yeah, I pitched it as a way to know how good your tests are at catching common bugs. Still no dice.

Did you considered doing it anyway? If a one-off run reveals some previously unknown flaw in the test suite you could present that.

Apologies for not seeing this.

I have added it. It got removed for adding to the build time. Never really found many issues, so I couldn't argue to keep it. Did point out that the argument was essentially against testing.

The logic of refactoring test code is very interesting: IMO one can't just do that, because it tosses out the guarantee that it still tests what one thinks it does.

So I actually break my application first to make the tests fail, and then refactor them while failing to ensure that they still fail. Finally, I undo my changes to the app code, and the tests should all pass again. I guess this is essentially a manually performed mutation test.

I do the same thing, either when refactoring test code or when I'm adding tests after writing the implementation. I've heard the manual approach referred to as "refactoring against the red bar"; see https://corgibytes.com/blog/2016/09/20/refactoring-against-t....

Reply on news.ycombinator.com