Mutation testing based on LLVM (2018) [pdf]

Hmm, I’m thinking that removing an assertion would potentially allow your program to continue into an undefined state that the assertion was guarding.

You’d also want to watch out for the particular implementation of sorting algos... if you swap out a stable quicksort for a nonstable bubblesort, your benchmark tests might still pass, but you could wind up with different orderings of “equivalent” items in your list before/after that could cause subtle issues.

If you have a test that truly still passes regardless of any mutation, I’d have to wonder if it should exist at all.

This is a valid concern, for sure. In fact, these are called Equivalent Mutants. We observed it several times, here are some examples. You flip the if/then branches and the code essentially does the same, switching from a fast to slow implementation of the same algorithm. Another example is mutating the code in a C++ destructor, which is not easy to test.

With that being said, even despite the equivalent mutants, you can get a pretty good insight. Here is an example of finding from a proprietory real-time OS used in the space industry: https://gist.github.com/AlexDenisov/b5d2e23457b88813b5ab9d5d... (this is a part of an email which I could safely post online).