Property-based testing in practice [pdf]

mutmut right now is implemented by writing changes to disk and starting new processes. But you can implement it via "mutation schemata" where you functionally compile all possible mutants ahead of time, plus the original function, and replace the original function with a trampoline that either calls the original or one of the mutants depending on some external state.

I have a prototype of mutmut that does this and it's 10 to 100x faster. It does have the downside of not being able to mutate stuff like static global variables and such though.

Maybe we're talking about different things then, but in my experience, MT is a reasonably involved procedure that requires configuring the MT harness to understand where the code lives, how to run the test suite, how to interpret the test suite, etc, then running the mutation tests for upwards of an hour as it repeatedly makes various changes and runs the tests. If you want to do something more complicated like only test a given region of the codebase, then the configuration becomes even more involved.

This is all significantly more involved than my experience with PBT, which tends to be something that can be added without much ceremony to an existing test suite when it makes sense.

To be clear, I love the idea behind mutation testing and I have given it a go a few times with limited success, but I think your comment is overselling its simplicity.

That said, I'd love your advice: how do you introduce mutation testing to a large codebase that currently has an extensive set of tests but hasn't used mutation tests yet? And how do you maintain the MT side of things? It seems far too slow to regularly run in CI: do you just run the MT tool every now and then to make sure that the tests are still covering all the mutations? Or do you have a more structured approach?

I love PBT and use it frequently.

So integration tests are generally IO-bound. They'll be slow.

With PBT you will generate 100, 1000 slow cases.

Integration tests are flaky in the sense that your components might not generate just the right inputs to trigger an error. They may not be configured under test the same way as they are in production. And so on.

PBT can be flaky in the sense that, even with carefully selected distributions, you can get a test that ran fine for months to fail on a run. Add that on top of integration test flakiness.

There's no silver bullet for writing bad specifications.

Tests that are slow to run don't get run often. People don't write tests first. They write fewer tests. They tend to test the happy paths.

I'm not saying we should avoid integration testing. It's necessary.

It doesn't mix well with PBT for a main test suite, in my experience.

You're better off making sure your domain types and business logic are encapsulated in your types and functions and don't require a database, a network, etc. PBT is great at specifying invariants and relations between functions and sets.

Use your integration testing budget for testing the integration boundaries with the database and network.

Update: there's no cure for writing bad tests. you just have to write better ones.

I started my familiarity with programming using a bunch of Micorosoft's products, but later was able to completely migrate away from them. Today, if I'm required to use a Microsoft's product, it's kind of like asking a North Korean who valiantly fought to get away from that cursed country to go back and live there again.

You see, my understanding of what's happening with MS is that their sales department is exceptionally successful, which lowers the requirements for everything else in the product. A particular quality of this sales department is that they rarely sell to individual end users: to be efficient, they sell to people who have absolute control over the end users, but are also incompetent when it comes to software. Think governments, hospitals, schools etc.

Another aspect of MS software is that they, internally, don't really allow anything that's not made by MS, and they are big enough for this policy to succeed. The unfortunate side effect of this policy is the Stockholm syndrome: MS developers not only get used to the MS tools, they genuinely seem to like the suffering incurred by using them. Another unfortunate side effect of this policy is that MS developers have no idea what normal users actually like. So, they are kind of like Bender (a robot, who has only a very faint idea of how humans eat) in the Futurama episode, where he enters a cooking contest. He then proceeds to put all kind of inedible components into his meals (but eventually wins by adding some LSD to the mix).

So... I will never touch any MS products with a ten feet pole, unless forced to by my employer. And if the employer goes as far as requiring me to use MS products a lot, I'll look for another place to work. My job today is quite meaningless, boring and I don't get to work with things I like or people I respect... but having to use MS tools for work would still be a lot, lot worse. It's a kind of a torture... well, of course, it's an exaggeration: it doesn't cause me physical pain, but it's a kind of frustration that makes me want to punch a wall every few seconds.

And to top it: using C# for anything is kind of like... starting a cooking recipe with "put a handful of chopped garlic into the pen". You just know it's either a mistake, or a prank, or total incompetence. You just don't keep reading after that line: there are plenty of recipes around, it's not worth your time to try to figure out why this one is defective.