In my experience such tests quickly become unmaintainable and suddenly the correctness of the test becomes dependent on the order tests are run in preventing you from running subsets, running them in parallel, being extremely flaky (since the shape of the data is random on each machine it’s run on). This is discussed but dismissed, but tests need to be as maintainable as the code itself.

I tried this on a codebase I came into that had maybe 10 end to end integration tests. This was a bit worse because each test relied on the one declared before it running. Refactored it so that we spun up a unique s3rver and tore it down for each test. Sure, this means only the test cases we write are tested. That’s a feature not a bug. If you want coverage of weird states then use proper testing tools (eg fuzzing and property testing) to generate those complex test cases. Doing so formally is a much better recipe for success then hoping upon success by incidental randomness. I went from spending a day and a half and failing to add one new unit tests of some functionality to a day of pretty straightforward test case writing once I refactored each test to be standalone. Make it easy to write and debug tests not harder and let your testing frameworks handle the heavy lifting of finding corner cases.

Also, converting serial tests to parallel isn’t as complicated as the author makes it sound. You would start piecemeal and let that grow by enforcing policies in code review about what tests look like going forward while you go through the backlog of converting brokenness (running the parallel pieces in parallel and the serial pieces serially)