Any time I see engineering time spent on splitting/sharding test suites, I can't help but wonder if access to a single beefier runner (e.g. 64+ cpus) would have alleviated all that work. Also always find the duplicated setup time a bit wasteful on resources.
Basically yeah. At my last company we switch from n GitHub runners to 2 128 core epyc boxes and massively (20x?) decreased our build times. A surprising thing is how much time is spent on IO to upload and restore the cache between steps that’s saved if the entire job runs on a single machine
Large box in the closet has always been a competitive strategy.
You have to keep your closet clean and organized or else everything is bad, but if you can, it works well.
Every time I see this type of story I can only think how people get rewarded for heroically/dramatically rescuing bad systems. While those who choose boring/proven technology at an earlier point, don't get recognition (or even hired in many cases).
They call pytest --collect-only and parse the output before distributing it to the Github Action python matrix. I'm a little surprised pytest doesn't offer the ability to cache collection. Though a slow collection time may be an issue that should be addressed since developers must suffer it locally!
Could do with more detail, is this using pytest xdist ?
They mention their Github pull request with all changes.
I must be misunderstanding. Every test run was spending 3 _hours_ just discovering which tests to run 9 times?