Ask HN: How to track performance issues with a test suite
So I can write tests that demonstrate certain things complete in reasonable amounts of time.
But: (A) If I write lots of these tests, my test suite is quickly going to grow to the point where I don't want to run all the tests on my own machine as I work (B) A test might take 2x as long to run simply because the machine doing the testing is busy doing other things at the same time, rather than due to a performance regression in the code (C) Maybe one day our cloud vendor will decide a 486dx2 is the machine our test server will live on, instead of the recent i7 we're used to. (D) It would be nice to see how our code is improving or getting worse over time, rather than just knowing "pass/fail".
Is there tooling that takes care of some/all of these issues nicely?