Continuous Load Testing
slack.engineering
slack.engineering
Pipelines take a lot longer to run (because the load test environments are expensive we have one or two and serialize test runs on them to stop test runs stepping on each other, but the teardowns are slow, especially on failed runs)
To keep the test run fast we limit them to about 5 minutes but this creates unrealistic expectations of how fast our code would actually need to scale.
But it is nice knowing which changes have broken the load test. Hunting them down is so painful.
What I’d to have is proper metrics showing how close we are to our Non Functional Requirements over time to make it easier to sell “Performance stories” ahead of breaking the NFR rather than just fixing it as we break.
But at least some people should get machines that are a few years old - because that's what the users will have.
This used to be a classic problem in PC game development - developers get kitted out machines (because they need to run more heavy duty tasks during development, or run GPU debug code, or need to test the limits of the engine, or maybe just because they are gamers themselves so they buy newer hardware more often), so they tend to miss the mark on how powerful the average consumer machine is and only optimize code until it runs smoothly on their machine.