Ask HN: How do you insure against regressions in performance critical codepaths?

If you work on performance-critical systems like database software or language runtimes, what sort of tooling do you use to insure against regressions in the critical codepaths in your system? Are such checks performed in CI or right before some sort of a release or elsewhere?

Tangentially, for folks who spend a lot of time benchmarking systems on non-bare metal instances on AWS EC2, how often do you have to deal with inconsistent benchmark results due to noisy neighbor issues?