Hacker News Reader

I have an upcoming article on my lack of understanding on how to do this also. It's not finished but you may enjoy a near-finished draft. https://two-wrongs.com/extreme-value-spc

I did forget to bring up the Poisson approximation you mention though. I'll include that too.

View on HN

The example of performance is interesting because as you say, there are often multiple jostling distributions under the surface (GC is one, but another doozy is CPU frequency scaling).

One possible way out is to look for measurements that contribute to running time but which are not affected by other factors. I remember the YJIT folks talking about using CPU instruction counters, but I can't find it on the benchmark website.

Reply on news.ycombinator.com