Hacker News Reader

Credible interval of difference of means, please.

https://link.springer.com/article/10.3758/s13423-015-0947-8

https://web.archive.org/web/20250418135701/https://jkkweb.si...

Neither are what matters in this case. Sure, the mean execution time can be useful for capacity planning, but the individual user will not experience the mean – they will experience individual run times. That makes predictions intervals the right choice. Ideally based on a more extreme quantile, like 99.9th, because even a single slow run taints the user experience quite a bit.

I would tend to surmise that in the vast majority of cases where microbenchmarking is used to evaluate a change, the mean and the 99.9th percentile change in consistent directions.

In cases where they don’t, I’m not sure prediction intervals (which require a parametric model that somehow models the tails correctly) are the right tool, as opposed to e.g. plain measured percentiles.

Reply on news.ycombinator.com