Ask HN: Are your code benchmarks statistically valid?

1 pointDenisM3 comments

It appears that some benchmarking tools provide average or percentile values but no information about statistical validity of the result. What did you observe or do in your career?

View on HN

I haven't got a clue since I have never used any benchmark tool in my whole life. Could you tell us in which professional context you would use that kind of tool?

Run a benchmark to compare performance of my product before and after making a change in code.

I run it 5 times and drop the two extremes. If the middle tree don't overlap, I'm probably fine. Not super precise, but good enough and easy.

Also, add a loop so the benchmark is like 1 second long, I'm old enough to be worried about anything smaller than 1/18 seconds.

Reply on news.ycombinator.com