Compilers Love Messing With Benchmarks

brendangregg.com

27 points·helper··4 comments

Measurement bias can have a huge effect, even when you think you are doing 'good' benchmarking.

See the paper 'How To Produce Wrong Data Without Doing Anything Obviously Wrong!' [1]. They just change the size of environment variables and the link order of files, and get wildly different performance results. Most people just assume if they close all other running programs they'll have a fair benchmark, but there is so much more to consider.

[1] - http://www-plan.cs.colorado.edu/diwan/asplos09.pdf

The only conclusion here is not to use old and/or synthetic benchmarks.

Nope. The conclusion is that it is really easy to benchmark the wrong thing. If you are trying to evaluate AWS vs DO vs Google Compute Engine vs Joyent you better make sure you are actually comparing apples to apples. Your comparison will probably invalid if you are using a different kernel version or compiler version or possibly even different shared libraries.

No, the conclusion is "If you want to compare different servers using benchmarks that you compile, you need the compilers to match, or you need to take that into consideration. This should be something you unearth by following an [active benchmarking](http://www.brendangregg.com/activebenchmarking.html) approach, where you study and understand what the benchmark really does."

Reply on news.ycombinator.com