The benchmark you’re reading is probably wrong

Another common flaw in broken benchmarks: use a single client instead of many simultaneous clients. In this configuration you are mostly measuring the RTT plus the client lib implementation latency.

Another one is testing on-disk DBs with datasets that fit in RAM: the whole DB data will be cached in OS buffers and you don't get the picture of how the DB will perform with big data.

Benchmarking is one of the most complex tasks, but for some reason benchmarking is usually not done by the most expert guys in the field.

It's been a long time since I've benchmarked a RDBMS, but it's always the same story: Benchmarks are rarely done correctly. People think a benchmark as the definitive way to prove their point, so they hastily and sloppily design a simplistic experiment to measure their desired quantities and then cite it all over the place as The Truth.

For almost 8 years I've had to constantly scrutinize people's benchmarks (worked in networking and distributed systems research, and measurements are crucial in those contexts), point out their errors, and make them think about the details and "hidden" factors that affect their experiments. I haven't met more than a handful of people that could get a benchmark right on their own, and none (including myself) that get it right the first time.

It's good that there are people that could point our errors out.

Ironically, this blog wasn't benchmarked either: 504 Gateway Time-out nginx/0.6.35 The main page works though.