Hacker News Reader

Comment by vardump | Hacker News Reader

Benchmarking like that is often broken because of continuous CPU core clock speed adjustments, system interrupts, SMIs, etc.

I tried to fix it by switching hyperthreading off, playing with the scaling governor, boost, setting a CPU frequency to no avail. The jitter was too much and the results were not reproducible, so I just gave up.

Of course your mileage may vary; this was on an AMD Zen 3 CPU.

View on HN

Intel published some guidance on doing precise benchmarks. Basically run your code inside the kernel, turn off interrupts, use the cpuid instruction to prevent out-of-order execution, use rdtscp instruction instead of rdtsc, etc.

> prevent out-of-order execution

Won't this make the results inaccurate?

You just want to prevent the CPU from capturing the end time before it has finished the benchmarking code. Or prevent it from running benchmark code before it has captured the start time.

ahh right, thank you

The way you work around this (apart from doing what you can to make the system as predictable as possible) is to accept that the data is noisy, and then working around it by doing multiple trials and following up with a statistical analysis on the results.

You can use the desired confidence to inform the warm-up, number of trials, benchmark duration, and so on.

If you end up with a multimodal distribution it can be worth tracking percentiles.

I did warm-up rounds, to prime caches and so on. I fiddled with statistics: IIRC simple average seemed to give the best results overall. I thought keeping the lowest n results would be the best, but that turned out to be wrong.

The results were indeed multimodal.

All I wanted was to have a reproducible benchmark to see how the code changes affected performance over time.

I've been doing some benchmarking recently and I've observed that on certain workloads, my CPU cores are not equivalent. (And this is an Intel CPU from 5 years ago, all cores are the same on paper.) In a particular very bad case I have a reproducible 50% performance difference between (physical) core 0 and core 5.

You might want to taskset to particular cores too, especially if you are looking at computational code with many almost-but-not-quite-random memory accesses.

Used to do this sort of thing for computations that needed to run in the 10 microsecond range (HFT stuff), circa 2008. Had very predictable results because:

a) language was not garbage collected (C++)

b) we avoided heap lock contentions in critical paths by pre-allocating object pools at startup

c) I/O operations were offloaded to separate threads, connected by mutex locked linked lists

d) processing thread was bound to its own CPU core

That's about as deterministic as we could get.

Did all of those trying to reduce the jitter. I think it was about the CPU clock not being stable.

Additionally I avoided core 0, because it was the noisiest and did some cgroups core pinning for the test workload.

There were zero page faults during the test runs and the CPU core was uncontested by other threads.

I think in 2008 CPUs were not so crazy about power and heat management.

Reply on news.ycombinator.com