SPEC says Intel's compiler used unfair optimizations to boost performance | Hacker News Reader

SPEC says Intel's compiler used unfair optimizations to boost performance

tomshardware.com

33 pointsksec9 comments

Cheating on SPEC benchmarks? Surely that has never happened before...

"We conclude this paper by citing cases [of] compiler optimizations specifically designed for SPEC programs, in which performance decreases drastically or the computed results are incorrect if the compiled program does not exactly match the SPEC benchmark program. These findings show that the fairness and value of the popular SPEC benchmarks are questionable."[1]

"Issue: Special library for a specific benchmark... Issue: Pattern matching for the inner loop of a specific benchmark... Issue: Compilers optimize too much for the SPEC CPU benchmarks... Issue: The compiler optimization parameters (flags) used are unrealistic..."[2]

See also: Goodhart's Law[3].

[1] N. Mirghafori, M. Jacoby, D. Patterson, "Truth in SPEC benchmarks", ACM SIGARCH Computer Architecture News, 1995

[2] R. Weicker, "SPEC OSG Benchmarks: History, Use and Current Issues", in "Performance Evaluation and Benchmarking with Realistic Applications", R. Eigenmann (Ed.), 2001

[3] https://en.wikipedia.org/wiki/Goodhart%27s_law

"SPEC has ruled that the compiler used for this result was performing a compilation that specifically improves the performance of the 523.xalancbmk_r / 623.xalancbmk_s benchmarks using a priori knowledge of the SPEC code and dataset to perform a transformation that has narrow applicability."

Does this mean that if I use their compiler in production I might get worse performance because they apply this optimization to my own code as well?

It’s probably designed to recognize a very specific code pattern in SPEC, so it’s very unlikely to affect any other code. If it wasn’t so hyper targeted to only kick in for those two benchmarks, then there’d be no issue: it’d just be a common (mis)optimization that happens to help SPEC.

My read is that they do what Volkswagen did with the emissions scandal: they detect when they're compiling code for the benchmark, and compile it in a way that makes the benchmark code (specifically) faster. That optimization wouldn't generalize to "real" code because the optimization relies on the specific code and data ("data" being fixed non-code inputs processed at runtime) being processed by the benchmark to have an effect. If you know based on prior knowledge that, for instance, code that uses a signed integer will always only handle positive integers, you might cheat by optimizing it to be unsigned integers. But a compiler couldn't possibly know that in advance because it can't see the data the code processes at compile time.

For your code to benefit, it would have to do the same things the benchmark is computing and have the same shape of data as the data included with the benchmark suite.

  According to Phoronix, the optimization could boost performance in SPECint by 9% overall. The publication also notes that versions 2022.0 to 2023.0 of the Intel oneAPI Compiler are impacted, meaning most of the now-invalidated results were run in 2022, largely on Sapphire Rapids CPUs.

9% bump for free.

Surely this says more about the value of SPEC benchmarks than it does about the “fairness” of the compiler.

Did any1 manage to find the PR which adds/removes this optimization?

It’s a closed source compiler.

Is there a way to see the optimization?

Reply on news.ycombinator.com