Hacker News Reader

Comment by systemvoltage | Hacker News Reader

Not everyone is a programmer though. Synthetic benchmarks don't just compute prime number or run mandelbrots. They test things like video compression, image processing, zip compression and decompression, etc. - things that most people do.

> Geekbench 5 measures the performance of your device by performing tests that are representative of real-world tasks and applications.

http://support.primatelabs.com/kb/geekbench/interpreting-gee...

I'd rather rely on Geekbench score than some programmer's compilation pipeline. Not saying programming benchmarks aren't important. Just that you cannot blame the reviewers testing comprehensively because your use case is specific.

View on HN

And CPU/GPU companies have been caught multiple times in the past optimizing for these particular benchmarks. To the point where if you changed the name of the executable, the performance would drop significantly.

Ah yes the classic Quack.exe[1].

[1] https://techreport.com/review/3089/how-atis-drivers-optimize...

Has the situation changed since then? I mean, there's a reason why e.g. Nvidia drivers know and care about what games you have installed. I can also imagine quite a bit of rationale behind this, from a positive aspect - paying some knowledgeable people to extensively tweak settings by hand for popular games, and then having these settings become part of the driver seems somewhat reasonable. The above example on the other hand, probably was done for nefarious purposes. But how would we know from the outside, when it seems to have become standard practice, and for good reason?

Do you have sources? I have never heard of any CPU manufacturer optimizing Geekbench scores.

Consumer grade CPUs are driven by a variety of things - marketing, brand loyalty, gaming benchmarks, reviewers such as Anantech and Extremetech, and yes, Geekbench scores. Datacenter processors are a whole another beast and driven by proprietary benchmarks by vendors.

Geekbench runs so many different tests, atleast a couple of dozens of tests. So, if CPU manufacturers are optimizing to those tests, its hard to believe that that's not a good thing and would apply to pretty much what an average user does in the first place.

There are multiple instances where phone manufacturers optimize the CPU performance when a specific app is loaded - e.g. Oneplus and Geekbench in the OnePlus 5 [1].

[1] https://www.xda-developers.com/oneplus-5-benchmark-cheating-...

There’s the saying that Intel used to employ one compiler engineer per line of SPECint/SPECfp source code; and I heard it from an Intel engineer!

(This is not something that should be taken literally, but rather seriously; a lot of money was hanging on these particular benchmarks at some point - when enterprises were buying serious enterprise hardware - so it made sense to invest in compiler tech to make sure you’re not wasting all the hard work done by hardware designers on stupid code generation.)

Emery Berger [0] at UMass Amherst does great work in this, and it made me very wary of a lot of performance benchmarks. A big source no one considers is alignment, wherein things like the executable name as you mentioned or the username or anything else in the shell environment (which iirc is copied into the first few pages of a process). I think in one of the cases, one of Emery's students with a very long username ran the same application significantly slower because the username pushed the pages of code over a boundary.

[0] https://emeryberger.github.io/

I would highly recommend Emery Berger's talks [1] for anyone interested in benchmarking software. The alignment and code layout in memory that huayra mentions has shockingly large performance implications.

[1] https://www.youtube.com/watch?v=r-TLSBdHe1A

There's essentially no programming oriented benchmarks out there. There are plenty of benchmarks for other professional applications.

I find this disparity perplexing and the OP all the more refreshing for it.

Reply on news.ycombinator.com