Analyzing Geekbench 6 under Intel's BOT | Hacker News Reader

Analyzing Geekbench 6 under Intel's BOT

geekbench.com

41 pointshajile15 comments

This suggests the checksum is used to identify whether the binary is known to BOT, and thus whether BOT can optimize the binary.

I do wonder what this "optimize" step actually entails; does it just replace the binary with one that Intel themselves carefully decompiled and then hand-optimised? If it's a general "decompile-analyse-optimise-recompile" (perhaps something similar to what the https://en.wikipedia.org/wiki/Transmeta_Crusoe does), why restrict it?

FYI, Geekbench 6 already optimizes for AVX512. Intel just optimizes it even more for them.

I'll take the side of Geekbench here. There is no reason for Intel to optimize a benchmark tool except to cheat. The goal of GB is to test how typical applications run, not the maximum performance possible under ideal scenarios.

Post link optimization (PLO) tools have been around for quite a while. In particular, Meta’s BOLT (fully upstream in LLVM) and Google’s Propeller (somewhat upstream in LLVM, but fully open source) have been around for 5+ years at this point.

It doesn’t seem like Intel’s BOT delivers more performance gains, and it is closed source.

Question: do those vectorize code as in the example here? I was of the understanding they performed a more limited subset of optimizations.

Propeller can’t really do many instruction level modifications due to how it works (constructs a layout file that then gets passed to the linker).

BOLT could do this, but does not as far as I’m aware.

Most of vectorization like this is also probably better done in a compiler middle end. At least in LLVM, the loop vectorizer and especially the SLP Vectorizer do a decent job of picking up most of the gains.

You might be able to pick up some gains by doing it post-link at the MC level, but writing an IR level SLP Vectorizer is already quite difficult.

Intel BOT seems to be patches for specific binaries (hence why they didn't see a difference for Geekbench 6.7), unlike BOLT/Propeller which are for arbitrary programs. The second image from their help page [1] showcases this.

[1] https://www.intel.com/content/www/us/en/support/articles/000...

Applying targeted binary patches shouldn't take 40 seconds... unless that's also a fake "so it looks like it's working really hard" delay.

I might be thinking of a different project then...

I swore Intel had their own PLO tool, but I can only find https://github.com/clearlinux/distribution/issues/2996.

Found it. It was https://www.phoronix.com/news/Intel-Thin-Layout-Optimizer.

It was open source, but has since been deprecated.

To me, the whole thing sounds like cheating in benchmarks.

Intel built a tool that will only activate for a specific benchmark - but not for real-world software which accomplishes similar things - and then the tool will replace generic bytecode with a (most likely) handcrafted and optimized variant for running this specific benchmark on this specific CPU. That means BOT will only boost the benchmark score, but not help at all with the end-user workflows that the benchmark is trying to emulate. Thereby, Intel's BOT makes the benchmark score misleading, which is why Geekbench is flagging them.

quack3.exe again in a way. If it's been done for years on GPU shaders, then why not CPU code?

While highly specific optimisations might give you a tiny bit of advantage, the main boost here is vector code which would work on any processor supporting the instructions. They could have looked at the vendor bits and use those to flag for optimization in any cpu but they didn't and limited it to a small subset of programs and cpus. It tingles the "PR above all else must have highest score" sense.

> BOT optimizations are poorly documented, aggressive in scope, and damage comparability with other CPUs. For example, BOT allows Intel processors to run vector instructions while other processors continue to run scalar instructions. This provides an unfair advantage to Intel

Wait until they hear about branch predictors.

The thing is BOT only applies to a handful of applications. So Geekbench scores with BOT applied aren't as representative.

Can we also end user tune our cpus for specific tasks we do?

Reply on news.ycombinator.com