We really need something like Phoronix, but without the fundamental flaws, such as benchmarking compilers instead of the OS, building stuff with wrong flags, and not even providing actually useful numbers - distribution, error bars etc.
That is not a flaw but simply benchmarking something else. "This would be the performance if you changed all those settings, fixed these errors etc." might be a nice academic exercise, but "this is the performance you get out of the box" seems much, much more relevant.
So you are not benchmarking "the OS" but rather the distribution, which includes whatever compiler version or flags they chose to use.
Honest question: If I take the exact same binary and shared libraries and run it on different distros, what am I actually measuring as differences? Kernel configs, filesystem, background processes? Does the distro even matter for that?
Benchmarks are vital. They can give you an estimate of how your application performs. They also often showcase how optimized a product is on different platforms.
But that's kind of obvious, so why are you asking this?
They consider difference due to compiler versions to be "wrong", differences in flags to be "wrong" etc. . So "the difference comes from the parts that hadn't changed", yeah which ones would you permit to change to consider the benchmark not "wrong"?
The problem with Phoronix v the build flags is not that they are different between systems/distros; quite the opposite: Phoronix builds the software in an unsupported way, ignoring system/distribution defaults. Which means, they measure software that’s miscompiled/misconfigured.