CPU-Z's Inadequate Benchmark | Hacker News Reader

CPU-Z's Inadequate Benchmark

chipsandcheese.com

69 pointspicture37 comments

CPU-Z's benchmark is designed to give results that reflect the "average IPC" across many generations of processor without privileging specific architectural and microarchitectural choices. In fact, the current benchmark was written because the old one accidentally privileged Ryzen due to microarchitectural quirks in the ALU [0]. On-CPU procedural generation didn't end up being the major thing people thought it would be, so it's not all that representative of modern applications, but it's fine if your goal is to make a reasonably fair social media leaderboard of "CPU performance numbers". It's hard to call it inadequate when that seems to be the intended purpose.

[0] https://www.cpuid.com/news/51-cpu-z-1-79-new-benchmark-new-s...

>CPU-Z's benchmark is designed to give results that reflect the "average IPC" across many generations of processor without privileging specific architectural and microarchitectural choices.

If it was the case, it should have limited itself to the top eight instructions that matter[0], rather than the relatively esoteric SSE.

CPU-Z is just a terrible benchmark. With this article, now we can point somewhere every time someone treats CPU-Z's result as if it was meaningful. Thus I am thankful for it.

0. https://www.anandtech.com/show/16762/an-anandtech-interview-...

SSE2 is the latest instruction set supported by all Intel/AMD processors from the past 5ish years. It's not remotely esoteric.

As for the "top 8 instructions that matter", you'd be better served by Agner Fog's documentation than any experimental benchmark you can run if that's what you're focused on.

I'm not saying CPU-Z is a useful benchmark to tell you how your system is going to run Excel or some HPC workload. I'm only saying it's not a terrible way to rank CPUs for a fun and is relatively fair as such things go.

I'm pretty sure the current 32-bit build of CPU-Z does not use any modern SIMD extensions for its benchmark, only pure unadulterated x87 FP instructions. And the results can be hilariously confusing unless you know what is going on.

I use it as a tool to show that a large part of IPC gains over the past 20 years come from new instructions sets and recompiling old code is a must since frequency gains have pretty much peaked.

To be fair, if I just throw some code at my compiler that also won't use and modern SIMD extensions. You either have to write extra code paths, which in most software only happens in imported libraries; or you have to change your compiler options to exclude older processors.

CPU-Zs benchmark is mostly confusing in that it's a benchmark of run-of-the-mill poorly optimized software. In a way that's bad because power users tend to care more about the performance of well-optimzed code; on the other hand seeing the performance difference in naive code is also useful.

5 years? More like 20 years. SSE2 is supported by all 64-bit Intel/AMD CPUs released since about 2005.

20 years is right.

SSE2 has been introduced in Pentium 4, in November 2000, then AMD has made it a standard part of x86-64, with the first Opteron launched in April 2003.

It was SSE2 which made obsolete the 8087/80387 ISA.

The Pentium G6405 was launched in 2021 and only supports SSE:

https://ark.intel.com/content/www/us/en/ark/products/201901/...

That literally says it supports SSE4.2, which is newer than SSE2.

SSE2 is supported by all x64 processors (SSE and SSE2 are literally part of the AMD64/x86-64 spec), so in terms of support, I don't see how that could be "relatively esoteric"...

And as a consequence if your compiler does autovectorization it will use SSE or SSE2, unless you specificly add compiler flags that make your binary incompatible with older processors (or you have an unusually smart compiler that adds seperate code paths for different CPU architectures)

Those top 8 instructions mentioned by Keller (load and store, integer add and subtract and compare, jump and call and return) are those that matter in general-purpose and interactive applications (like an office suite or Internet browsing), whose work consists mostly of copying and searching data, with only a little amount of computations.

Nevertheless, there are also people like myself, who use computers for scientific computations and for computer-aided design, for whom any computer that is optimized only for the execution of those 8 instructions would be a piece of useless junk that would be thousands of times slower than a computer with Intel or AMD CPUs, which dedicate considerable hardware resources for the execution of double-precision floating-point fused multiply-add and division and square root.

However the SSE/SSE2/SSE3/SSSE3/SSE4.1/SSE4.2 instructions have been obsolete for more than a decade, despite the stubbornness of Intel of not providing AVX in its cheapest SKUs until a year ago, so I do not care about any benchmark that tests SSE instructions, because it is completely irrelevant for the performance of the CPU for high-performance computing.

SSE2 it's used as the lowest common ground for software, for instance, Mozilla.

And for older i686 Atom CPU's, SSSE3 optimizations on some media players or emulators are a must have.

>which dedicate considerable hardware resources for the execution of double-precision floating-point fused multiply-add

I hope you realize that fma w/single rounding step is a requirement in 2008's revision of IEEE 754.

Thus, it's widely available, as current CPUs implementing double-precision floating point are IEEE 754 compliant.

> it is completely irrelevant for the performance of the CPU for high-performance computing.

No matter what choices you make, a single benchmark will be "completely irrelevant" for someone. Wanting to neatly capture performance in a single number is very understandable, but unfortunately not something that really works well, and you'll always end up having to make trade-offs.

The thing about CPU-Z’s benchmark is it highlights that all 100% CPU stress tests (and workloads) are not created equal. Your overclock may be able to survive CPU-Z and Cinebench, but will fall to its knees with Prime95. All are taxing the CPU to 100% in different ways.

I can't recall ever seeing the CPU-Z benchmark used anywhere. Didn't even know it existed until just now.

And yet, CPU-Z, is still a fantastic tool to see what ram your system has, what motherboard, what processor, especially in am old system you may want to upgrade. Just to know what family of processor you have, is extremely useful. It's a great tool at what it does well.

I would take whatever information they show with a grain of salt as it seems to simply relay superficial information the system BIOS/UEFI reports without verification.

For example, a stick or channel of system RAM may fail training at boot and end up being silently disabled. CPU-Z would blindly report all memory channels being present despite Windows task manager showing correctly that some of the memory are unavailable.

I love that it's a small portable exe. I don't want to install some app to view some info one time.

Alternatively there is HWinFO (which also has a portable version).

https://www.hwinfo.com

> Just to know what family of processor you have, is extremely useful.

Just start task manager, ctrl+shift+esc...

(BTW to me owning a computer and not knowing what's inside is like not knowing what brand of car you own [never owned a car personally but I imagine this analogy works for most].)

After owning computers for 40 years and helping friends and family with computers just as long, this gets harder and harder to keep track of. Especially now when there is no real need for a new computer unless something breaks. Then I need to know what I have and what I can replace it with.

So now I run the tools and save the info on my harddrive to be able to look up before I go home to someone to fix a computer.

> owning a computer and not knowing what's inside is like not knowing what brand of car you own

It's more like not knowing what type of engine your car has.

Most people don't know or care what type of engine/cpu they have, they just know the brand (and maybe model) of the entire machine.

I'm a tech person and I don't even know what CPU is in my computer right now. I know I have a thinkpad x1. I know I paid for the i7 upgrade. I know where to find the CPU model if needed. But day-to-day I just don't care.

Every time I use it I am irritated that it won't let me copy the name of my graphics card or other components and I have to retype them manually.

"They should run typical desktop applications"

This by itself is a terrible benchmark. What "typical desktop application" 1) does not slaughter your RAM usage, 2) does not require a high-end spec computer to run without hiccuping constantly, 3) does not open 10 processes for 1 application.

I have Adobe Creative cloud, I still use Photoshop CS6 because of how bad new Adobe products use resources. Same with Google Chrome - I only QA with it because it's a TERRIBLE daily browser resource-wise. Excel? F*ck I hate opening large data-sets with that.

Now if you steer away from business applications and look at gaming (what most CPU-Z users benchmark for) - look at the game engines or the games. 99% of games are absolute trash at utilizing resources correctly.

Valve and some Blizzard titles were the only companies that really cared and put forth effort behind optimizing their games so lower end PC's can play them, hence their explosive popularity world-wide.

Games and desktop applications are what people actually use.

Thus there's strong practical sense in benchmarking them.

Games and applications get updates that change the performance, not to mention other effects an performance by drivers etc.

"They should run typical desktop applications and monitor performance counters to see what typical branch and memory footprints look like. These types of data should be used to inform benchmark design for future CPU-Z versions."

You selectively quoted them to rant about something they weren't talking about.

The entire article was about how the design of their benchmark doesn't reflect how applications people use work.

~100% of games don't fit into the L1 cache of a CPU.

It's the weekend, so I am just gonna say that a Ryzen 7800x3d has 512KB of L1 cache. Plenty of games would fit. Not even talking about those tiny games that generate everything.

L1 cache isn’t shared between cores. One core in the 7800X3D only has 64k L1 cache.

Of the 64 kB L1 cache, only 32 kB is available for data.

Despite the fact that AMD writes "512 kB" in its spec sheet, this number does not matter, because it multiplies the cache size with the number of cores.

The Ryzen 7000 CPUs have 32 kB of L1 instruction cache, 32 kB of L1 data cache and 1 MB of L2 cache per core, and 7800X3D also has 96 MB of L3 cache shared by the 8 cores.

These numbers determine at which cache level the data used by an execution thread can fit.

No modern game uses data that fits in the 32 kB of L1 data cache, and the game code also does not fit in the 32 kB instruction cache.

Next you're gonna tell me that Bogomips aren't really MIPS.

I favor emBench[0].

Compile for the target platform, run, then note result.

0. https://www.embench.org/

emBench is designed for embedded platforms, not desktop systems; one of the design criteria is that the entire test suite should compile to under 64 kB of code, and should use under 64 kB of RAM. This has the effect of making the benchmark absolutely trivial for desktop-class CPUs -- most tests will run entirely in cache, placing no load at all on the memory controller, and features like branch prediction will be unfairly effective.

>most tests will run entirely in cache, placing no load at all on the memory controller,

Exactly. Like CPU-Z.

Yet unlike CPU-Z, it is not an extremely biased sse microbenchmark, but does instead provide decent coverage of a CPU's functionality.

The benchmarks within emBench include common code like gzip or jpeg crunch/decrunch routines, which are frequent stress points in real world applications.

Reply on news.ycombinator.com