Hacker News Reader

Comment by PragmaticPulp | Hacker News Reader

Thanks for sharing some programming-centric benchmarks using modern CPUs. I always wish that CPU reviewers would stop churning out pages of graphs with synthetic benchmark workloads and instead post things that actually matter, like compile times and other programming benchmarks.

The M1 processor is extremely impressive for the price and power envelope. However, I'm noticing a lot of people have taken Apple's marketing material a bit too literally and assumed that it somehow beats any and every desktop CPU out there, which is clearly not the case.

Moreover, as the author mentions: A significant portion of Apple's lead came from buying exclusivity on TSMC's 5nm process through the end of the year. It will be interesting to see how AMD stacks up as they roll out 5nm parts in the future, compared to Apple's scaled up M1 successors.

Exciting times. It's good to have some progress in CPU technologies again after years of Intel stagnation.

View on HN

Not everyone is a programmer though. Synthetic benchmarks don't just compute prime number or run mandelbrots. They test things like video compression, image processing, zip compression and decompression, etc. - things that most people do.

> Geekbench 5 measures the performance of your device by performing tests that are representative of real-world tasks and applications.

http://support.primatelabs.com/kb/geekbench/interpreting-gee...

I'd rather rely on Geekbench score than some programmer's compilation pipeline. Not saying programming benchmarks aren't important. Just that you cannot blame the reviewers testing comprehensively because your use case is specific.

And CPU/GPU companies have been caught multiple times in the past optimizing for these particular benchmarks. To the point where if you changed the name of the executable, the performance would drop significantly.

Ah yes the classic Quack.exe[1].

[1] https://techreport.com/review/3089/how-atis-drivers-optimize...

Has the situation changed since then? I mean, there's a reason why e.g. Nvidia drivers know and care about what games you have installed. I can also imagine quite a bit of rationale behind this, from a positive aspect - paying some knowledgeable people to extensively tweak settings by hand for popular games, and then having these settings become part of the driver seems somewhat reasonable. The above example on the other hand, probably was done for nefarious purposes. But how would we know from the outside, when it seems to have become standard practice, and for good reason?

Emery Berger [0] at UMass Amherst does great work in this, and it made me very wary of a lot of performance benchmarks. A big source no one considers is alignment, wherein things like the executable name as you mentioned or the username or anything else in the shell environment (which iirc is copied into the first few pages of a process). I think in one of the cases, one of Emery's students with a very long username ran the same application significantly slower because the username pushed the pages of code over a boundary.

[0] https://emeryberger.github.io/

I would highly recommend Emery Berger's talks [1] for anyone interested in benchmarking software. The alignment and code layout in memory that huayra mentions has shockingly large performance implications.

[1] https://www.youtube.com/watch?v=r-TLSBdHe1A

Do you have sources? I have never heard of any CPU manufacturer optimizing Geekbench scores.

Consumer grade CPUs are driven by a variety of things - marketing, brand loyalty, gaming benchmarks, reviewers such as Anantech and Extremetech, and yes, Geekbench scores. Datacenter processors are a whole another beast and driven by proprietary benchmarks by vendors.

Geekbench runs so many different tests, atleast a couple of dozens of tests. So, if CPU manufacturers are optimizing to those tests, its hard to believe that that's not a good thing and would apply to pretty much what an average user does in the first place.

There are multiple instances where phone manufacturers optimize the CPU performance when a specific app is loaded - e.g. Oneplus and Geekbench in the OnePlus 5 [1].

[1] https://www.xda-developers.com/oneplus-5-benchmark-cheating-...

There’s the saying that Intel used to employ one compiler engineer per line of SPECint/SPECfp source code; and I heard it from an Intel engineer!

(This is not something that should be taken literally, but rather seriously; a lot of money was hanging on these particular benchmarks at some point - when enterprises were buying serious enterprise hardware - so it made sense to invest in compiler tech to make sure you’re not wasting all the hard work done by hardware designers on stupid code generation.)

There's essentially no programming oriented benchmarks out there. There are plenty of benchmarks for other professional applications.

I find this disparity perplexing and the OP all the more refreshing for it.

> A significant portion of Apple's lead came from buying exclusivity on TSMC's 5nm process

It's hard to estimate how much of Apple's advantage came from the 5nm process, but we can get a guess by comparing the A13 and A14. One of the biggest differences between last year's A13 and this year's A14 is the 5nm process (vs 7nm+). The performance increase seems to be on the order of 15% [1], and Apple was able to fit in a lot more transistors- 11.8B vs. 8.5B, though we don't know how many of those transistors were spent on the CPU, and how many were spent in other areas like the GPU or ISP.

Seems like the additional transistors and lower power consuption were used to increase performance in a bunch of ways- higher clock speeds, more cache, and a bigger reorder buffer being the obvious improvements. Perhaps AMD can pull off a similar improvement when they switch processes.

[1] Spec2006 numbers from https://www.anandtech.com/show/16192/the-iphone-12-review/2

How much of that added density is used for what kind of performance and how much is used to prioritize efficiency is up to the chip designers -- for the M1 you get a couple cores that focus on performance and a couple cores that focus on efficiency, plus a lot of area for the GPU and a not insignificant area for the mysterious (to me at least) 16-core neural engine (aside: I wonder if e.g. the graphics pipeline is able to make use of that, as it seems like it should be able to perform matrix multiplications, and it'd be a bit of a waste if that just sat there most of the time).

So you'll probably see some variation but each time a process is scaled down to 0.7x of the previous size, you'll get smaller transistors that use less power individually and you could expect a "40% performance boost for the same amount of power and a 50% reduction in area" (according to https://semiengineering.com/5nm-vs-3nm/)

90 nm (2003) * 0.7 = 63 nm

65 nm (2005) * 0.7 = 45.5 nm

45 nm (2007) * 0.7 = 31.5 nm

32 nm (2009) * 0.7 = 22.4 nm

22 nm (2012) * 0.7 = 15.4 nm

14 nm (2014) * 0.7 = 9.8 nm

10 nm (2016) * 0.7 = 7 nm

7 nm (2018) * 0.7 = 4.9 nm

5 nm (2020) * 0.7 = 3.5 nm

To me, it's a bit of a miracle that Intel is still able to sort of compete on mostly 14 nm nodes, but maybe that's because "node size" basically just means "smallest feature size", and their 14 nm or new 10 nm process is a little better than e.g. competing 14/10 nm processes, or maybe their chip designs just prioritize different things that have a decent real world effect (e.g. Intel CPUs have AVX-512, but AMD CPUs don't).

In case anyone is wondering.

1. The 0.7x comes with 50% reduction in area. Where if you times 0.7 by 0.7 you get 0.49.

2. The original naming of Intel 14nm was actually 15nm. But Intel changed it which was part of the reason why it caused delays.

3. And Intel original 10nm naming was 11nm.

4. The node naming now mostly follows TSMC, the node after 3nm is 2nm, then 1.4nm, then 1nm, then 0.8nm.

minor correction: the geometric mean of the uplift of the specfp and specint benchmarks is 21.3%, which is perhaps notably better than 15% (up to interpretation, that).

The geekbench numbers comparing the iphone 12 pro to iphone 11 pro average to around 19% difference: https://browser.geekbench.com/ios-benchmarks/.

A few caveats: A compiler change made made the libquantum uplift (a part of the spec benchmark) huge, and non-representative of hardware changes. Conversely, the geekbench 5 multicore results are actually worse for the iphone 12 pro vs. 12; hinting that they're over-boosting the iphone 12 pro, so peak perf is likely actually a little better (from the hardware's perspective, not that an app can do much about it).

In any case, let's call it a 20% uplift with the same amount of threads, very respectable.

Another caveat: AMD may not be able to get the same 20% uplift from 5nm that apple does. Without any expertise in this matter I can only repeat what others claim, but I've heard people say that it's hard to make x86 wider (i.e extract more ILP) because the instructions are variable length, making lookahead more difficult. I guess we'll see next year with zen4 if that's true!

The only part of Apple's marketing material that even references desktops are related to integrated graphics.

So what's clear is that a lot of people are acting irrationally especially since the M1 is their low-end laptop CPU. At least wait for their mid-range laptop/desktop CPU scheduled for next year to make proper judgements.

Also let's not fail to mention the 5800x is a 600 dollar desktop CPU that consumes twice the maximum amount of power the M1 consumes....while idling

A lot of that is the IO die. The mobile equivalent will have 85-90% of the single-core performance with a 15W TDP. Apple still has better perf-per-watt, but it will be a much more apt comparison.

Also the MSRP is $450. It may be a bit more expensive now, but in a month or two the price will drop.

And the AMD part had 4x the memory at 64gb. That has to help a lot with garbage collected benchmarks.

This is why I really like phoronix, they show benchmarks aimed towards developers.

Here is a litle bit something:

https://www.phoronix.com/scan.php?page=article&item=ryzen-59...

"For the code compilation speed tests were compiking Apache, PHP, the Linux kernel, ImageMagick, GDB, LLVM, FFmpeg, and MPlayer."

Apple's material did say something about every desktop CPU?

Reply on news.ycombinator.com