Comment by GeekyBear | Hacker News Reader

Comment by GeekyBear | Hacker News Reader

Parent Full thread

SPEC has been the industry standard benchmark for comparing the performance of systems using different instruction sets for decades now.

Traditionally, Anandtech would have been the first media outlet to publish the single core and multicore integer and floating point SPEC test results for a new architecture, but hopefully some trusted outlet will take up the burden.

For instance, Anandtech's Zen 5 laptop SKU results vs the M3 from the end of July:

> Even Apple's M3 SoC gets edged out here in terms of floating point performance, which, given that Apple is on a newer process node (TSMC N3B), is no small feat. Still, there is a sizable deficit in integer performance versus the M3, so while AMD has narrowed the gap with Apple overall, they haven't closed it with the Ryzan AI 300 series.

https://www.anandtech.com/show/21485/the-amd-ryzen-ai-hx-370...

Zen 5 beat Core Ultra, but given that Zen 5 only edged out the M3 in floating point workloads, I wouldn't be so quick to claim the M4 doesn't outperform Zen 5 single core scores before the test results come out.

just considering the number of simple calculations a CPU can compute isn't a very good comparison. apple's chips are using an ARM architecture, which is a Reduced Instruction Set Computer (RISC) setup, vs x86-64, which is a Complex-ISC (CISC).

The only good comparison is to judge a variety of real world programs compiled for each architecture, and run them.

https://en.wikipedia.org/wiki/SPECint talks about a dozen programs they selected for single-core logic and discrete math, including gcc and bzip2 (there are more than a dozen others using floats).

Over time, RISC and CISC borrowed from each other: https://cs.stanford.edu/people/eroberts/courses/soco/project...

> The only good comparison is to judge a variety of real world programs compiled for each architecture, and run them.

I'm guessing that you don't realize that you are describing SPEC?

It's been around since the days when every workstation vendor had their own bespoke CPU design and it literally takes hours to run the full set of workloads.

From the same Anandtech article linked above:

> SPEC CPU 2017 is a series of standardized tests used to probe the overall performance between different systems, different architectures, different microarchitectures, and setups. The code has to be compiled, and then the results can be submitted to an online database for comparison. It covers a range of integer and floating point workloads, and can be very optimized for each CPU, so it is important to check how the benchmarks are being compiled and run.

More info:

> SPEC is the Standard Performance Evaluation Corporation, a non-profit organization founded in 1988 to establish standardized performance benchmarks that are objective, meaningful, clearly defined, and readily available. SPEC members include hardware and software vendors, universities, and researchers.

SPEC was founded on the realization that "An ounce of honest data is worth a pound of marketing hype".

https://www.spec.org/cpu2017/Docs/overview.html

took at look at these benchmarks. they appear to be using some extremely antiquated code, and workloads, that do not take advantage of any of the advanced features and instruction introduced over the past 15-20 years in the x86-64 architecture.

additionally, the only updates they appear to have made in the last 5+ years involve optimizing the suite for Apple chips.

thus, it leaves out massive parts of modern computing, and the (many) additions to x86-64 that have been introduced since the 00s.

i'd encourage you to look into the advancements that have occurred in SIMD instructions since the olden days, and the way in which various programs, and compilers, are changed to take advantage of them

ARM is nice and all, but the benchmark you've linked appears to be some extremely outdated schlock that is peddled for $1000 a pop from a web page out of the history books. Really. Take a look through what the benchmarks on that page are actually using for tooling.

I'd consider the results valid if they were calculated using an up to date, and maintained, toolset, like that provided by openbenchmarking.org (the owner of which has been producing some excellent ARM64 vs Intel benchmarks on various workloads, particularly recently).

Does optimising the tests towards one architecture seem like a fair way of testing?

It's optimizing as in previously it had far less attention.

> the only updates they appear to have made in the last 5+ years involve optimizing the suite for Apple chip

How do you theorize that generic C or C++ code that you compile using GCC has been "optimized for an Apple chip"?

Feankly, it's impossible to take any of this comment seriously.

Reply on news.ycombinator.com