Zen, CUDA, and Tensor Cores, Part I: The Silicon
computerenhance.com
computerenhance.com
A top of the line Zen core is a powerful CPU with wide SIMD (AVX-512 is 16 lanes of 32 bit quantities), significant superscalar parallelism (capable of issuing approximately 4 SIMD operations per clock), and a high clock rate (over 5GHz). There isn't a lot of confusion about what constitutes a "core," though multithreading can inflate the "thread" count. See [1] for a detailed analysis of the Zen 5 line.
A single Granite Ridge core has peak 32 bit multiply-add performance of about 730 GFLOPS.
Nvidia, by contrast, uses the marketing term "core" to refer to a single SIMD lane. Their GPUs are organized as 32 SIMD lanes grouped into each "warp," and 4 warps grouped into a Streaming Multiprocessor (SM). CPU and GPU architectures can't be directly compared, but just going by peak floating point performance, the most comparable granularity to a CPU core is the SM. A warp is in some ways more powerful than a CPU core (generally wider SIMD, larger register file, more local SRAM, better latency hiding) but in other ways less (much less superscalar parallelism, lower clock, around 2.5GHz). A 4090 has 128 SMs, which is a lot and goes a long way to explaining why a GPU has so much throughput. A 1080, by contrast, has 20 SMs - still a goodly number but not mind-meltingly bigger than a high end CPU. See the Nvidia Ada whitepaper [2] for an extremely detailed breakdown of 4090 specs (among other things).
A single Nvidia 4090 "core" has peak 32 bit multiply-add performance of about 5 GFLOPS, while an SM has 640 GFLOPS.
I don't know anybody who counts tensor cores by core count, as the capacity of a "core" varies pretty widely by generation. It's almost certainly best just to compare TFLOPS - also a bit of a slippery concept, as that depends on the precision and also whether the application can make use of the sparsity feature.
I'll also note that not all GPU vendors follow Nvidia's lead in counting individual SIMD lanes as "cores." Apple Silicon, by contrast, uses "core" to refer to a grouping of 128 SIMD lanes, similar to an Nvidia SM. A top of the line M2 Ultra contains 76 such cores, for 9728 SIMD lanes. I found Philip Turner's Metal benchmarks [3] useful for understanding the quantitative similarities and differences between Apple, AMD, and Nvidia GPUs.
[1]: http://www.numberworld.org/blogs/2024_8_7_zen5_avx512_teardo...
[2]: https://images.nvidia.com/aem-dam/Solutions/Data-Center/l4/n...
A really big CPU is a couple of hundred cores, a big GPU is a few hundred SM / CUs. Some low power chips are 8 x64 cores and 8 CUs on the same package. All roughly lines up.
For decades the GPU was "special purpose" hardware dedicated to the math of screen graphics. If the type of larger-scale numerical computation that has been popularised with LLM's is now deemed "typical use", then the distinction may be becoming irrelevant (and even counterproductive from a software development perspective).
I think the APU designs from AMD are the transition pointing to the future. The GPU cores will gain increasing access to the raw hardware and the user interface until the CPU cores are optional and ultimately discarded.
In the Intel/AMD CPUs, the 512-bit width, i.e. 64 bytes or 16 FP32 numbers, matches the width of the cache line and the width of a DRAM burst transfer, which simplifies the writing of optimized programs. This SIMD width also provides a good ratio between the power consumed in the execution units and the power wasted in the control part of the CPU (around 80% of the total power consumption goes to the execution units, which is much more than when using narrower SIMD instructions).
Increasing the SIMD width more than that in CPUs would complicate the interaction with the cache memories and with the main memory, while providing only a negligible improvement in the energy efficiency, so there is no reason to do this. At least in the following decade it is very unlikely that any CPU would increase the SIMD width beyond 16 FP32 numbers per operation.
On the other hand, the AMD GPUs before RDNA had a SIMD width of 64 FP32 numbers, but the operations were pipelined and executed in 4 clock cycles, so only 16 FP32 numbers were processed per clock cycle.
RDNA has doubled the width of the SIMD execution, processing 32 FP32 numbers per clock cycle. For this, SIMD instructions with a reduced width of 32 FP32 have been introduced, but they are executed in one clock cycle versus the old 64 FP32 instructions that were executed in four clock cycles. For backwards compatibility, RDNA has kept 64 FP32 instructions, which are executed in two clock cycles, but these were not recommended for new programs.
RDNA 3 has changed again all this, because now sometimes the 64 FP32 instructions can be executed in a single clock cycle, so they may be again preferable instead of the 32 FP32 instructions. However it is possible to take advantage of the increased width of the RDNA 3 SIMD execution units also when using 32 FP32 instructions, if certain new instructions are used, which encode double operations.
So the AMD GPUs have continuously evolved towards wider SIMD execution units, from 16 FP32 before RDNA, to 32 FP32 in RDNA and finally to 64 FP32 in RDNA 3.
The distance from CPUs has been steadily increasing, there is no convergence.
It didn’t really work out, in part because it was too far from a regular old Xeon to run your normal code well without optimizing. On the other side, Intel couldn’t keep up with NVIDIA on the sort of metrics people care about for these compute accelerators: memory bandwidth mostly. If you are going to have to refactor your whole project anyway to use a compute accelerator, you probably want a pretty big reward. It isn’t obvious (to me at least) if this is the result of the fact that the Phi cores, simple as they were, were still a lot more complex than a GPU “core,” maybe the design just had various hidden bottlenecks that were too hard to work out due to that complexity. Or if it is because Intel just wasn’t executing very well at the time, especially compared to NVIDIA (it is Intel’s dark age vs NVIDIA’s golden age, really). The programmer’s “logical or” joke is a possible answer here.
But, you can’t do everything in parallel. It is a shame the Phi didn’t survive into the age where Intel is also doing big/little cores (in a single chip). A big Xeon core (for latency) surrounded by a bunch of little Phi cores (for throughput) could have been a really interesting device.
To be fair, yes most GPUs still do come with things like texture units, video transcode units, ray tracing cores, and a framebuffer and video output. But that’s already changing and you have, for example, some GPUs with ray tracing, and some without that are more designed for data centers. And you don’t have to use the graphics functionality; for GPU supercomputers it’s common for the majority of GPU nodes to be compute-only.
In the mean time we now have CPUs with embedded GPUs (aka iGPUs), GPUs with embedded CPUs, GPUs that come paired with CPUs and a wide interconnect (like Nvidia Grace Hopper), CPU-GPU chips (like Apple M1), and yes CPUs in general have more and more SIMD.
It’s useful to have a name or a way to distinguish between a processor that mostly uses a single threaded SISD programming model and has a small handful of hardware threads, versus a processor that uses a SIMD/SIMT model and has tens of thousands of threads. That might be mainly a question of workloads and algorithms, but the old line between CPU and GPU is very blurry, headed towards extinction, and the “graphics” part has already lost meaning.
[1] https://en.wikipedia.org/wiki/General-purpose_computing_on_g...
In many systems-on-a-chip, the 3 blocks, GPU in the strict sense, video decoder/encoder and display controller may even be licensed from different IP vendors and then combined in a single chip. Also in the CPUs with intgrated GPU, like Intel Lunar Lake and AMD Strix Point, these 3 blocks can be found in well separated locations on the silicon die.
What belongs into the GPU proper from the graphics-specific functions, because these perform operations that are mixed with the general-purpose computations done by shaders, are the ray-tracing units, the texture units and the rasterization units.
A minor quibble amidst your good comparison above ;)
For a zen5 core, we have 16-wide SIMD with 4 pipes; 2 are FMA (2 flop), and 2 are FADD @ ~5GHZ. I math that out to 16 * 6 * 5 = 480 GFLOP/core... am I missing something?
So a 9950X can do 256 FMA + 256 FADD for FP64 or 512 FMA + 512 FADD for FP32, per clock cycle.
Using FP32, because it can be compared with the GPUs, there are 1536 Flop per clock cycle, therefore about 6 FP32 Tflop/s @ 4 GHz for a 9950X (around 375 FP32 Gflop/s per core, but this number is irrelevant, because a single active core would go to a much higher clock frequency, probably over 5 GHz). For an application that uses only FMA, like matrix multiplication, the throughput would drop to around 4 FP32 Tflop/s or 2 FP64 Tflop/s.
The values for the FP32 throughput are similar to those of the best integrated GPUs that exist at this time. Therefore doing graphics rendering on the CPU on a 9950X might be similarly fast to doing graphics rendering on the iGPU on the best mobile CPUs. Doing graphics rendering on a 9950X can still leverage the graphics and video specific blocks contained in the anemic GPU included in 9950X, whose only problem is that it has a very small number of compute shaders, but their functions can be augmented by the strong CPU.
Granite Ridge core = Zen 5 core.
Agreed. Some quibbles about the slipperiness of the concept.
flops are floating point operations. IMO it should not be confusing at all, just count single precision floating point operations, which all devices can do, and which are explicitly defined in the IEEE standard.
Half precision flops are interesting but should be called out for the non-standard metric they are. Anyone using half precision flops as a flop is either being intentionally misleading or is confused about user expectations.
On the other side, lots of scientific computing folks would rather have doubles, but IMO we should get with the times and learn to deal with less precision. It is fun, you get to make some trade-offs and you can see if your algorithms are really as robust as you expected. A free 2x speed up even on CPUs is pretty nice.
> and also whether the application can make use of the sparsity feature
Eh, I don’t like it. Flops are flops. Avoiding a computation exploiting sparsity is not a flop. If we want to take credit for flops not executed via sparsity, there’s a whole ecosystem of mostly-CPU “sparse matrix” codes to consider. Of course, GPUs have this nice 50% sparse feature, but nobody wants to compete against PARDISO or iterative solvers for really sparse problems, right? Haha.
Or they could be measured in bits per second.
Actually I’m pretty interested in figuring out if we can use them for numerical linear algebra stuff, but I think it’d take some doing.
Depends on what you're comparing with what, and the context, of course.
Casey is doing education, so that people learn how best to program these devices. A mere comparison of TFLOPS of CPU vs GPU would be useless towards those ends. Similarly, just a bare comparison of TFLOPS between different GPUs even of the same generation would mask architectural differences in how to in practice achieve those theoretical TFLOPS upper bounds.
I think Casey believes most people don't know how to program well for these devices/architectures. In that context, I think it's appropriate to be almost dismissive of TFLOPS comparison talk.
Agreed
It's a classic question of 'what is an "embarrassingly parallel" problem' (e.x. physics calculations, image rendering, LLM image creation or textual content generation) or not.
https://old.reddit.com/r/hardware/comments/baajes/rtx_adds_1...
This could have changed somewhat in newer releases, but probably not too drastically, since NVIDIA has never really increased raw ray performance since the 20-series launch. And while there have been a few raytracing features around the edges, raster and cache have been bumped significantly too (notably, ampere got dual-issue fp32 pipelines... which didn't really work out for NVIDIA that well either!) so honestly there's a reasonable chance it's slightly less in subsequent architectures.
Perhaps, it could be more interesting to compare without L2 cache.
The GPU cores (in the classic sense, i.e. not what NVIDIA names as "cores") also include cache memories and also local memories that are directly addressable.
The only confusion is caused by the fact that first NVIDIA, and then ATI/AMD too, have started to use an obfuscated terminology where they have replaced a large number of terms that had been used for decades in the computing literature with other terms.
For maximum confusion, many terms that previously had clear meanings, like "thread" or "core", have been reused with new meanings and ATI/AMD has invented a set of terms corresponding to those used by NVIDIA but with completely different word choices.
I hate the employees of NVIDIA and ATI/AMD who thought that it is a good idea to replace all the traditional terms without having any reason for this.
The traditional meaning of a thread is that for each thread there exists a distinct program counter a.k.a. instruction pointer, which is used to fetch and execute instructions from a program stored in the memory.
The traditional meaning of a core is that it is a block that is equivalent with a traditional independent processor, i.e. equivalent with a complete computer minus the main memory and the peripherals.
A core may have only one program counter, when it can execute a single thread at a time, or it may have multiple program counters (with associated register sets) when it can execute multiple threads, using either FGMT (fine-grained multithreading) or SMT (simultaneous multithreading).
The traditional terms were very clear and they have direct correspondents in GPUs, but NVIDIA and AMD use other words for those instead of "thread" and "core" and they reuse the words "thread" and "core" for very different things, for maximum obfuscation. For instance, NVIDIA uses "warp" instead of "thread", while AMD uses "wavefront" instead of "thread". NVIDIA uses "thread" to designate what was traditionally named the body of a "parallel for" a.k.a. "parallel do" program structure (which when executed on a GPU or multi-core CPU is unrolled and distributed over cores, threads and SIMD lanes).
There are Zen processors that use 7 cores per CCD, e.g. Epyc 7663, 7453, 9634.
The difference between Ryzen and Epyc is the I/O die. The CCDs are the same so that's presumably where they go.
Another reason you might not see this on the consumer chips is that they have higher base clocks. If you have a CCD where one core is bad and another isn't exactly bad but can't hit the same frequencies as the other six, it doesn't take a lot of difference before it makes more sense to turn off the slowest than lower the base clock for the whole processor. 6 x 4.7GHz is faster than 7 x 4.0GHz, much less 7 x 2.5GHz.
In theory you could let that one core run at a significantly lower speed than the others, but there is a lot of naive software that will misbehave in that context. Whereas the base clock for the Epyc 9634 is 2.25GHz, because it has twelve 7-core CCDs so it's nearly 300W, and doesn't want to be nearly 1300W regardless of whether or not most of the cores could do >4GHz.
The extra cores in the 73F3 aren't necessarily bad, they're disabled so the others can have their L3 cache and so they can pick the two cores from each CCD that hit the highest clock speeds. Doing that is expensive, especially if the other cores aren't all bad, but then you get better performance per core. Which some people will pay a premium for, so they offer models like that even if yields are good and there aren't that many CCDs with that many bad cores.
At which point your premise is invalid because processors are being sold with cores disabled for performance reasons rather than yield reasons.
what or where does that follow from? One can take a CCD with 2+ cores and pin a process to a set (of the fastest) cores based on profiling the cores and those 2+ cores could use the L3 cache as needed; disabling cores at the hardware level is the waste as if they were not disabled, then that would allow other processes to be able to benefit from more than 2 cores to run when desired. The latter point of disabling cores for "better [frequency] performance per core Which some people will pay a premium for" is dubious especially for the Epyc server line. If that were true, then there should at least be 4 core or fewer SKUs for desktop Ryzen variant where apps like games are more likely to benefit from the higher clock.
You're assuming that the buyer knows how to do this and wants to do it themselves, rather than buying a piece of hardware which is configured from the factory to do it for them, and getting a modest discount over the processor with more of the cores operational because some of the cores they weren't going to use anyway might be defective.
> disabling cores at the hardware level is the waste as if they were not disabled, then that would allow other processes to be able to benefit from more than 2 cores to run when desired.
This is exactly the thing some buyers want to avoid. Many applications will spawn a thread for each hardware thread, but each thread for each application will consume shared resources like L3 cache and memory bandwidth. That adds up fast if you have 96 cores per socket. When these are your bottleneck you don't want to spend your time changing the defaults in every application to not do this, you just want a processor with fewer cores and more L3 cache.
> The latter point of disabling cores for "better [frequency] performance per core Which some people will pay a premium for" is dubious especially for the Epyc server line. If that were true, then there should at least be 4 core or fewer SKUs for desktop Ryzen variant where apps like games are more likely to benefit from the higher clock.
Base clocks are limited by power. AM4 supplies up to 170W, which is sufficient to run 12 cores at a base clock of 4.7GHz and is approximately the limit of the architecture rather than the socket. The Zen4 processors with fewer cores than that don't have higher base clocks, they have lower TDPs.
SP5 supplies up to 400W, but supports up to 12 Zen4 CCDs, which is 96 cores. If you enable all 96 cores then you have ~4W/core rather than ~14W/core, and on top of that the Epyc I/O die is bigger and consumes more power than the I/O die for AM5. The highest base clock with 96 Zen4 cores is the 9684X at 2.55GHz and the full 400W. So if you want a >4GHz base clock you have to reduce the core count, and doing this by reducing the number of cores per CCD leaves you with more L3 cache than reducing the number of CCDs. There is no reason to do the same for AM5 because the lower core count processors already aren't limited by socket power.
The big exception is that the new Zen 5 Strix Point chip has a 4 core CCX for the non-C cores. I think the Zen 4 based Z1 has a similar setup but don't remember and couldn't quickly find the actual information to confirm.