Hacker News Reader

Comment by JonChesterfield | Hacker News Reader

The CPUs are good at serial code and GPUs are good at parallel code is kind of true but something of an approximation. Assume equivalent power budget in the roughly hundreds of watts range, then:

A CPU has ~100 "cores" each running one (and-a-hyperthread). independent things, and it hides memory latency by branch prediction and pipelining.

A GPU has ~100 "compute units", each running ~80 independent things interleaved, and it hides memory latency by executing the next instruction from one of the other 80 things.

Terminology is a bit of a mess, and the CPU probably has a 256bit wide vector unit while the GPU probably has a 2048bit wide vector unit, but from a short distance the two architectures look rather similar.

View on HN

GPU has 10x the memory bandwidth of the CPU though, which becomes relevant for the LLMs where you essentially have to read the whole memory (if you're batching optimally, that is using all the memory either for weights or for KV cache) to produce one token of output.

GPUs also have 10x-100x FP/INT8 throughput watt-for-watt.

GPU also has 10x memory latency compared to CPU.

And memory access order is much more important that on CPU. Truly random access has very bad performance.

I’m always surprised there isn’t a movement toward pairing a few low latency cores with a large number of high throughput cores. Surround single Intel P core with a bunch of E cores. Then, hanging off the E cores, stick a bunch of iGPU cores and/or AVX-512 units.

Call it Xeon Chi.

Neat idea, probably even viable!

I think they may have a hurdle of getting folks to buy into the concept though.

I imagine it would be analogous to how Arria FPGA’s were included with certain Xeon CPU’s. Which further backs up your point that this could happen in the near future!

I think one possible reason for that, ideally these things need different memory.

If you use high-bandwidth high-latency GDDR memory, CPU cores will underperform due to high latency, like there: https://www.tomshardware.com/reviews/amd-4700s-desktop-kit-r...

If you use low-latency memory, GPU cores will underperform due to low bandwidth, see modern AMD APUs with many RDNA3 cores connected to DDR5 memory. On paper, Radeon 780M delivers up to 9 FP32 TFLOPS, the figure is close to desktop version of Radeon RX 6700 which is substantially faster in gaming.

Hmm, that is a good point. Since it is a dream-computer anyway, maybe we can do 2.5d packaging; put the ddr memory right on top so the P cores can reach it quickly, then surround the whole thing with GDDR.

You mean like an iGPU?

Edit: Oh, thanks for the downvote, with no discussion of the question. I'll just sit here quietly with my commercial OpenCL software that happily exploits these vector units attached to the normal CPU cores.

I’m not sure who downvoted; I think it isn’t possible to downvote a response to one’s comment.

I did decide not to engage because “you mean like <very common well known thing>” seemed a bit brusque and dismissive, commenting on this site is just for fun, so I don’t really see the point in continuing a conversation that seems like it is getting off on the wrong foot.

Reply on news.ycombinator.com