Hacker News Reader

Comment by Lwrless | Hacker News Reader

The "4x AI performance" smells suspicious to me, M5 Pro does have slightly higher memory bandwidth than the M4 Pro, but there are also the new Neural Accelerators in the GPU cores, I've always wondered what they actually do, since the SoC has AMX (Apple Matrix Coprocessor) since the original M1 chip era.

So is this performance gain probably due to the new Neural Accelerators?

View on HN

Yes, the "neural accelerators" in the M5 GPU cores really do 4x pre-fill performance over M4. You can find benchmarks online since M5 has been out for months now. I assume the M6 has whatever the next-gen version of those is. These are not the "neural processors" that have been there since M1 (those still exist) but are additionl matrix math accelerators inside the actual GPU cores. Like "tensor cores" on Nvidia GPUs.

Thank you, found some benchmarks and they look really promising. To my understanding those Neural Accelerators are like AMX but for GPUs. With those accelerators the GPU performance on a M5 Max in LLM inferencing would totally be on par with a 5090, that's quite impressive!

Which inference benchmark are you looking at? The prefill speeds should be comparable on some LLMs, but the 5090 has much a higher theoretical max decode speed.

I found some on Reddit, and here are the two benchmark results from review sites: - against a desktop 5090: https://nanoreview.net/en/gpu-compare/geforce-rtx-5090-vs-ap... - against a laptop 5090: https://nanoreview.net/en/gpu-compare/geforce-rtx-5090-mobil...

With larger (also really fast) unified memory on the chip, it could easily load larger weights, and the thing is that the quantized results on the M5s are really good, I believe that for most local inferencing, users would be using INT4 (maybe more bits per weight sometimes) quantized models, that might be where the Neural Accelerators kick in. In raw power, a desktop 5090 easily outperforms the M5 Max, but for this specific use case, I believe that the M5 Max is good enough.

Reply on news.ycombinator.com