Hacker News Reader

Comment by sowbug | Hacker News Reader

Simon has mentioned in multiple articles how cool it is to use 64GB DRAM for GPU tasks on his MacBook. I agree it's cool, but I don't understand why it is remarkable. Is Apple doing something special with DRAM that other hardware manufacturers haven't figured out? Assuming data centers are hoovering up nearly all the world's RAM manufacturing capacity, how is Apple still managing to ship machines with DRAM that performs close enough for Simon's needs to VRAM? Is this just a temporary blip, and PC manufacturers in 2025 will be catching up and shipping mini PCs that have 64GB RAM ceilings with similar memory performance? What gives?

View on HN

LLMs run on the GPU, and the unified memory of Apple silicon means that the 64 GB can be used by the GPU.

Consumer GPUs top out at 24 GB VRAM.

llama.cpp can run LLMs on CPU. iGPU can also use system memory, the novel thing is not that, it's that the LLM inference is mostly memory bandwidth bound and memory bandwidth of a custom built PC with really fast DDR5 RAM is around 100GB/s, nVidia consumer GPUs at the top end are around 1TB/s, with mid range GPUs at around half that. M1 Max has 400GB/s, M1 Ultra is 800GB/s, but you can have Apple Silicon Macs with up to 192GB of 800GB/s memory usable by GPU, this means much faster inference than just CPU+system memory due to bandwidth and more affordable than building a multi-GPU system to match the memory amount.

It'd be really nice to have good memory bandwidth usage metrics collected from a wide range of devices while doing inference.

For example, how close does it get to the peak, and what's the median bandwidth during inference? And is that bandwidth, rather than some other clever optimization elsewhere, actually providing the Mac's performance?

Personally, I don't develop HPC stuff on a laptop - I am much more interested in what a modern PC with Intel or AMD and nvidia can do, when maxxed out. But it's certainly interesting to see that some of Apple's arch decisions have worked out well for local LLMs.

Apple uses HBM, basically RAM on the same die as the CPU. It has a lot more memory bandwidth than typically PC dram, but still less than many GPUs. (Although the highest end macs have bandwidth that is in the same ballpark as GPUs)

Apple does not use HBM, they use LPDDR. The way they use it is similar in principle to HBM (on-package, very wide bus) but it's not the same thing.

Right so Apple uses high-bandwidth memory, but not HBM.

Its not HBM, which GPUs tend to use, but it is on package and wider interface than other PCs

Apple designs its own chips, so the RAM and CPU are on the same die and can talk at very high speeds. This is not the case for PCs, where RAM is connected externally.

It's on the same package but the same die?

Reply on news.ycombinator.com