Hacker News Reader

Comment by Kim_Bruning | Hacker News Reader

Up to 128 GB RAM available for running AI models.

Theoretically awesome, but this might have some interesting market consequences for everyone else.

$2,0000 to $3,000 desktop devices with 128GB of shared GPU/CPU RAM seems to be a segment that is seeing lots of announcements from lots of vendors.

With what memory bandwidth? Remember, without e.g. speculative decoding you need to read the entire model and KV cache for every token. Let's be extremely generous and say you get 512GB/sec in memory bandwidth, on par with a high end M4 MacBook Pro. This means you can only read the entire DRAM 4 times a second, generating at most 4 tokens per second. Smaller models will of course run proportionally faster, but 128GB isn't by itself a sufficient statistic to say whether this is "theoretically awesome" or not.

> With what memory bandwidth?

256GB/sec, so roughly M4 Pro throughput.

What's frustrating is there's no real reason why regular DDR5 can't reach 1TB/sec with a sufficient number of channels. The manufacturers are just holding it back to drip feed that memory bandwidth over several generations. Except for Apple, which lets you have 800GB/sec now, and will let you have 1TB/sec+ in M5 Ultra next year. It's $$$$, but still - the true alternatives are much less cost effective.

Right, like the mac studio ultra.

Or a dual socket AMD Turin.

Or a grace+hopper (assuming you offload to the hopper).

Latency is dictated by the laws of physics, more bandwidth is easy, but not cheap.

It can be relatively cheap too under the constraints imposed by typical AI workloads, at least when it comes to getting to a 1TB/s or so. All you need is high-spec DDR5 and _a ton_ of memory channels in your SOC. During transformer inference you will easily be able to use those parallel, multichannel reads. I get why you'd need HBM and several TB/s of memory bandwidth for extremely memory intensive training workloads. But for inference 1TB/s gives you a lot to work with (especially if your model is a MoE), and it doesn't have to be ultra expensive.

The bandwidth still doesn't quite compare to a GPU, and 128GB doesn't fit DeepSeek R1, however. If they bump it to 512GB for $5000 or so, that will disrupt the market.

256GB/s is on par with the M4 Pro at a much lower price.

You could run R1 671B using unsloth’s quantized version that fits in <80GB. Not sure why that would be a benchmark though, there’s nothing that can run the model at full precision right now except for (very slow) server hardware.

But you can use an AMD EPYC cpu and get 460GB/s bandwidth. Could probably get 512GB RAM for a lot cheaper than 5K.

You’ve gotta deal with amd’s AI software stack though. How is that these days? I assume cuda is still king?

I've been running ollama on an XTX7900 (AMD GPU with 24GB of RAM) with any model that fits in it, and absolutely no issues there.

> You’ve gotta deal with amd’s AI software stack though

Not if you are using the CPU. I am under the impression most inference use cases are memory bandwidth limited, not compute limited, so running on the GPU would gain you little to nothing unless the GPU has faster access to the shared memory.

Inference fine, training less so

Reply on news.ycombinator.com