Hacker News Reader

Comment by revolvingthrow | Hacker News Reader

> 320B total parameters and just 18B active parameters

This is pretty hefty for a "flash" model, even a 256 GB setup is insufficient at q4 - and q4 is already the worst-but-still-acceptable quant in my experience. The benchmarks look great, especially since GLM tends to be more honest than the average Chinese lab, but you’ll need to splurge to run it at home.

@edit: so many releases that I forgot to math. This fits just fine in q4, realistically the minimal hardware would be 192gb - so blazing fast on double rtx 6000 pro and usable on 256gb unified memory. You could even go with 5bit quant on 256gb.

… you’ll still need to splurge, though.

View on HN

That's 160GB-ish for Q4...how is 256 insufficient?

Looks like the M5 Ultra Studio wait times are going to increase again. Already at 10-12 weeks, I wonder how long it'll go?

I guess like the M3 Ultra, at some point normal customers won’t be able to buy it.

That M3 had an older type of RAM. Apple hopefully secured sufficient supply of the newer variant for the M5 Ultra.

Both use LPDDR5x, they're not shipping LPDDR6 (yet).

All M3 variants use LPDDR5.

Speaking as someone who isn't really well versed in this, does 18B active parameters mean that you could potentially hold only the 18B parameters in RAM and stream the rest from a fast NVMe SSD for acceptable performance similar to how Colibri works?

https://github.com/JustVugg/colibri

Sorta, but you're off by one layer. You can store the 18B in VRAM and stream the rest from RAM. There's still a performance hit relative to storing it all in VRAM, but it's tolerable.

Generally, for local consumer use, these large MOE models are best for unified RAM systems like DGX Spark or Mac Studio.

Normal MoE is switch-weights-per-token so you would se substantial slowdowns that way. Apple did a More that switches weights per prompt (instruction-following pruning, https://arxiv.org/abs/2501.02086) but you have to design the model that way which I don't think the have.

Reply on news.ycombinator.com