Hacker News Reader

Comment by LeoPanthera | Hacker News Reader

I bought a 128GB M4 Max Mac Studio a while back, and for a while I thought like I had done really well to buy it when I did.

The problem I'm having now is that no models are targeting RAM of that size. Everything is either much smaller, targeting laptops, or much larger, targeting hardware well out of reach of enthusiasts.

Please, AI people, start making models targeting 128GB machines again. The last interesting one was Qwen 3.5 122B.

View on HN

If I had 128GB unified RAM I'd try hf.co/unsloth/Qwen3.8-27B-GGUF:BF16 which needs 55GB for just the weights.

Something like this would give you three concurrent sessions, each with 240k token context:

  sudo sysctl iogpu.wired_limit_mb=110000
  
  llama-server -hf unsloth/Qwen3.8-27B-GGUF:BF16 -c 786432 --parallel 3 -ngl 99 -fa on

Great news. Qwen 3.8 Flash Next (125B A6B) is coming out tomorrow. 4 or 6-bit should run nicely on 128GB.

Should bench better than Opus 4.7.

I have had a difficult time with running 120b models on my 128gb setup, especially with any larger context size. The 6bit of Qwen 3.5 is already just over 100gb, and when you go down to 4bit it seems a bit lobotomized.

It's not exactly straight out of one of the labs -- it's heavily quantized -- but have a look at [1].

[1] https://github.com/antirez/ds4

Well the upside is that you can run a laptop-sized model and still have enough memory left over to run a couple of Electron apps.

Reply on news.ycombinator.com