Hacker News Reader

Comment by andy99 | Hacker News Reader

I only see a 1-bit quant posted on unsloth HF and it’s 72.5 GB. Is that what you mean? That’s much bigger than I expected. If you can’t run a 4 bit quant in on Strix Halo it becomes a lot less interesting. https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF

View on HN

In their page they say it will need at least 112GB[0], so including context, that would be a tight fit. I'm also hoping I can make a q4 fit on my 128GB strix halo

[0]: https://unsloth.ai/docs/models/qwen3.8-next#qwen3.8-flash-ne...

in llama-server PR 27742 it fits fine in 128GB RAM on a CPU only system , this is with --load-mode mlock to stuff the whole thing persistently into memory at llama-server launch time, no mmap

0.01.033.250 I common_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |

0.01.033.261 I common_memory_breakdown_print: | - Host | 118186 = 106166 + 8898 + 3122 |

0.01.092.684 I common_params_fit_impl: projected to use 118186 MiB of host memory vs. 128855 MiB of total host memory

Just a hunch, but it might be because of the 51B parameter n-gram embedding. At 125B, you'd expect ~16gigs for a 1-bit quant. Add 51gigs for the n-grams and you're not far off the actual size.

If that's true, it'd scale linearly with number of bits in the quant with an offset of about 51gigs. So Q4 should be a bit bigger than 82gigs, I'd guess in the 90s (as opposed to a ~280gig q4 if the whole 70gigs of the 1-bit quant scaled linearly).

Reply on news.ycombinator.com