Hacker News Reader

Comment by flifenstein | Hacker News Reader

Author here(I'm on the team). We updated this post after Monday's Kimi K3 weights release: fitting the 2.8T model means going from 8×B200 to 8×B300, ~20% more hardware cost, and concurrency drops from 24 to 16 users vs GLM-5.2. Caveat we're upfront about in the post: our 64-task SWEBench Pro subset may be in Kimi's training set, so the 86% resolve rate is an upper bound.

Let us know your thoughts, we really value feedback!

View on HN

Like I said above, you should benchmark quantized versions. With quantization, the same models can be ran at much cheaper hardware, but quality loss is real and this kind of benchmark is an ideal place to put a finger on it.

I second this. Anyone considering running their own hardware would also consider quantization since they would have full control over what quant to run.

The value proposition changes a lot if you can get 90% of the quality for 50% the price with a quant due to halving your hardware requirement.

Thank you for the article, though! Very informative.

Kimi k3 is natively mxfp4 w/ mxfp8 activations. Since they talk about glm 5.2 running on a 8x b200s, that's likely fp8 since it wouldn't fit at bf16 (1.5TB weights alone)

HuggingFace used an NVFP4 quant of GLM-5.2 to investigate their latest hack, so that might also be worth a try:

https://huggingface.co/nvidia/GLM-5.2-NVFP4

Solid point. We debated multiple optimization methods and decided to do a more vanilla run first for the setups tested. We’re moving on to some more typical optimization methods for our next article… As soon as our teams comes out a well deserved summer break. Thx for the input.

So the 'unsloth' Kimi K3 Q8 quantization is about 1.56TB which is close to the original safetensors file in size, and should be nearly lossless. Realistically with enough context you'd need 2TB total RAM whether it's run on a CPU-only platform (like a dual socket xeon with 1024GB connected to each CPU) or some combination of CPU-connected-RAM and GPUs.

Unsloth Q4 is 1.51TB which doesn't really help much and isn't likely to be any different in performance.

Unsloth Q2-K-XL is 861GB and could possibly fit on 1TB resources but I would want to see a very thorough series of tests to see how much knowledge and capability is lost between it and the full thing.

Using an example from the much smaller gemma 4 31B because it's a decent set of charts I could find quickly, I don't know if anyone has published KL divergence charts for Q4 vs Q4 vs Q8 of Kimi K3:

https://localbench.substack.com/p/gemma-4-31b-gguf-kl-diverg...

> Aggregate token throughput is about 30% lower (122 vs 170 tok/s at 16 users)

Is that actually the aggregate throughput? 8xB300 (with 4TB/s/GPU bandwidth) is only pushing 8 tg/s/session? That seems… incredibly low, even for an A100B model.

Is it actually 122 tg/s per session? (1952 tg/s aggregate throughout)?

Reply on news.ycombinator.com