Hacker News Reader

Llama.cpp can quantize without special training, but I'm not sure if any special model architecture support is needed to read it in the first place. If it can be converted to gguf at all and you know what tensors to target, it can get the full ternary bonsai treatment today.

View on HN

Sure, but that's for "personal" serving. I meant for 3rd party providers. Usually we get a good indication on what it costs to host this, as the prices settle on open router. That's why I said it's tougher to serve than kimi k3 on launch. As a provider you'd do fp8 if the model creator didn't do QAT on q4, or until someone does a good calibrated nvfp4. And that's usually nvda :)

That makes sense, but your specific phrasing precluded the possibility of non-QAT quantization.

Should have worded that better, my bad.

QAT is an optimizing quantization algorithm, not naive quant.

Right, but the way they phrased it suggested that without QAT it could not be quanted at all.

Isn’t QAT a training approach (roughly, simulating quantization in the forward pass during training so that quantization of the level targeted in training has close-to-optimal behavior), not a quantization algorithm? Hence, the name?

sounds like a repeatable method of optimizing quantization to me, friend

Reply on news.ycombinator.com