Hacker News Reader

> 4-bit models and smaller _can_ perform well but not through just naively quantizing the existing weights of a large model.

Doesn’t that describe the majority of quants on higgingface?

View on HN

Yeah, there's a tradeoff. pre-training with QAT needs more data and a ton of hardware, so it's easier to just take an existing open weight model and quantize it. This works, but it'll underperform a model that had quantization as a training target.

Matters more for 1-bit and 2-bit models.

Reply on news.ycombinator.com