Hacker News Reader

While this is generally true, it's _a little_ less true the larger the model is.

Also, quantization techniques have improved - the I in IQ3 stands for imatrix - Importance Matrix - it is a bit more surgical in what it cuts. The result is a model where the most important weights are even Q6 or above, the least important Q2 or even below, overall it takes the space of a Q3 but with better results.

A lot of words, but no benchmarks. I'm a little tired of all the "finger in the air" vibe checks. You can say all you want, but you'll only know once you put out some numbers. Which people have in other threads, and 4 bit 27B beats Flash at 3 bit

This is the conventional wisdom, but in practice what matters is how reliably the model performs on your tasks in the real world. I have an R9700 and an RTX 5060 Ti and I've been running an IQ3_S quant of 27B on the 5060 Ti vs a Q6 quant on the R9700. I still manage to get stuff done with the IQ3_S quant.

To add to the other comment, there's also Ridge quantisation - the majority of weights are indeed Q3_x, but the most sensitive layers are FP8.