Hacker News Reader

waleedkon Numbers every LLM developer should know

[Author] You're welcome -- glad it was useful!

waleedkon Numbers every LLM developer should know

[Author] Completely disagree. Any analysis shows that you see perplexity reduction at 4 bits. Have a look at llama.cpp's results here:

https://github.com/ggerganov/llama.cpp#quantization

4 bit has a perplexity score 0.13 or so higher.

waleedkon Numbers every LLM developer should know

[Author] Fair point. Adjusted the language.

Nonetheless people do tend to use 16 bit huggingface models, and if you do go to 8 bits and it's wrong, you're never quite sure if it's the quant or the model.