Hacker News Reader

FYI the model is almost 150GB: https://huggingface.co/Qwen/Qwen2-Math-72B-Instruct/tree/mai...

Nobody runs these things at full precision, a 4 bit quant of 70B range models fits into 64 GB. Though you would need more for long context.

A simple equation to approximate it is `memory_in_gb = parameters_in_billions * (bits/8)`

So at 32 bit full precision, 70 * (32 / 8) ~= 280GB

fp16, 70 * (16 / 8) ~= 140GB

8 bit, 70 * (8 / 8) ~= 70GB

4 bit, 70 * (4 / 8) ~= 35GB

However in things like llama.cpp quants sometimes it's mixed so some of the weights are Q5, some Q4, etc, so you usually want to take the higher number.

Well that and you also need a fair bit more space for the KV cache which can be a bit unpredictable. Models without GQA, flash attention or 4 bit cache support are really terrible in that regard, plus it depends on context length. Haven't found a good rule of thumb for that yet.

How much ELO on chatbot arena does 4 bit lose vs full fp16/bf16?

I think this is the only known benchmark that actually compares different quants: https://oobabooga.github.io/benchmark.html

It's not really all that consistent, but larger models can be compressed more without as much loss.

Reply on news.ycombinator.com