Hacker News Reader

My understanding is that as a general rule:

( Q / 8 ) * B = GB RAM, where Q is the Quant level and B is the model size.

So a Q4 7B model is ( 4 / 8 ) * 7 = 3.5GB RAM (or VRAM).

A non-Q model is 16, so 2 * B.

I believe this is before context, which adds a bit.

Definitely does not seem too far off:

Llama3 8B fp16 is 16GB, q8 is 8.5GB and q4 is between 4.3 and 4.9GB.

Llama3 70B fp16 is 141GB, q8 75GB and q4 is between 40 and 43GB.

based on your formulas, I am definitely going to need a larger card.

I run 8b on my Macbook air m2 with 16gb's of vram at decent speed of 12 t/s (mlx can probably get 15 t/s).

I use the Q5_K_M GGUF, which is >99% the same as the original.

I've seen tests that there is close to no divergence with these quantisations, but it rises steeply going lower:

I can run the 8B with my M2 Pro Mac mini with 32GB via Ollama. It takes around one second to get a response from the API.

I have enough VRAM on my 6900XT (16GB) for 13B quantized models alongside desktop/hyprland usage.

~8GB RAM for a quantized version of 8B