Hacker News Reader

Comment by sva_ | Hacker News Reader

Note how they carefully avoid naming an actual tokens/second number for any gpu.

Are there any resources that do provide this? I’ve tried on numerous occasions to search for stats on model XYZ puts out x tokens/sec on this hardware setup. It leads me to believe the only way to experiment with running open models is with cloud computing and expensive GPUs and consumer hardware is out. Even for 7B parameters.

7b can run fine on consumer hardware if you quantize to 4bit. I get around 50-100 token/s on 7b q4 models on older 8gb gpus I have lying around, which is plenty for chat. I don't know of any good resource for "consumer hardware leaderboard" though. In general:

1.3b is about the max you can get for 4gb vram

7b will work on 8gb vram if you quantize

13b will work on 16gb vram if you quantize

33b can work on 24gb if you quanitze

If you lack vram but have ram and infinite patience you can do offloading shenanigans with cpu but it tanks performance to unusable levels in my experience.

Considering quantizing takes a while, care to list some settings guide or a link for people to setup?

I use Phind_34B for a 24GB gpu. When in use I typically set gpu memory limit to 22.5GB.

Quantization: 1hr 3min. 4bit, group_size=32, desc=true, damp=0.01, trust_remote = false, dtype torch.float16, use_triton = false.

I'd defer to the llama.cpp or exllamav2 repos for quantizing help, I just use their guides myself. For practical use, I'd say searching for the quantized model on huggingface.co is the way to go. Chances are high that someone else already did what you're interested in and it's a lot faster to download a pre quantized model.

Thebloke, turboderp, lonestriker, bartowski. Between these 4 users on huggingface you can find pretty much every common model already quantized.

What do you use to run them? My 3060 is generating just 40token per second with llama.cpp

I use either llama.cpp or exllamav2 as the backend, but speed is mostly dependent on how many layers fit in your gpu. I'm not sure which gpu I was thinking of for that 50-100 tok/s, on my 2070 I can only get around 15 tok/s on 7b models. I have to use 1.3b models to get to 100 tok/s on this gpu.

Reply on news.ycombinator.com