Hacker News Reader

Considering quantizing takes a while, care to list some settings guide or a link for people to setup?

I use Phind_34B for a 24GB gpu. When in use I typically set gpu memory limit to 22.5GB.

Quantization: 1hr 3min. 4bit, group_size=32, desc=true, damp=0.01, trust_remote = false, dtype torch.float16, use_triton = false.

View on HN

I'd defer to the llama.cpp or exllamav2 repos for quantizing help, I just use their guides myself. For practical use, I'd say searching for the quantized model on huggingface.co is the way to go. Chances are high that someone else already did what you're interested in and it's a lot faster to download a pre quantized model.

Thebloke, turboderp, lonestriker, bartowski. Between these 4 users on huggingface you can find pretty much every common model already quantized.

Reply on news.ycombinator.com