Hacker News Reader

HNHacker News

Top New Best Ask Show Jobs

Comment by MacsHeadroom | Hacker News Reader

Parent Full thread

MacsHeadroom·

You link to Guanaco-33B, but Guanaco-65B is much more capable.

CPU Version: https://huggingface.co/TheBloke/guanaco-65B-GGML

GPU Version: https://huggingface.co/TheBloke/guanaco-65B-HF

4bit GPU Version: https://huggingface.co/TheBloke/guanaco-65B-GPTQ

View on HN

It irritates me to no end that people don't list the system requirements of various models.

How much ram and vram does one need to run 4,13,33,65B models at a reasonable speed?

edit: instead I'll ask this, what's the best model to run on a system with a 24gb 4090 and 64gb of ram?

You can just barely fit a 33B GPTQ model in 24GB VRAM. It will be in 4-bit mode, and without maximum context size, but it will be quite fast. Or you can run from RAM+VRAM in GGML format with llama.cpp (or a derivative), which will easily fit 65B models even at 5 or 8 bits, but at much lower speed.

It's fairly simple to estimate ram requirements based on parameter count: https://blog.eleuther.ai/transformer-math/

A 4-bit quantized 33B parameter model will fit on your GPU and you'll be able to use a 2048 token context too. (4-bit quantized larger models are better than smaller 8bit/16bit models)

You can run 4-bit quantized 65B models on your cpu, but it is slow, 1-2 tokens a second instead of 8-15 people typically get with a gpu, but you need two 24gb or an enterprise card with 48gb of ram to load them there.

https://old.reddit.com/r/LocalLLaMA/wiki/models has the information you are irritated about not being listed.

The first link there gives very specific RAM requirements for each of the models.

"Reasonable speed" is subjective, though.

Reply on news.ycombinator.com