LLaMA Text-Generation-Webui

github.com

11 points·0xbadc0de5··6 comments

text-generation-webui supports state of the art 4bit GPTQ quantization for LLaMA[0], reducing VRAM overhead by 75% with no output performance loss compared to baseline fp16.[1]

LLaMA-13B, rivaling GPT-3 175B, requires only 10GB* of VRAM with 4bit GPTQ quantization.

LLaMA-30B fits on a 24GB* consumer video card with no output performance loss, beating GPT-3 175B.

Multi-GPU support[2] means LLaMA-65B, rivaling PaLM-540B, runs on 2x3090.

*Further improvements in active development will reduce VRAM requirements another 30-40% with no performance loss (ex. flash attention).

[0] https://github.com/qwopqwop200/GPTQ-for-LLaMa

[1] https://arxiv.org/abs/2210.17323

[2] https://github.com/oobabooga/text-generation-webui/issues/14...

For LLaMA set-up instructions, including GPTQ 4bit, refer to this wiki article: https://github.com/oobabooga/text-generation-webui/wiki/LLaM...

So, is anyone doing open source work on finetuning this model? Can finetuning be done with the 4bit weights?

Close, fine tuning LLaMA in 8bit has been done. It requires very little VRAM compared to typical fine tuning.

https://github.com/zphang/minimal-llama#peft-fine-tuning-wit...

Is 2x4090 possible or just 3090?

Yes, text-generation-webui has multiGPU support regardless of your GPU(s). They do not even need to be the same GPU. You can have an AMD 6900xt combined with an RTX 4090, combined with an RTX 3050, if you want. It makes no difference.

Note that the notebook supports GPT-J 6B, OPT, GALACTICA, and Pygmalion, not just LLaMA.

Reply on news.ycombinator.com