Hacker News Reader

Comment by brucethemoose2 | Hacker News Reader

This seems ridiculously excessive.

Stable Diffusion runs well on small GPUs... My 6GB laptop 2060 can run reasonably big resolutions. It can run Facebook AITemplate inference or train a LORA. A 12GB-24GB GPU can run pretty much anything with no issues or fuss.

A single 3090 (or 7900 XTX) can run LLaMA 65B reasonably quickly with llama.cpp. 2x 3090s can run it very quickly with exLLaMA.

MPT and Falcon quantization are extremely new (and TBH the finetunes are not as good as LLaMA yet), but MPT 30b will fit on a single 3090 (or a smaller GPU with llama.cpp) and Falcon (I think) can be split with GPTQ.

View on HN

Thanks.

Based on your comment, I added a note that the GPU recommendations are for GPU clouds (where if you're running stable diffusion, ~$0.50/hr for a 4090 is pretty practical, and it's not worth going to a cheaper card for most), and I added this note re local GPUs now:

  If you're using a local GPU:
  
  * Same as above, but you probably won't be able to train or fine tune an LLM!
  * Most of the open LLMs have versions available that can run on lower VRAM cards e.g. [Falcon on a 40GB card](https://huggingface.co/TheBloke/falcon-40b-instruct-GPTQ)
  * Thanks Bruce for prompting me to add this section

The UIs/containers people use for 4090s specifically have some kind of performance issue with Stable Diffusion, last I checked. It runs no faster than a 3090.

I think this can be solved by running RTX 4000 cards with CUDA 12.1 and pytorch nightly cu121... But thats a very uncommon setup.

Also, if we are talking cloud deployments, cheap CPU instances with 32GB-64GB of RAM are a somewhat sane way to test LLMs, thanks to llama.cpp. A GPU will give you better throughput, of course.

But CPU support for Falcon and MPT specifically are in flux, see this post for instance: https://huggingface.co/TheBloke/mpt-30B-instruct-GGML

For people reading using a 4090, this comment from March might help: https://www.reddit.com/r/StableDiffusion/comments/y71q5k/com...

Wait, can llama.cpp accelerate inference without full VRAM for the model? Or do you mean Llama 65B quantized to fit?

Yeah, it can split weights. Whatever fraction of the weights that don't fit into vram will be computed on the CPU (with reasonable speed).

Additionally, prompt processing will work with large models even with low vram GPUs.

You're able to train LORAs with 6gb VRAM? Do you have any guides or references by chance?

Reply on news.ycombinator.com