For falcon 40b you probably need an A100 40gb or so.
Every model is drastically different.
If you want to run something on consumer hardware, your best bet is using anything ported to the ggml framework, especially if you're on Apple silicon.
Every model is drastically different.
If you want to run something on consumer hardware, your best bet is using anything ported to the ggml framework, especially if you're on Apple silicon.
The number after q determines how many bits the weights are. Eg q4 means that is 4-bit.
If you use something like KoboldCPP you can only put some of the layers onto the GPU and be able to run larger models that way.
Eg the above linked Vicuna model requires about 10GB of memory at q4, but I have less VRAM than that. I can still run it though.
For example, assuming GPTQ 4bit a 100GB 16bit model needs 26-30GB of VRAM. The smallest video cards which meet this requirement will be 32GB or 40GB cards. (two 24GB cards in parallel work as well, e.g. 2x3090)
If you want to train a model, that’s a different story.