However, newcomers (like me) are pretty blind about minimum system requirements.
Could you please add them to the models list?
For example: what minimum hardware do I need to run Falcon-40b?
PS: If you only have a few setups "known to work" (or just one), listing that would be helpful too.
Every model is drastically different.
If you want to run something on consumer hardware, your best bet is using anything ported to the ggml framework, especially if you're on Apple silicon.
The number after q determines how many bits the weights are. Eg q4 means that is 4-bit.
If you use something like KoboldCPP you can only put some of the layers onto the GPU and be able to run larger models that way.
Eg the above linked Vicuna model requires about 10GB of memory at q4, but I have less VRAM than that. I can still run it though.
For example, assuming GPTQ 4bit a 100GB 16bit model needs 26-30GB of VRAM. The smallest video cards which meet this requirement will be 32GB or 40GB cards. (two 24GB cards in parallel work as well, e.g. 2x3090)
If you want to train a model, that’s a different story.
> You will need at least 85-100GB of memory to swiftly run inference with Falcon-40B.
So it may be possible with less (swap around method), though not as efficiently and also slower.
One can simply do
```openllm start falcon --model-id tiiuae/falcon-40b-instruct --quantize int4```
Beware that there is no free lunch, meaning the quality of inference will degrade by alot when using int 4 quantization