You can run a 7B model on CPU relatively quickly. If you want to go faster, the best value in public clouds may be a rented Mac mini.
Do you have any resources to read on how to host LLMs in general? I am looking for scaleable ways to host our own models. Thanks.
Sorry I haven’t followed the latest developments to run at scale since the summer. I don’t have concurrent users so llama.cpp or diffusers are good enough for me.
Could it run on a 4x 3090 24GB rig?
These can be built for about $4500 or less all-in.
Inference FLOPs will be roughly equivalent to ~1.8X A100 perf.
You could run it on a single high end GPU. I can run llama2's models ,(except 70b) on my 4080.
This can run on 1x 2060S 8 GB
With what degree of quantization?
None, just the default weights using ollama. It's fast too. 13b is where things get slow
does a 4x 3090 rig need nvswitch
Presumably some compute per hour service would make more sense for playing around with it?