Do you guys know a website that clearly shows which OS LLM models run on / fit into a specific GPU(setup)?
The best heuristic i could find for the necessary VRAM is Number of Parameters × (Precision / 8) × 1.2 from here [0].
[0] https://medium.com/@lmpo/a-guide-to-estimating-vram-for-llms...
Your equation is roughly correct, but I tend to multiply by a factor of 2 not 1.2 to allow for highly concurrent traffic.
https://huggingface.co/settings/local-apps
Then on the model pages, it will show you whether you can use it.
Also most of the times they are split up and, sometimes, you’ll get an indicator on the splits.
It’s still a work in progress to check all hardware and model format compatibility but it’s a great start until GGUF becomes the standard.
So in the end, trying to actually run them seems to be the only fool-proof way of knowing for sure :)
While it is seemingly hard to calculate it, maybe one should just make a database website that tracks specific setups (model, exact variant / quantisation, runner, hardware) where users can report, which combination they got running (or not) along with metrics like tokens/s.
Visitors could then specify their runner and hardware and filter for a list of models that would run on that.