Currently trying to work to have a plugin of similar to post back to get accurate benchmarks.
Take a look if you're interested!
You need to get more desktop GPUs on here, for the inference at home people.
Part of the problem we have is some of those desktop GPUs don't have power and flops data. We're toying with the idea of writing a CLI to run/publish these but obviously a lot of work.
Cheers for engaging!
https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_proces...
If you know what any given model uses for BW per token and flops per token, you can roughly infer for many models until you get better benchmarks.
There is also:
CUDA: https://github.com/ggml-org/llama.cpp/discussions/15013
HIP: https://github.com/ggml-org/llama.cpp/discussions/15021
Vulkan: https://github.com/ggml-org/llama.cpp/discussions/10879
Metal: https://github.com/ggml-org/llama.cpp/discussions/4167
These three are measured against the old model of Llama 2 7B, but it also means the numbers are without any per-model quirks that reduce compute and memory BW usage, thus more or less generically applicable.
Will take a look