No, with limited VRAM you could offload the model partially or split across CPU and GPU. And since CPU has swap, you could run the absolute largest model. It’s just really really slow.
The M4 pro was able to answer the test prompt twice--once on battery and once on mains power--before the AMD box was able to finish processing.
The M4's prompt parsing took significantly longer, but token generation was significantly faster.
Having the memory to the cores that matter makes a big difference.
> VRAM is what takes a model from "can not run at all" to "can run" (even if slowly), hence the emphasis.
Is false. Regardless of how much VRAM you have, if the criteria is "can run even if slowly", all machines can run all models because you have swap. It's unusably slow but that's not what OP was claiming the difference is.
My memory is wrong, it was the 32b. I'm running the 70b against a similar prompt and the 5950X is probably going to take over an hour for what the M4 managed in about 7 minutes.
edit: an hour later and the 5950 isn't even done thinking yet. Token generation is generously around 1 token/s.
edit edit: final statistics. M4 Pro managing 4 tokens/s prompt eval, 4.8 tokens/s token generation. 5950X managing 150 tokens/s prompt eval, and 1 token/s generation.
Perceptually I can live with the M4's performance. It's a set prompt, do something else, come back sort of thing. The 5950/RTX3080's is too slow to be even remotely usable with the 70b parameter model.
Otherwise, you don't even need a computer. Pen and paper is plenty.
For all practical purposes, VRAM is a limiting factor.