> The great thing about gguf is that it will cross to system RAM if there isn't enough VRAM.

No… that’s not such a great thing. Helpful in a pinch, but if you’re not running at least 70% of your layers on the GPU, then you barely get any benefit from the GPU in my experience. The vast gulf in performance between the CPU and GPU means that the GPU is just spinning its wheels waiting on the CPU. Running half of a model on the GPU is not useful.

> Then again, one could have grabbed 2x 3090s for the price of a 4090 and ended up with 48gb of VRAM in exchange for a very tolerable performance hit.

I agree with this, if someone has a desktop that can fit two GPUs.