Benchmarks I've run with a Ryzen 7950x, 128 GB RAM with Nvidia GeForce 3060 12 GB VRAM show a slowdown less than half when not using the GPU, with LLama.cpp as the inference platform and various ggml open source models in the 7B-13B parameter range.

The Ryzen does best with 16 threads, not the 32 it is capable of, which is expected due to it having 16 CPU cores.