Have you tried a mixture of experts model? Dense models have been quite slow for me, as I have only 6 GB VRAM. But with llama.cpp and --cpu-moe I get 200 t/s input and almost 30 t/s output with Gemma 4 26B A3B, which feels ok to use. Would be interested about your mileage there.