I have been looking for a good local setup for a while now. Qwen 3.8 27b is really good for a dense model of this size IMO. I already had an RTX 4090 and I forked ninfer [0] with the obsession to squeeze everything out of this card for this model. Results: 149 tok/s decode speed (aggregate with concurrency about ~270 tok/s) with prefill speeds faster than 2500 tok/s. And all of this with full 262k albeit quantized context. Fast prefill speed is really important when launching multiple clients such as opencode or pi at the same time and especially if they launch subagents. This is why I also implemented a caching tier so computed contexts can be faster loaded from RAM (or disk). Speeds feel almost like with official SOTA openai or anthropic models.
Im currently measuring a pareto front in J/tok in order to set power limits of this card without sacrificing too much performance. Since we are talking about full power draw of ~480W which is fine during the day (with solar panels) but during night when the sun doesn't shine (even with a battery) I'd like to limit this a little bit.