You can have that! Qwen 3.8 Flash-Next is ~Opus 4.6 and runs nicely on a DGX Spark. And that’s just an architecture preview. The Qwen 4 family is expected to arrive this fall.
Do you know what kinda throughput you’re getting on that kinda setup?
(I have a secondary problem of being “locked into” Claude Code by it being good enough for me, I’d probably need to investigate the other harnesses… my impression is other harnesses are a bit more aggressively OK with nuking your setup from orbit)
The throughput in a single stream is about 50 tokens/sec (a bit less for prose, a bit more for code due to speculative draft acceptance rates) and about 2,000 tokens/sec for prefill. Both numbers are flat and stable as context accumulates. That’s what finally tilted me away from the Mac Studio despite its much superior memory bandwidth.
I think these numbers may improve because the model is pretty new and optimizations aren’t done.
The only reason to run locally is privacy.
Things get cheaper at scale but that's where the provider's margins come in!
I do think there's also an interesting idea: you buy a box like this and run it at a fixed-ish cost (well, electricity). Your demand goes up but your supply is fixed... and that back pressure means that you still have good cost control.
With cloud providers it's a _biiiiit_ too easy to just increase spend.
Sometimes it's OK for things to just be slow.
Renting tokens from open model providers is cheaper but it incurs the same issues: unexpected changes in model quality, inconsistent speeds, service outages.