Don't forget about Macs. Top models can run even fairly large LLMs (although time to first token is... not great). But even midline can certainly run LLMs in the ballpark of ~30B params very well, which is where things start to get interesting IMO.
The token cache works on CUDA too. So yeah, the initial loadout sucks, but almost everything from then on is solid.