What’s the time horizon do you think for free models matching today’s SOTA on average consumer hardware? I see people building 6k+ machines to run the best of them at the moment, which are behind SOTA by maybe 6 - 12 months or so right now.
The problem there isn't the models, it's consumer hardware. Even 16GB cards aren't the norm, and even with massive improvements in per-parameter performance we probably still need 48GB memory to get models that feel smart enough to trust.
If you scoped it to “average gaming desktop”, double digit VRAM is pretty normal at this point. If costs came down, I imagine the higher end GPUs would start including enough VRAM for 30B-ish models.
SOTA in open source (frontier Kimi MOE) requires terabytes of RAM. At DDR5 prices, that's $40k alone. For HBM, higher. We're years away from consumer hardware matching the power and latency of e.g. Claude.