2028 Could Bring the Most Mind-Bendingly Expensive Apple Product of All Time

gizmodo.com

6 pointsmegamike3 comments

Are we going to get 1000 tps local?

I’ve been using local models on a Framework desktop. I get > 50 tokens/s on Qwen 3.6 35B A3B, and I find that speed ok for coding (though the model not quite). If machine could do a SOTA model with say 0.5-2T parameters like GLM 5.2 at 100 tok/s, it would be very useable.

In 2 years presumably the SOTA models will be much bigger, but I won’t be surprised if there’s a fable level sub 50B parameter model by that point either*, in which case 1000 tok/s may be reasonable.

* on artificial analysis intelligence index, GPT 4o is 12 and Qwen 35B A3B is 32.

It's just assumed that we need to make models bigger to get more out of them. That is from observation, no one has actually distilled different kinds of intelligences from an LLM to say this is the minimal model size needed for this kind of problem.

Examples: Databases scale with data stored in them. It's a linear relationship. We have databases that load pages on demand as needed and keep hot ones in cache.

There is no 'database page' alternative for LLM's yet. Someday there will be and the paradigm shift will happen.

Local 100 tok/s+ is a gamechanger for sure.

Reply on news.ycombinator.com