Hacker News Reader

Comment by ricardobeat | Hacker News Reader

Open-weight models are going to completely shatter these forecasts. It takes a little more effort – right now, probably won’t be true in three months – but you can achieve the same at 1/10th of the cost.

View on HN

> but you can achieve the same at 1/10th of the cost.

For some tasks, sure. But not for all tasks. And for some tasks, cost per token is irrelevant if it provides real benefits that are oom compared to what you had.

Local models are indeed becoming "good enough" for some tasks, but there are still tasks that they can't touch. There's a recent benchmark for kernel writing. Fable wrote a kernel that provides ~30% more throughput per unit of compute compared to the latest Opus max / gpt max. Does it matter how much that session cost in terms of one session if you can take that kernel, deploy it on your inference fleet and "magically" get 30% more tokens served to your clients? There are companies that would pay millions for such a "leap". Because they can make more millions down the line.

That’s true, there will always be demand for ultra-intelligent assistants, especially if they surpass what humans can achieve at similar cost. For the other 90%, the average frontier model will be good enough.

I don't disagree with you, but it's important to pay attention to where the money is. Cheap non frontier models is something that Anthropic and open AI could do too, but who's willing to pay a premium for using them? It will be like competing to sell rice, lots of demand at Rock bottom margins.

You're saying it like it would be a problem.

The question is: what proportion of tasks can not be handled by GLM5.2?

How many software developers were working on code like the one you describe?

On that aspect, I agree. Smaller / open models are becoming "good enough" at an increasing number of tasks. And that's great for us, consumers. But there will always be tasks that are "worth" pursuing with better models, and cost is irrelevant for those tasks. That was the point I was trying to make.

You're looking at the status quo and ignoring the trajectory. The best current open models are about as good as closed models from ~1.5 generations ago. The rate of improvement of all models is converging to zero. It follows that in a few generations, open models inferencing will be about as good as closed model inferencing.

The problem is going to become that there's no incentive for anyone to run the stupidly-expensive training phase. May God have mercy on the stock market.

> The rate of improvement of all models is converging to zero.

That's so obviously not true that I don't even think it's worth the energy to even debate it. It's been said for years, yet here we are, constantly improving. People really don't get RL / the bitter lesson, do they?

> It follows that in a few generations, open models inferencing will be about as good as closed model inferencing.

Not a chance. There's hundreds of billions of dollars on one side, and oom less on the other. There's also scaling laws and information theory. No matter how good, a 30B model will not be able to be better than a 3T+ model, all things being equal.

You are mistaking models becoming "good enough" for an increasingly number of tasks, which I agree is happening, with SotA models stagnating, hitting walls etc. That will not happen for many many years to come.

>The rate of improvement of all models is converging to zero.

Curious where you draw this conclusion from? Most benchmarks still show continual steady progress https://metr.org/time-horizons/

To me it seems like a first-year physics scaling laws problem. To get linear improvements in capability, you appear to need need exponential (or at least superlinear) increases in model size. We have no technical nor business solution for that kind of scaling, so the long-term outcome is obvious.

Everything depends heavily on the segment. Open-weight models already look good enough for a ton of internal tasks right now but there is always gonna be that small percentage of workloads where the last few percent of quality are totally worth the money.

I feel like the market is just gonna become way more mixed. Not like "everyone is switching to open" or "everyone is staying on frontier" but a mix of multiple models for different scenarios

And 10x the headache. Money can be exchanged for goods and services, and people pay money to not have to deal with things. If you don't have the money for it, you pay for it in dealing-with-bullshit credits.

Reply on news.ycombinator.com