In other comments people mention the "flywheel" of data and money feeding training, but there's a view that at some point the baseline open-weight models are "good enough" that the money will dry up.
baseline open-weight models are "good enough" that the money will dry up.
My guess is that the industry will consolidate. The winners will absorb the losers and focus on generating revenue.
Therefore, there will be a growing gap between open and free models and the proprietary SOTA models.
If there is consolidation by absorption, that derisks attempting to challenge the SOTA providers, and so they will keep facing attempts.
The ones that are already released are, and they're already very good for most purposes and can be fine-tuned indefinitely, includin months or years down the line when processes have been optimized and things aren't as compute-heavy as they are now.
And the fact that the open-weight as well as cheaper tier 2 offers exist both place a ceiling on the prices the SOTA companies can demand - and as far as we know current prices don't even fully pay for inference alone already, at least not for OpenAI.
It's not clear. Z.ai is definitely not profitable.