What about the cost? Inference looks like a viable business model for those operators that have access to SOTA model and serving infrastructure, but the investment required to have it is enormous, and appears to be never-ending, because if an operator stops investing aggressively, its model and serving infrastructure quickly become non-competitive, and customers will quickly leave for alternatives.
The 20 euros GPT plan with GPT 6 Luna (and Astra for doing the tough stuff) and OpenCode Go (8 euros) offer me both way more intelligence and tokens than spending 400 euros just at the beginning of the this summer did.
Today I worked two hours with GPT 6 Luna, have been very productive tackling some hard stuff and...my weekly usage went from 57% to 56%?
You really don't need to use the strongest model for everything. And even chinese open models are effectively few months old sota. You can use mimo 2.6 or DS 4.1 flash on 8 $/months OpenCode and barely ever have to worry about consumption.
What I don't know, and was asking about, is whether the cost of providing that service is also falling rapidly, including the cost of training.
At some point this will probably happen, but we’re not quite there yet.