This model might be the first step in that direction, as competition heats up between OpenAI and Anthropic.
Have they, actually? A lots of speculation and claims but what is the level of admittance?
It's also used to describe the SFT bootstrapping for posttraining, which is what people generally refer to as Chinese labs "distilling".
I would almost guarantee that smaller US frontier models (ex Luna/Sonnet) are distilled from their respective large models.
My main worry is that we get to a point where they have something much, much smarter than anything public and access is gated by extraordinarily high costs.
That probably can't happen though right? Inference is surprisingly cheap compared to the training.
- Training data, harnesses, and processes, that drives the quality of the models. Several companies have those. Some of those companies are in China.
- Infrastructure and funding for running those. That's a scarce commodity currently, training the latest frontier models cost billions of dollars apparently. And if you don't have the infrastructure already and don't have the suppliers on speed dial, good luck getting anything.
- Infrastructure for running inference for running what comes out of those. Several of the key providers of this infrastructure are using their own in house chips for this now. At scale this means huge cost savings.
If you start from scratch without infrastructure, there are a bunch of open source things you can find. But beyond that, you'll have a lot of catching up to do. That's the very definition of a moat.
Infrastructure is readily available. Anthropic and OpenAI don’t own anything, they’re just paying for compute. You might not be able to buy 100k GPUs right now but you can rent it.
You only need to look at how Jev immediately became one of the highest volume models after launch on OpenRouter to see how this industry is moatless.
OpenAI and Anthropic employees frequently leave to start their own labs and raise hundreds of millions for it which they can use to immediately pay for infrastructure and training data. At most OpenAI and Anthropic have… brand and talent.
If anything, OpenAI and Anthropic are heavily disadvantaged because they have huge long term financial commitments that have backed them into a corner, likewise the regulatory pressure… startups have none of that.
And I'm going to keep telling people that it is not unreasonable for individuals to spend $30-40K on an AI rig if they are as productive and useful as many people imagine, my friends and family can take advantage of my rig when I'm not using it, and I'm independent of some snooping corporation - totally private.
There is little reason for these things to be concentrated in far away data centers other than that pooled usage is more elastic (but if I'm sharing my rig with many other people, AI is agentic now, and I can lay off some of my compute to others and they can lay it off on me I'm pretty elastic on my own.) The real reason is because we moved to a rentier economy before AI with the cloud and locked-down phones, and it is a very profitable model that doesn't care if it's rational.
You could subscribe to training material like you would subscribe to a newspaper. The most important training material would be your system learning off you, so training will have to be done in place (we can't rely forever on injecting specifics into contexts.)
These things should be in people's houses, and in the winter you should be using them as radiators. Don't think the power grid can handle it? Put solar on the roof and have batteries at home; helps both you and the grid.
This depends on what governments do. Crony capitalism in the US can lead to bans on home AI even if they have to root all general purpose computers to do it. The same is inevitable in Europe due to both total elite disdain of democracy and their slavish service to US interests.
Even if Western governments just lock down and antitrust-exempt the frontier labs without rooting personal computers in order to keep out Chinese/open models, I'm in far from a comfortable position when I'm depending on the Chinese government to protect my freedom.