They want a 10x improvement from scaling and a 10x improvement from data and algorithmic changes
The sources of public data are essentially tapped
Algorithmic changes will be an unknown to us until they release, but from published research this remains a steady source of improvement
Scaling seems to stall if data is limited
So with all of that taken together, the logical step is to figure out how to turn compute into better data to train on. Enter strawberry / o1, and now o3
They can throw money, time, and compute at thinking about and then generating better training data. If the belief is that N billion new tokens of high quality training data will unlock the leap in capabilities they’re looking for, then it makes sense to delay the training until that dataset is ready
With o3 now public knowledge, imagine how long it’s been churning out new thinking at expert level across every field. OpenAI’s next moat may be the best synthetic training set ever.
At this point I would guess we get 4.5 with a subset of this - some scale improvement, the algorithmic pickups since 4 was trained, and a cleaned and improved core data set but without risking leakage of the superior dataset
When 5 launches, we get to see what a fully scaled version looks like with training data that outstrips average humans in almost every problem space
Then the next o-model gets to start with that as a base and reason? Its likely to be remarkable