To be fair, o1 is a major breakthrough in the field. If other AI labs can't crack scaling useful inference compute, OpenAI will maintain a big lead.
I think it's as they say: reinforcement learning applied to cause it to generate a relatively long 'reasoning trace' of some kind from which the answer is obtained through summarisation.
I think it's likely a cleverly simplified version of QuietSTaR, with no thought tokens, just one big generation to which the RL is applied.
The way I believe it's trained in practice is as follows: they have a bunch of examples, some at the edge of GPT-4s ability to answer, some beyond it, some that GPT-4 can answer if you're lucky with the randomness. Then they give it one of these prompts, generate a fairly long text, maybe 3x the length of the answer, and summarize that to produce the final answer. Then they use REINFORCE to reward the generated texts that increase the probability of the summary being correct.
Is it? I feel like if you don't care about the cost it's pretty easily replicable on any other LLM, just with a lang-chain sort of approach
this may have consequences for how useful it is.