"Reasoning" will be disproven for this again within a few days I guess.
Context: o1 does not reason, it pattern matches. If you rename variables, suddenly it fails to solve the request.
Context: o1 does not reason, it pattern matches. If you rename variables, suddenly it fails to solve the request.
These models can and do work okay with variable names that have never occurred in the training data. Though sure, choice of variable names can have an impact on the performance of the model.
That's also true for humans, go fill a codebase with misleading variable names and watch human programmers flail. Of course, the LLM's failure modes are sometimes pretty inhuman, -- it's not a human after all.
One of the interesting DeepSeek-R results is using a 1st generation (RL-trained) reasoning model to generate synthetic data (reasoning traces) to train a subsequent one, or even "distill" into a smaller model (by fine tuning the smaller model on this reasoning data).
Maybe "Data is all you need" (well, up to a point) ?