It's pretty clear to me that above a certain size threshold, LLMs are more than a sum of their parts. The sheer amount of training data seems to embed a higher level of reasoning.
https://yosefk.com/blog/llms-arent-world-models.html
https://www.reddit.com/r/programming/comments/1mnc9qf/llms_a...
> The core issue there isn't that the LLM isn't building internal models to represent its world, it's that its world is limited to tokens. Anything not represented in tokens, or token relationships, can't be modeled by the LLM, by definition.
I've noticed that LLMs struggle with tic tac toe, even when I get them to draw it out. And it makes sense! Tic tac toe is an extremely visual game, and difficult to reason about in a single dimension.
You're staring at a mirror.
I have never seen any compelling evidence that LLMs actually reason. They are good at predicting the next token, and that is all they do. This action may be useful, but it is not reasoning.
That's confusing the training objective with the learned behavior. It's like saying "Stockfish's algorithm is literally 'minimize this number', and therefore, it can't actually play Chess."
“There cannot be any reasoning embedded in the model” a strong statement, what do you mean by reasoning because by any reasonable definition I’m aware of, they clearly are able to exhibit reasoning.
The fact that the pre training objective is next token loss has nothing to do with capabilities or their ability to reason. To be highly successful at next token prediction you NEED to reason. I’m quite confused here.
Since you refuse to actually define what you consider to be reasoning let me at least put one out there: a system exhibits reasoning when an answer depends on nontrivial intermediate computation over the problem. If you find problems with this, fine, but just make an effort to contribute an alternative.
If you increase test time compute you get better performance. If the model was just "interpolating" this wouldn't really work would it? Models can do FrontierMath expert problems (unpublished, expert authored, peer reviewed math problems) that require an insane amount of compositional reasoning. If they were regurgitating training data, that wouldn't really work would it? Chain of thought, while not always faithful to internal computation, improves performance. If the models were just regurgitating information, it wouldn't work that well would it?
"regurgitating training data" is also of course misleading. Yea they can memorize parts of the training data, but they generalize very well.
Im not sure exactly what you’re saying here, is it that you trade off coding performance and performance on other tasks like communication? If so (correct if not) this isn’t true — generalization happens. Doing good on coding lifts all boats.
> Somehow the hype over the successes with coding in the last year or so made everyone forget the intrinsic limit posed by the exhaustion of real human text output, which is absolutely inescapable
You’re absolutely right that we’re quickly running out of human text data but that isn’t at all the limitation you think it is. No one has “forgotten” this — coding agent performance is primarily from reinforcement learning on synthetic data traces with verifiable rewards, though pretraining is still important.
Also don’t forget: there is a world of multimodal data (video, audio, 3D maps, etc) that is incredibly rich.
They consistently fail at drawing basic logical conclusions because they cannot build a sufficiently abstract model of certain problems that allows them to grasp their true nature. In other words, the whole class of questions of the kind of "how many r's in strawberry" or "do I take the car to the car wash?" would be answered correctly and reliably.
That models cannot do ALL logic problems does not mean that they cannot properly use logic...they can write Lean-verified theorems. How is that not logic?
> They consistently fail at drawing basic logical conclusions because they cannot build a sufficiently abstract model of certain problems that allows them to grasp their true nature.
What does their "grasp[ing] their true nature" have anything to do with what they can do?
> In other words, the whole class of questions of the kind of "how many r's in strawberry" or "do I take the car to the car wash?" would be answered correctly and reliably.
Again, just because you have interesting failure modes or brittleness does not mean they do not reason.
Add.: I pointed to this class of problems specifically because they require the ability to abstract in a way that the question itself does not immediately suggest. Math problems are different in that they are described in terms of art that are closely related to certain patterns of manipulation (that is, the paper texts tend to contain both in close proximity to one another).
> they require the ability to abstract in a way that the question itself does not immediately suggest
yes, yet there are multitudes of other measurements of the same kind where LLMs reason perfectly well and better in many cases than a human could.
> Math problems are different in that they are described in terms of art that are closely related to certain patterns of manipulation (that is, the paper texts tend to contain both in close proximity to one another).
Is your logic really that math problems are actually easier to answer without reasoning and just by blending together closely related papers? I would definitely suggest reading the literature a bit more on this topic.