"Just" guessing the next token requires understanding. The fact that LLMs are able to respond so intelligently to such a wide range of novel prompts means that they have a very effective internal representation of the outside world. That's what we colloquially call "understanding."
This becomes pretty clear when you get to more complex algorithms or low level details like drawing a stack frame. There is not logic there.
I can ask ChatGPT questions that require logic to answer, and it will do just fine in most cases. It has certain limitations, but to say it isn't able to apply logic is just completely contrary to my experience with ChatGPT.
ChatGPT answers:
> Yes, if we assume the statement "all snakes have legs" to be true and accept that a python is a type of snake, then logically, a python would have legs. This conclusion follows from the structure of a logical syllogism:
> 1. All snakes have legs.
> 2. A python is a snake.
> 3. Therefore, a python has legs.
> However, it’s important to note that in reality, snakes, including pythons, do not have legs. This logical exercise is based on the hypothetical premise that all snakes have legs.
ChatGPT clearly understands the logic of the question, answers correctly, and then tells me that the premise of my question is incorrect.
You can say, "But it doesn't really understand logic. It's just predicting the most likely token." Well, it responds exactly how someone who understands logic would respond. If you assert that that's not the same as applying logic, then I think you're essentially making a religious statement.
An animation looks exactly like something in motion looks, but it isn't actually moving.
>A Z is an X.
>Therefore a Z has Ys.
I am fairly certain variations of this are in the training set. The tokens following that about "in reality Zs not having Ys" are due to X, Y, and Z being incongruous in the rest of the data.
It is not not performing a logical calculation, it is predicting the next token.
Explanations of simple logical chains are also in the training data.
Think of it instead as really good (and flexible) language templates. It can fill in the template for different things.
Those two things are not in any way mutually exclusive. Understanding the logic is an effective way to accurately predict the next token.
> I am fairly certain variations of this are in the training set.
Yes, which is probably how ChatGPT learned that logical principle. It has now learned to correctly apply that logical principle to novel situations. I suspect that this is very similar to how human beings learn logic as well.
If it were, these LLMs wouldn't hallucinate so much.
Semantic understanding is still a ways off, and requires much more intelligence than we can give machines at this moment. Right now the machines are really good at frequency analysis, and in our fervor we mistake that for intelligence.
In order to do that effectively, you have to have very significant understanding of the world. The texts that LLMs are learning from describe a wide range of human knowledge, and if you want to accurately predict what words will appear where, you have to build an internal representation of that knowledge.
ChatGPT knows who Henry VIII was, who his wives were, the reasons he divorced/offed them, what a divorce is, what a king is, that England has kings, etc.
> If it were, these LLMs wouldn't hallucinate so much.
I don't see how this follows. First, humans hallucinate. Second, why does hallucination prove that LLMs don't understand anything? To me, it just means that they are trained to answer, and if they don't know the answer, they BS it.
This results in the appearance of an arms race between world model refinement and user cleverness, but it's really a fundamental expressive limitation: the user can always recurse, but the model can only predict tokens.
(There are a lot of contexts in which this distinction doesn't matter, but I would argue that it does matter for a meaningful definition of human-like understanding.)