Given that large language models don't have any actual knowledge (in a human sense) of the data they've been trained on other than raw statistics, can they be said to "reason" about anything?
- What distinguishes thinking and computing?
- What is more fundamental? Form or meaning?
> is this all just a game of syntax that _feels_ like semantics from the inside?
The point is that in case of LLMs, there is no mystery here: it is syntactic templates (x) statistics. But do we know that 'syntax' is all there is to our mental reasoning? We don't know this (yet). I have flirted with the notion - heretical for me since I 'believe' in primacy of meaning, not form - and can kinda squint and see meaning possibly being a 'virtual' phenomena and possibly simply a 'narrative' of 'arcs' connecting 'facts' mapped to a syntactic template. That meaning is simply a process. [So, not even "42" ..]
I am not trying to argue that language model are close to human level reasoning or whatever, but it is not obvious to me that a language model of some sort is fundamentally unable to to achieve this.
However, the current incarnation of LLMs like ChatGPT fail at straightforward reasoning around things like categorizing sailfish and mathematical proofs. I'm incredibly impressed at how powerful the simplistic methods they use are, but the deficiencies make it clear that it's not abstractly reasoning from definitions in the same way humans are able to.
As far as I understand, LLMs have no model of reality to base results on; they're just an enormously complex statistical model of "what is the next word in the series".
Because humans learn about language in real world contexts, accompanied by multiple sensory streams. LLMs learn about it solely by being exposed to text. Imagine a child kept in a closet who learns about the world solely by reading books, with no indication which are fact and which are fiction, and no exposure to the physical world outside. LLMs cannot reason, only regurgitate probabilistically.
The next two were, "The Chinese Room: Searle and Artificial Intelligence" edited by John Preston and Mark Bishop (2002) (Preston and Bishop did publish Views into the Chinese Room in 2002, but I can't find anything by the given title) and "The Chinese Room Argument" by David Chalmers (2006) (Chalmers has written on the topic, and published three articles in 2006, that's not one of them.
Also:
"Two cities: one, a city of “knights”, and the other of “knaves”. The residents of the two cities like to visit each other. One day, a man enters one of the cities, but doesn’t know which. He decides to ask the first person he meets where he is. The answer he receives is that he is in the “knaves” city."
produced the result:
"The man cannot be sure that the person he met was telling the truth, because the person could be a knight from the city of knights, who always lie, or a knave from the city of knaves, who always tell the truth.
"To determine which city he is in, the man could ask the second person he meets the same question. If the first person was a knight, then the second person would also be a knight, and would lie about being in the city of knights. If the first person was a knave, then the second person would also be a knave, and would tell the truth about being in the city of knaves.
"By asking the same question to multiple people and analyzing their responses, the man can eventually determine which city he is in. This problem, known as the "Knights and Knaves" puzzle, is a classic example of logical reasoning and problem-solving."
It seems to recognize the "knights and knaves" form of puzzle, but doesn't get the knights and knaves right. And it's not getting the reasoning part at all even given varying prompts.
As it turns out, we kinda do know what is going on in a LLM, and there's no model of reality for it to base reasoning on in there.
Does that mean that's impossible? No. Does it mean seeing reasoning there is making the same mistake as those who enjoyed chatting with Parry and Eliza? Probably.
I had to add the part about the behavior of knights and knaves, because ChatGPT seems to recognize the puzzle but gets them wrong, and it seems to be completely ignoring the visiting behavior. Further, the answer doesn't really work.
It seems that nonlinear interpolation in a rich enough latent space can give you a sufficiently good approximation to reasoning.
Do we know this to be true? I guess I'm not aware that we understand very well how memories are stored/recalled in a human brain.
To call the model "just statistics" is meaningless. To call it "just like a brain" is unsupported. There are better ways to debate this.
So which mental model is superior when evaluated under the criterion of "ability to interact productively with the tool"? (a) It's just statistics / stringing together phrases from the internet or (b) it's more like a human brain with vastly superior general knowledge?
Personally, I would put money on (b) resulting in better interactions. This is probably something we could study empirically.
But ultimately I don't really care that much about the philosophical issues, my mental model of it (a more nuanced version of b) has enabled me to use it productively almost every day to do things that I couldn't do before.
I don't know if the knowledge in an LLM is stored like in my neurons - I know nothing about neurobiology. But after having used ChatGPT extensively I can certainly tell that ChatGPT does not mobilize knowledge like humans.
Except on raw power, and this is an architecture where processing power and memory lead to each other, so it's missing a lot of raw memory too.
We don't know that this is true. But even if it is, there is a critical difference in the knowledge stored. Human knowledge incorporates direct experience of the world through multiple senses, while an LLM's knowledge is solely derived from text and so misses some critical physical constraints and contexts.