Large Language Models and Intelligence Analysis
cetas.turing.ac.uk
cetas.turing.ac.uk
1. They claim that "an LLM does not understand the semantic meaning of a sentence in a linguistic sense, but rather calculates mathematically what the most likely next word should be based on the input to the model." My opinion, shared by many others in the field: In order to compute an accurate probability distribution over all possible next words, an LLM must understand semantic meaning, internally, in some fashion. How else could an LLM do that?
2. The authors claim that an LLM "is extremely good at determining the most likely next sequence – and convincingly so – but has no inherent representation of what those words mean." My opinion, shared by many others in the field: In order to compute accurate joint probability distributions over all possible subsequent sequences of words, an LLM must have some kind of internal representation of what those words mean. How else could an LLM do that?
3. The authors claim that "LLMs do not encode an understanding of our world." My opinion, shared by many others in the field: In order to be able chat with people, write essays, pass exams, etc., LLMs must encode an understanding of our world, even if that understanding is obtained second-hand from statistical patterns seen in language, images, etc. How else could an LLM do all those things?
- The LLM is always answering "what it interprets that YOU want to hear", including all the input biases (its training data, its construction and settings, your prompt, etc.).
- Instead, if the LLM "thought" ("understood" with an "inherent representation" to "encode an understanding of our world"), then the LLM could answer "what IT thinks and applied reasoning (in some way, consciously or not) to arrive at", then.
To restate that, it's the difference between GUESSING WHAT YOU WANT TO HEAR, GIVEN ALL INPUTS versus GIVING ITS OWN HONEST ASSESSMENT GIVEN ALL OF ITS INPUTS.
In other words, IT HAS NO HONEST ASSESSMENT OF ITS OWN, per se. It cannot assess on its own behalf. It only reacts in the way it has been configured to.
Am I on the right track, trying to tease apart the disparity of opinion on those 3 claims?
As Hans Moravec wrote in 1998, when he predicted intelligent machines would appear in the 2020's:
> Only on the outside, where they can be appreciated as a whole, will the impression of intelligence emerge. A human brain, too, does not exhibit the intelligence under a neurobiologist's microscope that it does participating in a lively conversation.
Perhaps by having a dataset of symbols that corresponds to symbols that you similarly understand how they can be put together, as well by having a model for how to respond to inputs given some prompts.
You can fill a SQL database with different kinds of apples and their prices.
You can "ask" the price of a Golden Delicious apple, and the DBMS responds--intelligently, with the "right" answer given the question asked in semantic business language.
How could the DBMS do that, if it didn't "understand" the data?
The answer is, the machine system contained a system of symbols and a model for how to give you want you want. But the system didn't utilize any kind of first principles in its "understanding".
I hypothesize there is a "true sense of understanding" or "understanding before anything else" that machines can mimic but that--currently--only humans are good at (even with LLM advances). At least, I haven't seen any evidence to the contrary.
As an aside, if there really were an LLM that could understand, reason, "think" (so that it "understood" with an "inherent representation" to "encode an understanding of our world"), then it would, by all likelihood, be spouting some disruptive-seeming output (in the best sense possible--as in, output supportive of paradigm shifts from status quo positions in place for no reason other than momentum) across domains, and I'm just not seeing that. (Sure, the machine might be coerced to APPEAR to do such things, but that would be an illusion, and not doing it on its own--not true intelligence analysis.)
That's the same as saying the machine has some kind of internal model of the world, with symbols it has learned standing in for concepts in the world, as understood by the machine. Representing things with symbols is in fact what GOFAI tried to do, but explicitly instead of via learning.