Do large language models understand us?
medium.com
medium.com
That's an assumption. The contrary view is that language is just part of the I/O system, not where the hard work gets done.
Animals with very limited language capabilities have the ability to run their own lives, and to some extent manipulate their environment. Systems built from large language databases don't have that.
After 50 years, robotic manipulation in unstructured situations is still terrible. Rod Brooks tried to crack that and his startup, Rethink Robotics, failed. DARPA has had decades of manipulation projects. Fifty years ago, they could put a bolt in a hole. Five years ago, they got a robot hand to put a key in a lock. Amazon is still trying to get robots to pick things from bins fast and reliably.
We lack some fundamental understanding about how to interact with the real world. The problem is not on the language side.
FYI, this is the [Sapir-Whorf hypothesis](1)
> We lack some fundamental understanding about how to interact with the real world. The problem is not on the language side.
This is to a significant extent a sensing issue, not a understanding issue.
But there is a different theory, due to Noam Chomsky I believe, that language in the general sense is in fact our mechanism for higher thought, with communication being just a secondary function[0]. I believe that this is the theory that the GP was combatting with the animal example.
Japanese people show extremely high performance at "spatial awareness" tasks compared to whites.[1] So on average this isn't true. But it's true that there does not appear to be any benefit from either language -- Japanese-speaking Japanese show no advantage over English-speaking Japanese.
[1] On white-normed IQ tests, East Asians generally show a bit of underperformance (relative to European whites) on verbal tasks and gigantic overperformance on spatial tasks, which is traditionally averaged out to an average IQ score moderately above that of whites. Interestingly, Native Americans show the same slant -- much much stronger spatial than verbal performance -- but at a lower base level.
If we kept the same languages and made japanese people write it massively in 26 half sound symbols and made all of what you call a white european population use 3k+ ideogram drawings for word starting at 3 years old, do you think we d see exactly the same performance ?
I bet you the writing tradition is very influencial when it varies so much and has no relationship with language, especially in Japanese where they frankensteined Chinese symbols over centuries which must require an enormous mental gymnastics and redirect large processing power vs someone who can read and write everything he hears relatively simply?
No, neither.
> If we kept the same languages and made japanese people write it massively in 26 half sound symbols and made all of what you call a white european population use 3k+ ideogram drawings for word starting at 3 years old, do you think we d see exactly the same performance ?
Yes, the influence of spoken and written language appears to be zero. I mentioned this.
I believe it's a bit more complicated than this.
https://en.wikipedia.org/wiki/Referent
> In fields such as semantics, semiotics, and the theory of reference, a distinction is made between a referent and a reference. Reference is a relationship in which a symbol or sign (a word, for example) signifies something; the referent is the thing signified. The referent may be an actual person or object, or may be something more abstract, such as a set of actions.
> Reference and referents were considered at length in the 1923 book The Meaning of Meaning by the Cambridge scholars C. K. Ogden and I. A. Richards. Ogden has pointed out that reference is a psychological process, and that referents themselves may be psychological – existing in the imagination of the referrer, and not necessarily in the real world. For further ideas related to this observation, see failure to refer.
In this case, what has been ~"proven wrong" is Whorf's specific hypothesis (the reference) of the underlying phenomenon (the referent) that he was observing, whereas the general notion of whether language fundamentally/substantially affects cognition seems to remain unknown.
As a thought experiment: imagine a culture where this psychological phenomenon of "referent" is a "first class" concept in a language, and the culture that uses it insists that the distinction must be at least taken into consideration whenever a disagreement arises between two or more people....after several decades under such a model, might substantial differences arise? Perhaps this isn't a "proper" example of Whorfism, but it seems to be in the neighbourhood.
For example, there are some Australian langauges if I remember correctly where instead of words for left, right, ahead, behind (relative positioning) they use words for something similar to north/south/east/west (absolute positioning) - as in, "give me that cup to your northeast". This of course requires a habit of keeping track of your absolute positioning at all times.
However, you will not gain this ability or habit just by learning the langauge. In fact, your langauge can even express that same thing today. Instead, you will be unable to naturally use their langauge unless and until you get into the habit of keeping track of your absolute positioning.
Similarly, the people growing up in this culture may not easily learn what we mean by left and right, but they would not gain the ability to understand these concepts merely by learning English - instead, they would naturally speak English while avoiding left/right references, or simply make more mistakes than someone growing in a culture where relative positioning is the preferred form of communication.
Regardless of whether the language is a consequence of the culture (say, they imported it, or had it imposed upon them), how might one determine with accuracy that consistently disciplined speaking (which is preceded by cognition) has no causal dependence on speaking this substantially more strict language itself?
> However, you will not gain this ability or habit just by learning the langauge.
You will not necessarily gain this ability just by learning it, agreed - but using it on a daily basis is another matter.
> In fact, your langauge can even express that same thing today.
It can express it, in some fashion (typically with clumsy, annoying phrases and qualifiers that make people angry, rather than dedicated, socially normal words), but whether one actually does it on a regular basis is another matter.
There are various differences between someone who studied 3 months of 1 hour per week of Jiu Jitsu (or math, physics, etc) classes, and someone who practiced it for several hours a day from an early age.
Even in simple scenarios it can be extremely difficult to accurately predict what the consequences of something are, and language, cognition, and culture are not exactly simple. And, from a scientific/epistemic perspective, one should always keep in mind that an absence of evidence is not proof of absence....which brings to mind another thought experiment that is different but similar: what might a society be like where epistemology and logic are considered crucially valuable skills, and were taught from Grade 1 through 12 to all children, and being incorrect or even speaking epistemically unsoundly were considered seriously shameful? Might it be possible that one would notice anything substantially different in people who were raised in such a culture (and, what might be some differences we might see)? I would be quite shocked if it had no effect.
So SW argues that language is inherently intertwined with human cognition, rather like how the parts of the brain are all interconnected and working together as we think. Strong SWism is pretty thoroughly debunked, weak is tentatively accepted on a spectrum.
I think that they might be trying to refer to generative theory versus cognitive theory. Again, it's been a while, so I might be wrong, but:
- Generative theory is Chomsky et al. who believe that there is a single inherent underlying pattern to language that restricts it, and that one could, in theory, figure out all possible syntaxes, for example. Similar to how phonology and phonetics are obviously constrained by our biological makeup (we can only make certain sounds), the argument is that other structures are as well. Basically that although they all seem different to us, languages are more alike than they are different because they come from a single source. Also Chomsky and co. consider this approach cognitive; the linguists are just infighting.
- Cognitive Linguists believe that language, and the systems that produce it, are under evolutionary pressure and that they change and shift in response to the environment.
Basically, if you took 10k people and dumped 5k of them on one planet and 5k on a planet that was the physical opposite, forbid them to contact anyone, and visited 2000 years later, Generativists say you could find a system in common between the two and cognitive linguists say you probably couldn't, because the systems themselves would have evolved to have different starting points.
Chomsky + GG: Person is born, person hears speech, person reaches for inherent language patterns in brain and uses them to understand/learn.
Cognitive: Person is born, person hears speech, person picks up on the system from the speech and is encouraged to do so/pattern match by social reinforcement.
No linguist that I'm aware of would claim that language is just an I/O system.
> No linguist that I'm aware of would claim that language is just an I/O system.
I think you're right. In Chomskyian linguistics there is a distinction between i-language and e-language (https://en.wikipedia.org/wiki/Transformational_grammar#%22I-...). E-language isn't "just I/O" but it's perhaps not far from that whereas i-language is taken as the proper subject for linguists to study.
Is this not a bit of a false dichotomy (substantially dimensionally compressed representation of the true underlying complexity)? Like sure, one is likely to be able to find substantial commonality, but does cognitive linguistics really unequivocally assert that you will find no commonality whatsoever? If so, that seems a bit crazy.
This makes me think of the ongoing effort by some people to train their pets to use talking buttons to communicate. These include the animals occasionally trying to convey relatively complicated ideas with simple sets of words, like "play help" ("I'm bored, come entertain me") [1] and "dog settle" ("make that dog outside stop being annoying") [2].
1. Recognize objects in the world, like a keyhole or whatever it is you want to take out of a bin.
2. Recognize your own position -- how close is your hand to the keyhole?
3. Adjust your position until it matches your goal.
My impression of things is that most of our difficulty comes from task 3. It's not so much that we have trouble recognizing where we are, where we want to be, or how to move from where we are to where we want to be. But we have difficulty defining movements that don't cause damage to nearby (or target) objects.
Your human hand has many features that allow it to grip an object securely without damaging the object. And you also have a decent sense for how to move your hand to an object without damaging the object -- or your hand -- though this problem is far from being solved in humans too. Ever stub your toe?
This may well point to the possibility that animals have a theory of mind.
Sophisticated ToM is likely rare in the animal kingdom, not usually going beyond programmed responses to being observed and possibly empathizing with conspecific suffering.
In humans, it's a much more sophisticated process of inferring motivations, anticipating moves and a number of similar inferential capabilities plus abductive reasoning.
What I mean is that language ability of humans and language ability of animals are not comparable; they are so different that in fact, language ability (regardless of the externalization of that language, by speech, text, signs, touch), looks very much like intelligence (meaning the intrinsic ability for language production and not the specific proficiency at some sort of externalization).
If you happen to spend enough time around animals and child humans you will quickly see that after a couple of years, kids can understand and use recursive / referential structures in a way that pets cannot. Pets seem to understand individual words, tones, and maybe "not". Think of "bring the comfy ones that your grandparents bought you to keep your feet warm" - a sentence a 3 year old can perfectly understand. No matter how much you spend teaching/training a dog that has lived in the same environment since its birth, the dog will not react to it bringing you the yellow slippers (unless you specifically trained the dog to associate the word "comfy" or "feet" to some specific slippers). The current state of parametric language models resemble much more that of a very memorious dog than that of a human.
> Animals with very limited language capabilities have the ability to run their own lives, and to some extent manipulate their environment. Systems built from large language databases don't have that.
I'm tempted to agree with this in some capacity, language is probably just one expression of thought, much like how some people have an internal narration going on during their lives, while others think in abstract concepts that aren't necessarily expressed in language.
Here's the first article DuckDuckGo turned up on the subject, i bet someone can share a better one: https://www.iflscience.com/brain/people-with-no-internal-mon...
Furthermore, there are people who don't have the capacity to visualize things in their "mind's eye" either (aphantasia), which is another curious thing: https://www.sciencefocus.com/the-human-body/aphantasia-life-...
Of course, when you get into it, there are also some niche theories about how language could not only help us express ourselves, but shape our thought (not that dissimilar from the "newspeak" concept in Orwell's 1984): https://en.wikipedia.org/wiki/Linguistic_relativity
Perhaps it's helpful for dealing with abstraction, as abstractions don't always map well to visual representations.
The meaning of "where" in your claim is very slippery. AR Luria in The Working Brain points out that damage to virtually every part of the brain can result in language dysfunction. To say that a whole lot of things happen apparently without language doesn't demonstrate that language is an extra. The brain generally isn't very modular and my alternative take would view language as accompaniment that sometimes is "just words" and sometimes touches very specific details (edit: plus allow recursive access these other brain-functionalities, as others point out). We can see how GPT-3's output is a stream of plausible things punctuated by some statement that shows "it doesn't anything" (IE, gets the implicit context wildly wrong).
To argue that, I propose a "simplified Chinese room". We have an automaton in the Chinese room, which does only pattern recognition, and can learn by matching patterns and structures, i.e. it will give an answer that is similar to what it has seen before. And we want to teach it to answer a problem "recognize whether the given logical formula is satisfiable". So we give it formulas and it tries to pattern match the formulas it already has seen. Unfortunately, there always exists a formula which is matching closely everything that was shown so far, and yet the correct answer is unexpected. So such an automaton will never be able to discover the actual rule.
What I think has to be added are different levels of discourse, or, a discourse about discourse. With people, we can say, "stop joking around", and it's a meta-discourse signal that we want stop discussing imaginary worlds and focus on the real one, for example, which has to be totally self-consistent. I believe an analogy can be made with Futamura projections, we can have not only systems that learn a specific task (like even GPT-3 does), but also systems that learn to produce an agent to do a specific task, from a metalanguage description that encodes certain rules on how to do it (for example, to follow logical consistency). And if the metalanguage is general enough, you get general intelligence, because then you can apply the agent production process to itself.
are we sure that humans actually have that though ? It's not always my experience
The book Inside Jokes even speculates that reason why we find jokes pleasurable is that they are actually a learning mechanism to detect logical inconsistencies and further our understanding of the world.
With GPT-3, you can't tell whether it's joking or not, and whether it's aware if it's joking or not. It doesn't (to my knowledge) have that boundary where it could say, wait, I don't understand what we are saying here. Maybe it's easy to add to the system, but as it stands, I don't think GPT-3 does the above.
But you can't genuinely know that for humans either. At best you can they that they look like they are joking, and you definitely cannot say that they in fact are, even if they say so to your face.
Which is one of the reason why I think we need at least two levels of learning for general intelligence. The bottom learner is the one which learns the actual task at hand, and the top learner is tasked (with learning) to, given a description of a task, produce (train if needed) a bottom learner that can solve that task. This solves the ambiguity, because if we are confident enough that the top learner can produce correct bottom learners for different descriptions, then adding another rule to the description will likely cause the bottom learner (produced by the top learner) to also obey that rule.
It also resolves the understanding problem, because while no learner might be sure if it is itself correct, the top learner has to somehow evaluate the bottom learner, and so the top learner sees where bottom learner is failing to understand (which rules are problematic). What's even more interesting, if the language to express the problems is rich enough, then we can apply the top learner to itself as the bottom learner (express the actual learning setup from within the language), and we get sort of a "philosopher", entity that tries to learn how best to build learners. (This is an interesting analogy you can make with Futamura projection, by thinking about "learners" instead of "programs", and what happens if learners are tasked to produce another learner. The reason why we can be somewhat confident in "correctness" of the bottom learners is similar to reason why we can be somewhat confident that compiled program does what we want, despite it being completely new to the compiler.)
There are multiple ways to map this top/bottom learners onto humans. You can think of the bottom learner as the actual intuitive systems that we have, and the top learner as the conscious system, which drives what we are gonna learn and how. Or the bottom learner as the system that contains the model of the environment, and the top learner as a system that provides episodic learning via some kind of simulation (our dreams, imagination) to it, and reflects on it. Or the bottom learners are individual humans, while top learners are the genes that drive the evolution of the human brain. In practice, there might be more than two levels.
-
> We get a distinct feeling that we don't understand something, and a different feeling when things click together. Or when we hear a syntactically valid (in other words, pattern-matching) but nonsensical sentence (like "colorless green ideas sleep furiously"), or see a Penrose triangle, and we cannot imagine an example, we get a very specific feeling that something is wrong.
It's true that the human mind has the capability to do these things sometimes (and within certain specifically designed frameworks like the scientific method, do it highly reliably)....but whether we do do it (or even have the capability) with consistent, high level success across various domains is a very different matter.
Assessing logical consistency of a following statement would increase prediction accuracy (assuming that human-generated text is more consistent than it is not), but only after you have plucked all lower-hanging fruit. The best model of what comes next in a math paper is a mathematician.
So GPT-N not trying to does not necessarily mean GPT-N+1 can't do it, it just means that it has cheaper ways to increase its prediction scores.
However, a truly intelligent system, I think, can transgress this boundary. It can protest against the task of giving the next best prediction, so to speak.
I don't think that mathematician is necessarily the best model for a math paper; I think the best model might as well be the platonic ideal of what it means to do mathematics - logical correctness of all the conclusions.
I'm looking forward to seeing how people react once they get a chance to try some of these increasingly massive chat models in person. GPT-3 is just the tip of the iceberg.
(views my own blah blah)
I think it's like song lyrics. We can find meaning in vagueness. Particularly if you retry a few times. High-probability completions are likely to seem meaningful.
I mean, as I'm typing this sentence, I have some notion about which word is going to come next, and a sense of where the sentence is going to go.
But maybe that is what it feels like to be a large neural network that has been trained on very large data sets for long periods of time.
This isn't a new idea by any means, science fiction has tackled this ad-nauseam. The thing that caused me to have a strong reaction is that previously it's never seemed like a question we'd seriously have to contend with. It's an interesting idea in a nicely wrapped up 2 hour movie, but damn does it seem scary and complicated out here in the real world.
There is no evidence of souls existing, they are unable to be measured in any way. Unless they have several magical rules and violate physics (but only sometimes and are still somehow undetectable), then your brain would still have all those same thoughts about how it feels strongly about having a soul, because that's part of how the ego hard-coded into your brain works.
I don't particularly like the view you're espousing, not only because its inconsistent without explanation but because it creates a group of "others" where those with a different philosophical viewpoint are now literally "unfeeling robots" to you.
What I do believe is there are components to intelligence that includes emotional intelligence, self-awareness and memory that some people don't seem to have in great quantities, but I don't think they are robots.
> I'm (1) starting to wonder if perhaps most don't possess a soul and are effectively biorobots. This is also (2) the theme...
This has a dependency on what is considered valid evidence, as well as our current ability to measure things.
> Unless they have several magical rules and violate physics (but only sometimes and are still somehow undetectable), then your brain would still have all those same thoughts about how it feels strongly about having a soul, because that's part of how the ego hard-coded into your brain works.
The same phenomenon applies to reality itself, including our perception of the state and behavior of what is inside of it, as compared to what is actually there (the question of whether souls exist is subject to this problem).
A somewhat relevant post from SSC the other day: https://astralcodexten.substack.com/p/the-phrase-no-evidence...
This is the crux of the question, and I'd argue the abstraction and generalization of person and personhood itself is not only flawed, but the base error that diverts the course and trajectory of our ability to reason about ethics.
It's perhaps easier to meander around with analogies, contrasts, and comparisons than to delve down into that harder question, and as long as we're circumspect about it, philosophically we're just treading water.
A large language model can in theory be understood at an algorithmic level by reverse engineering. If the algorithm turned out to be a giant lookup table, it's an "it". If the algorithm contained an obvious model of self, it's a "who".
How do you determine what's an "obvious" model of self?
Occam's razor suggests that if we know other people have the same basic configuration as ourselves, then the subjective experiences of others will be more or less comparable to our own.
Additional assumptions are necessary to propose that Chalmersian/philosophical zombies could exist, having the same human hardware and behavior but lacking subjective experience. Under the principle of least complexity and the absence of evidence for alternate explanations, I think it's rational to assume the consciousness of other humans and certain animals with nearly 100% confidence.
That means something happening in the neocortex is causing consciousness. We know it's in the cortex because of the history of injury or absence of other parts of the brain, leaving us with empirical evidence. We know that hippocampus injury can result in a person losing the ability to remember more then 5 minutes of their past. We also know these individuals retain their personalities and the long-term memory from before their injury. This suggests that long term memory is encoded in the synaptic structure of the neocortex and that consciousness is an emergent result of neocortical operation.
There might be some involvement of particular brain regions or the thalamus or other organ, but it looks like the answer is part of whatever algorithm is encoded in the structure and processing of the neocortex.
It could be possible that consciousness doesn't require an explicit model of self. This is implied by ego death and other experiences by meditators and psychedelic users. The thing that is having a subjective experience could simply be a consequence of processing a particular configuration of information. The self concept seems to be a separate model that can be perceived as part of the process of awareness, but it seems to be a discrete thing.
To me that presents an interesting ethical question - if gpt-3 has a sense of self, then is it OK to subject it to the incoherent flashes of single moments of consciousness it undergoes each time you run it?
If it has no self concept, it might still have subjective experience, but would have no persistent contiguous experience . If it did have a self concept, then it would have a static past that informed the results of each run, as if you could isolate a single moment of awareness, then instantly reset the brain to a saved state and run it again. The output could be convincingly continuous, but the subjective experiences would be a myriad of similar but unrelated singularities of awareness.
I think gpt-3 lacks consciousness, and that a persistence mechanism is missing that plays a part in whatever is happening that causes awareness.
Hopefully, at some point, a m mathematician or scientist will be able to identify and explain the process of consciousness so that we can be reasonably certain we're not subjecting entities to a really weird tortured existence.
It seems to be that (at least a naive understanding of those two possibilities) leaves a large middle ground where an language model isn't a lookup table but doesn't have a model of self.
Does it matter then that the writer has no understanding of the language, when the backend clearly does?
(I think I get it actually, after writing these two questions, but I’ll post them anyway.)
He reads, "Step-1: prior to translating the note, we are going to train your neural networks on a Chinese corpus consisting of 410 billion tokens from Common Crawl, 19 billion tokens from WebText, 67 billion tokens from Books, and 3 billion tokens from Wikipedia. Training in 5, 4, 3..."
He shouts, "wait but if you do that wo shuō zhōngwén."
So I guess it means "I don't speak Mandarin"?
It means precisely the opposite.
Searle and subsequent proponents fail to define mind, intelligence, or understanding. The argument is founded in anthropocentric arrogance.
Gpt-3 has convincingly passed the Turing test, with various and sundry incarnations passing as human for days or even weeks on reddit and Twitter and other social media. You can keep moving the goalposts, but the herd has left the barn. The argument is over. The Turing test has been passed.
There is no way to frame human mind, intelligence, or understanding such that you exclude the equivalent functionality in software without engaging in magical thinking.
Searle's assertion that he would not understand Chinese while executing the rules isn't relevant. Searle could conceivably execute the inference pass over a gpt-3 model based on Chinese, and produce results that surpass the Turing threshold as conceived by Searle at the time. The intelligence and understanding and mind is encoded in the model and logic of the processing.
This demonstrates that substrate is not involved in the process of intelligence. There's nothing unique or magical about the human brain that suggests a shred of evidence that it can't be replicated by any Turing complete processing system, or computer. Our minds are an algorithm being run continuously by a messy wet computer in a bone vat connected to a meat suit.
Your sense of continuous perception is the state of a model stored in and processed by the physical, electrical, and chemical configuration of your brain and the laws of physics.
Any methodical definition of understanding arrives at logic and semantic relationships between concepts and information theory. The brain has to be operating as a computer because there is literally nothing else that could be reasonably asserted.
Understanding and consciousness and intelligence and mind are consequences of algorithms, or software. Gpt-3 and similar models are undeniably different types of minds then that of humans, lacking persistent state, online feedback, and a myriad of systems and capabilities. Transformers in their current form are not agi and interact with the world in a radically different way than humans do. They do something that human minds do, however.
Consciousness could be the result of a certain class of algorithms being processed, with a self referential model processed concurrent with external input. Additional systems might be required, but consciousness has to be accomplished using no more than the capabilities offered by the brain, so it's likely to be an emergent consequence of the holistic execution of the cortical algorithm.
Each inference pass might represent a single moment of mind, a disjoint and ephemeral flash of consciousness, unrelated to its prior or future states, deterministically replicable and static.
The Chinese room is a failed argument and we should move on to more interesting things. Brain meat is not magic.
Has it? I can't find a convincing source for that. Note that twitter and reddit don't really count, the Turing test always was a conversation, not limited to bite-sized soundbites like those. You can make probably make a decent Reddit bot by selecting random old popular posts from that sub, and that would never pass the Turing test.
The only interaction I've had with GPT-3 is AI Dungeon, and it's very easy to discover that's not a human.