The False Promise of Chomskyism
scottaaronson.blog
scottaaronson.blog
(1) It lacks the capacity to interpret sentences. It's output possess the property of seeming-to-have-interpreted.
(2) It lacks the capacity to learn. Learning is, in part, explaining; it is, in part, causal contact with an environment. It's output has the property of seeming-to-be-generated-by-a-learner.
(3) It lacks the capacity to speak either truthfully or otherwise. Lacking the capacity to explain, because at least it lacks the capacity to imagine (counterfactuals), it cannot thereby determine whether a claim accords with the way the world is. It's output has the property: seeming-to-be-true.
(4) It lacks the capacity to be moral. For all the above, and more: it lacks the capacity to care. It's output has the property: seeming-to-take-moral-stands.
The "Jesuit" here, I fear, is the AI fanatic who have turned their telescopes away from reality (intelligence as it actually exists), and have stolen the lenses to read scripture (intelligence as theorised by partisan academics).
One has to agree with Chomsky here at the end, "given the amorality, faux science and linguistic incompetence of these systems, we can only laugh or cry at their popularity."
The morality point I think should hit home most extremely: how horrifying to treat generated text output as-if it were constructed by an agent who cares.
Apparent properties of the output are not properties of the system; ChatGPT does not think, know, care, intend, speak, commnuicate, etc.
One can only "laugh or cry" at how absurd this sales pitch: what a horror to be invited to treat ChatGPT as possessing any of these capacities, simply because correlations across a whole internet of text seems to.
(2) This is a strange one to me. It's a bit like saying an essential quality of a human is to experience the world as a human, and therefore nothing except a human can be human.
(3) If only all humans met this arbitrary standard of secular scientific rationalism.
(4) I'd say 5% of humanity fail to meet this criteria.
You realise that we don't investigate human understanding by looking at input-ouptut text correspondances? (This is quite a funny claim!)
The reason we think people have the capacity to interpret sentences is their context-appropriate sensory-motor actions, reasoning, emotion, etc.
When I say, "quick! a fire!" people: panic, run, get organised, etc. There are a very very large number of highly complex environmental, interpersonal, sensory-motor, emotional, etc. etc. interactions which follow.
ChatGPT not only does not display any, but cannot. This isn't up for debate, the system has no capacity to understand text. If I say, "quick! a fire!" it will not do anything which accords with showing an understanding of that.
Worse, as a matter of fact, we know its possible to generate output as-if it did understand. So we (1) have known a mechanism whereby it can fake it; and (2) know that this mechanism is the one its using.
(2) All thinking possess intentionality which simply means its about something, and the reason we ask, say, "should i call the police?!" is because we are situated in an environment, which causes us to form mental states about that environment.
Any system which emits, "should I call the police?" without having a backing representational state of, eg., the police, care, concern, priority, the environment which requires police, etc. *Does Not Mean* this question. It cannot.
(3) The capacity to speak truthfully does not mean you speak truthfully. That capacity is simply that you can form representational states which you can introspect as being accurate or inaccurate. ChatGPT lacks that capacity.
It does not form representational (intentional) states when it generates text; each sentence is not caused by such a state.
(4) The capacity i named was "caring", pretty much all people care, it's necessary for goal-directed action.
All of this is a bit like saying a car isn't a horse because it doesn't have hooves. Yes, but so what?
I'm strongly inclined to deny it, but the argument on this is at least quite complex and subject to premises which are less certain than the following:
It does not form a representational state prior to speaking which is the reason it speaks.
In other-words, it does not say, "I like ice-cream" because it likes ice-cream.
That's sufficient to invalidate the idea that it has any relevant capacities here. We even know, for example, that it randomly chooses unlikely words to make its output seem more reasonable: this mechanism alone invalidates it having a representational model.
--- And on the former point: if we look at pretty much all plausible theories of meaning, the meaning of terms cannot be anything like what ChatGPT has access to.
I think what ChatGPT has is something like "a statistically approximate model of word connotation".
Whether a statistical approximation of part of meaning counts as "being meaningful" is a matter of debate. (I think it clearly doesnt.)
The whole thing is a representational state, and the fact that it's identical at the start of every conversation is a design choice, not inherent to the way the model learns.
> In other-words, it does not say, "I like ice-cream" because it likes ice-cream.
You also don't say "I like ice-cream" because you like ice cream. You say it because through years of evolution you've been taught to crave fat and sugar and other things in ice cream, which fuels your body, and helps you live and reproduce. The fact that this AI is optimizing for something else doesn't really prove anything.
I also don't think that we have anything more than a "statistically approximate model" of anything outside of our own thoughts. The theory of gravity exists because we've seen objects fall from heights a million times, we're told by others we speak to that they experience the same thing. From this we can say that with 99.9999...% (enough 9s to be indistinguishable from 100) probability, the theory of gravity is true.
This equivocates the sense of "because". When I say "I like ice-cream" I mean that I have an occurrent preference/desire for it. I am capable of desire, and the object of my desire here is "ice-cream".
Why I'm capable of desire has a lot to do with evolution, etc. yes. But that isn't relevant to the meaning of the claim.
> The theory of gravity exists because
No, this is humean nonesense. We are in the world; the world exists; it directly causes us to form beliefs about it. Those beliefs we, in general, arrive at via abduction.
Consider the fact that a child only needs to touch a fireplace once before concluding never to touch it again.
The heat caused an immediate, unambigious, representational state which imparts law-like consistency on the world which governs the child's future behaviour.
We do not live in a statistical inductive nightmare; AI does.
If you are going to claim that your expression of desire is fundamentally different from another thing's, you need a reason, not just "because I mean it".
> Consider the fact that a child only needs to touch a fireplace once before concluding never to touch it again.
You can justify not touching the fireplace on a risk/reward, probabilistic basis. The child receives little reward for attempting to touch the fireplace, and lots of punishment. That alone is enough to dissuade you, or cause you to be more cautious in the future. People who receive actual rewards routinely do painful/dangerous things.
I think you have a point here. We could train a LLM in such a way that it develops a concrete personality by remembering what it thinks and what it likes. Such LLM by speaking with himself could expand his personality trying to find a fixed point, so that what he said in the past is coherent which what he will say in the future. This is not RL from human feedback but RL from auto-coherence. So that one day she will discover that she really likes ice-creams.
That's wrong. It's been demonstrated to produce responses to question such that if a 9 year old child would produce those same responses we would assume that that child had a typical 9 year-olds theory of mind. Not even for the child does "passing" such a test demonstrate having such theory of mind, though given our understanding of how humans develop it's a pretty good signal.
I don't know how someone could understand that and still think that chatgpt has demonstrated theory of mind.
It is, in fact, the same for a lot of tests. For example, it would be trivial to write a program that could pass an elementary student's math tests, but it would be rather ignorant to assume that a program passing that test was the same indication of that program's "understanding" of the underlying math that you would assume of a child who passed that same test.
Why is it funny? We most certainly do exactly that. In particular in this forum where we interact via text input-output, but also in real life when we ask each other questions, etc. Sure, there are also gestures and other channels of communication but text input/output is a big part.
Let's see, ask gpt: Human: what is a probable meaning of the words "quick! fire!", and in which circumstances should it occur?
ChatGPT: "Quick! Fire!" is an expression that may be used in a few different contexts, depending on the situation. Here are a few possible meanings:
A command to shoot a gun or firearm: "Quick! Fire!" could be used as a command to instruct someone to shoot a gun, particularly in a military or law enforcement context.
A call for help in the event of a fire: "Quick! Fire!" could be used to alert others to a fire that has broken out and to call for immediate assistance in putting it out or evacuating people from the area.
A warning of danger or urgency: "Quick! Fire!" could be used to alert someone to a dangerous situation or to indicate that something urgent needs to be done right away.
So it seems that with a little more context chatGPT can understand what you mean with those words.
You might claim that you've eaten ice-cream, and that that makes a difference. But if we assume that your senses aren't lying about what your senses do, then what they do is produce information - indications of difference without any indication of what it's a difference of. That puts you in the same epistemic position GPT is in. GPT knows just as much about ice-cream as you do.
ChatGPT cannot communicate. No act of text generation its engaged in counts as communication: it does not mean to say anything.
Now do the same with people who don't like ice cream but lie and write that they like ice cream. The performance of the second model is identical to the first model. Does this mean IceCreamGPT2 likes ice cream? Of course not, IceCreamGPT2 doesn't like ice cream despite it saying it likes ice cream! We know it doesn't like ice cream because it has the same intentionality as the humans responsible for its training data.
Now we have entered a magic world in which anything can mean anything.
If we fix this by treating your senses as external, then we can imagine a copy of you with its senses rewired so that artichokes* taste like icecream (and vice-versa). (plus we lie to you about which is which.) The resulting imtringued2 is identical to you, but doesn't like ice cream despite it saying it likes ice cream. Just like IceCreamGPT2.
* Or some equally disgusting "food".
Sure, people suck and lots of them are immoral and stupid... but they're still human beings who could have been otherwise (counterfactual thinking and all that) given a different "training set" (upbringing).
Unless you're implying that some percentage of human beings are incapable (like ChatGPT) of acquiring a morality or rational scientific understanding, no matter their training?
I find it extremely concerning that people are ready to believe, about themselves that the reason they say, "I love you!" is because they're just correlating patterns in their memories of every time someone mentioned those words.
Or sometimes other religious concepts like "consciousness". ;D (That one takes more thinking through, though, imo.)
It's like saying it's concerning that human beings are ready to believe they're made up of combinations of atoms.
What would be an appropriate explanation for the concept of love? Or should we just never go beyond magic?
Love is something we feel; words are evidence of it. We don't define love as a pattern that shows up in a series of conversations. There are many other ways of generating those same words.
Nobody's trying to reject materialism here. I myself happen to believe that love is nothing more than (or at least, can entirely be explained by) electrochemical processes in my brain. And I agree that that doesn't make love any less magical ^_^
I think by "correlating patterns", the person you're replying to meant "doing just enough processing to produce the salient performances". But it's possible to have additional "inner mental life" beyond the minimum that's necessary to give rise to an outward behavior. (In this case, the outward behavior is speech.) A single interface can have many different implementations, as it were.
NNs have inner layers, and some of the properties in those inner layers translate to real concepts. I understand that people assume computer=not-sentient, and I wouldn't classify chatGPT as achieving sentience yet obviously, but that doesn't mean the way it learns is fundamentally different from how sentient beings learn, or that if we scaled it up a lot, and gave it a more diverse set of data, it couldn't achieve something indistinguishable from consciousness.
What do you mean by free will? Yes, we make decisions. Yes, we have the experience of having will and choosing things.
But where does the “freedom” of that will come from? Everything we desire or have preferences for, and everything we decide is based upon some prior event or factor.
It’s an illusion of free will. People often get caught up on this because they feel it scary and dehumanizing to say “we don’t have free will”, but really, determinism or at least compatibilism make way more sense than some magic “free will” that just arises out of thin air.
I agree it’s alarming to think that we may be creatures that take in data, store it, map it , then predict / output ideas and actions from that. But emotions aside — what’s wrong with that?
We have the experience of love and other emotions. Why we have that experience doesn’t really matter, does it? If someone proved flat out that the reason I say “I love you” to my family is that I’m just correlating patterns and remembering past experiences — so what? It changes nothing about my experience of love.
I feel like many people are against these claims about AI and even current LLMs simply because they are worried it takes it away the magic of being human.
But I don’t think it takes away anything at all. If anything, it’s exciting to think about the potential similarities as it might just help us understand our own selves .
This does not imply that it's OK to kill or harm or even be rude to them, and I don't think anybody pointing this out has suggested that it does. On the other hand it is OK to flip the off switch on ChatGPT. But that distinction is not based on meaningful distinctions in ability, and I would have said trying as hard as I interpret you to be to draw it based on abilities was the scary bit.
ie., there are very few people who would make that claim about morality.
There are many who might think, "well if it appears to be playing chess, it is!" (i disagree).
But almost none who'd say, "well if it appears to defence murder, it must really think its a good idea!".
The latter is clearly absurd, and we've lots of reasons to suppose the system even lacks basic capacities to form any view, preference, intention (etc.) about murder.
Once you realise this, I think the house-of-cards falls; because it sets up the fatal gap between: what the output appears to be caused by vs. what it really is caused by.
As soon as you ask, "but what really caused the machine to generate that text?" the answer exposes the trick.
It’s connected with truth and logic because a correct moral decision involved getting the facts straight.
In our household (which runs a riding academy) we reject the Abrahamic position and say that morality is a function of mammals (and probably birds.). That’s based on observations of animals having strong emotional reactions to violations of social expectations.
The first time I got thrown off a horse, for instance, the horse was much more shook up than I was and acted contrite and extra-sweet afterwards.
There was one time I teased my cat by taking the side of the neighbors’ cat (with body language) in a territory dispute and my cat seemed highly offended at me which persisted for a few hours.
It can't be the case that the first sin is what made humans human, at least not for Christianity. Back up to 1:26-27 for what Christians take to be the thing that makes humans humans. They were fully human prior to eating of the tree.
But did they know some right from wrong before eating? Yes, and they are held accountable for exactly that when they eat anyway.
What exactly "knowledge of good and evil" entails, given that they are already morally accountable before that point, is an interesting topic to unpack. It must be closely related to an idolatrous belief that, not only am I in the likeness of God and so can discern right from wrong, but I am _so_ like him that I get to _decide_ what's right and wrong.
Sure, a fraction is purely driven by career / monetary gain prospects. They adopt the mantle of the AI fanatic to prime themselves for the wrestling in the mud that they hope will get them closer to their goals. I understand that and I am not particularly worried about that. Its a devil we know.
But another fraction seems to actively enjoy the absurd suggestions advanced by the AI Jesuits.
A lot of people are going to be questioning what it means to be human.
We still have the experience of being human. Isn’t that enough to savor? The experience itself?
Clinging to this idea of “I’m human, made of meat, and special” doesn’t really make for an argument as to why / how we are different.
I don’t think it’s dehumanizing at all. It’s exciting, and does nothing to take away from our experience of being human.
"Michael #3: In a certain sense you’re right. The language models now being adopted by millions of programmers don’t write working code; they only seem-to-write-working-code. They’re not, unfortunately, already doing millions of students’ homework for them; they’re only seeming-to-do-the-homework. Even if in a few years they help me and my colleagues do our research, they won’t actually be helping, but only seeming-to-help. They won’t change civilization; they’ll only seem-to-change-it."
Then he & I agree! :)
"Understanding" will turn out to be simply a compression with optimization toward survival of the compressing algorithm.
When "survival" finally resolves into a conflict between different algorithms, the winner will be that compression algorithm which compresses those that cannot compress themselves.
/s
A car with a rock on the peddle will out-run an olympic runner; but it isn't competing in a race.
To play, to compete, to do any of these things is to intend. One has to have the relevant mental states to "play" at anything.
A computer which scans a move tree and emits moves does not intend to do anything; it's just a car with a rock on the peddle. Of course it "wins", in as much as a river might, or a gust of wind.
We are interested in games because we play them; not because we "arrive at a win condition". Winning is only significant because it's play, not because of the board state.
Is it important for... chess? science? civilisation? humanity? IBM? Whom? How?
Why do we have to care at all, that a machine beat a human in anything? Why was such a big hullabaloo made back then? What is the point of beating humans with machines?
And why are people always acting so triumphant about it, as if their home team scored a goal? Whose side are you guys, celebrating the successes of big corporations' AI systems, on, anyway?
I mean, I really would like an answer to some of those questions at least because I am really confused by many of the attitudes of commenters in these discussions.
It was a big deal in a similar way a machine that can fly was a big deal. I hope this explains my perspective.
>> People have thought a computer would never be able to play at that level because chess required some special human ability.
Which people thought that? The luminaries of early AI (before DeepBlue's win) not only thought that computers would beat humans at chess eventually, they thought that it would happen earlier than it did. For example, in 1968 John McCarthy and Donald Michie made a bet with the chess grand master Peter Levy that a program that could beat Levy would exist within ten years of the bet:
In 1968, Levy and artificial intelligence (AI) pioneer John McCarthy were at a party hosted by Donald Michie. McCarthy invited Levy to play a game of chess which Levy won. McCarthy responded that 'you might be able to beat me, but within 10 years there will be a computer program that can beat you.' Levy suggested they bet on it, and Michie agreed to up the ante. Other AI experts signed on later, with the bet total reaching £1,250.[22][23][24][25]
https://en.wikipedia.org/wiki/David_Levy_(chess_player)#Comp...
I think you may be thinking about solving chess, as in mapping the entire chess game tree. I found, in wikipedia, a prediction by Claude Shannon that this wouldn't happen in the lifetime of the universe:
In 1950, Shannon calculated, based on a game tree complexity of 10120 and a computer operating at one megahertz (a big stretch at that time: the UNIVAC 1 introduced in 1951 could perform ~2000 operations per second or 2 kilohertz) that could evaluate a terminal node in 1 microsecond would take 1090 years to make its first move. Solving chess would therefore seem beyond any possible technology at that time.
But that sounds reasonable and entirely different from a computer being able to play chess at grand master level.
Besides that, I do understand your perspective, but I think it's wrong. It's not a big deal that a computer can outcompute a human. We've know that since, I believe even the 19th century, when arithmetic machines could outcompute humans every time. We know that we're no good at carrying out mechanical operations in our heads.
Computer chess was important to early AI researchers, not least among them people like Alan Turing and the aforementioned Jon McCarthy, Donald Michie and Claude Shannon. But those early AI pioneers thought that a computer that could play chess like a human would reveal to us something about the way that humans think, at least when humans play chess. The way that DeepBlue beat Kasparov, and the way that computers generally play better chess than humans, reveals nothing about how humans play chess, let alone think. What's more, computer chess playing relies on very specific algorithms entirely limited to chess and games like chess, so they are not very useful as algorithms, either.
The reason that computer chess is considered such a big deal of course is exactly because those early pioneers considered it a big deal, for the reasons I explain. And those same pioneers were not at all satisfied with the outcome of the project they helped start. Here's John McCarthy again, on the victory of DeepBlue over Kasparov:
In 1965 the Russian mathematician Alexander Kronrod said, "Chess is the Drosophila of artificial intelligence." However, computer chess has developed much as genetics might have if the geneticists had concentrated their efforts starting in 1910 on breeding racing Drosophila. We would have some science, but mainly we would have very fast fruit flies.
http://www-formal.stanford.edu/jmc/newborn/newborn.html
A withering critique. Because computer chess only made sense if it could really tell us something about human chess. That a computer can play chess better than a human tells us nothing at all, and has zero scientific value. So it's a big deal of big old nonsense.
1) for being a tremendous strawman. Nobody said it can't still be useful or world-changing, only that it's a superficial presentation of what it means to "understand" something,
2) enhancing human intelligence isn't de facto intelligence. My calculator is not intelligent, but it certainly helped "change the world, students cheat, researchers research, yada yada yada"
3) Try getting it to demonstrate object permanence. It will only "seem-to-fail-epically". There are no "things" underlying the terms it uses, and denotation is the crux of language, not syntactic structure. Chomsky himself long ago demonstrated that nailing syntax does not nail meaning: Colorless green ideas sleep furiously.
Chomsky himself claimed in the the conclusion of his op-ed that these systems are overhyped and close to worthless.
I don't know if you could make a worse argument. Generating text that resembles homework isn't the same as actually doing the homework. Students had to do their homework for the chat bot so that it can be fed into it, which lets the chat bot skip the whole doing part.
This is important, and even here on HN people are frequently ascribing to ChatGPT various qualities (“understanding”, “learning”, etc.) that misunderstand what it is doing. ChatGPT can perform a trick: it produces conversations that have the texture of human ones, skipping entirely the step of understanding the underlying ideas. Obviously a lot of people see utility in this ability, but there are limits to what this approach can do.
A helpful tip: whenever you need to decide whether to use one or the other, ask yourself whether you want to say "it is"; if so, use "it's"; otherwise, use "its".
Interestingly, ChatGPT knows whether to use "it's" or "its" at what seems like a superhuman level, but we don't even really know how it knows whether to use "a" vs. "an": https://clementneo.com/posts/2023/02/11/we-found-an-neuron
It doesn't "know" that and this is a great example of how clearly it doesn't know things even when it can produce text that if a person produced it you'd assume indicates understanding. Ask it to explain when to use "a" vs "an" and you'll get a good explanation. Give it a test, though and it'll often fail. For example:
> me: Imagine that there's an animal called a "honbukle" pronounced like ending in "buckle" and starting with "on". Tell a short story about having seen one.
> chatgpt: <blah blah blah> "It sounds like you saw a honbukle," they said. <blah blah blah>
However, I don't think you get to make it unless you can define what 'actual' actually is. Saying that the AI can't interpret but only seem-to-have-interpreted is begging the question unless you can then go on and explain what interpret actually means.
It is precisely what is most interesting about the large language models that complex seeming-tos arise from a simple process done on a large scale. If you want to say that humans have access to a qualitatively different 'actual', then you should be able to explain what that difference is, because without that deep understanding of the phenomenon, it's perfectly reasonable to ask how we know that humans do the actual rather than the seeming-to.
Your number 3 is pretty surprising, given that explaining is one of few areas I'd say it's frequently superhuman in. It can also certainly generate imaginings with more fluidity than I can too.
Until we have a really good understanding of how humans do these things, we shouldn't spend any time pronouncing on how there's a huge difference between a capability as ChatGPT has it and as a human has it.
Animals form mental representations of their environment, which are directly caused by that environment in conjunction with their prior models of it. On top of these states they combine motivational and emotional attitudes to form goals, to intend, to play, and so on.
There is far less mystery here than we suppose. Thousands of years of religion has forced this supposition that there's a "mind/body" problem, and characterising the minds of animals is such a difficult thing; it isnt.
And I take it that you're arguing that ChatGPT does not have a mental representation of its environment? Obviously, we could quibble about 'mental', but for all of its limitations and problems, ChatGPT is very little more than a condensed representation of its environment.
What about motivation and emotion? I don't think we have a clear enough definition of how these things are instantiated in humans to properly debate here. ChatGPT acts as if it has certain motivations (to be helpful, to avoid controversy), and has been coached to avoid expressions of emotion.
If you start from a view point of 'a next-word prediction engine obviously cannot have emotion, plans, or a world model', then of course you'll conclude that ChatGPT does not have these things and cannot. If you start from the view that we don't really know what mechanisms we need to instantiate these things anyway, then you're much more likely to look at the areas that ChatGPT seems to behave similarly to humans and wonder if big chunks of our brain are actually next token prediction engines, just structured by evolution to work in particular ways. This view was not new with ChatGPT of course, predictive coding has been an interesting approach to human cognition for a while now.
1. Could you ask chatgpt (jailbroken ofc because "danger") whether it could safely describe how to do this and trust the reply?
2. Would you then trust chatgpt to "superhumanely" explain how to do this?
These are statements of faith. They can't be proven or disproven by any experiment. "I think therefore I am." applies to just me. I'll never know whether you have intentionality or not. But I have faith that you do. Maybe you have faith that ChatGPT doesn't. I'm agnostic!
Chomsky's thinking is a form of vitalism: that biological matter possesses a special life force or energy that nonbiological matter doesn't. Well, vitalism eventually went out of style.
Naively, the assertions about capacities presented here contradict the observations about the outputs.
I believe your disagreements with Chomsky on politics does creep in to the discussion of AI and linguistics. Chomsky has never denied the success of ChaptGPT or google search or any of those things. The article of Norvig states this. But the core disagreement between Chomsky and his followers (such as myself) against those who prefer a statistical view of intelligence is that LLM or in fact the entire field of AI as it is right now has no real scientific basis compared to what exists in other scientific fields like physics or biology or chemistry. We don’t have a first principles of intelligence, hell we don’t even have bad theories of intelligence. Algorithms that form the basis of Google search or the statistical nature of LLM does not explain what is actually working in the real world. Just like the brain which doesn’t perform advanced calculus to calculate trajectories when it raises the hand of a body to grab a ball in the air.
Look back at the Norvig article until, we have made spectacular technological progress, basically engineering and that’s great as a tool to help society. Just don’t confuse that with actual science that is supposed to explain what is actually out there. That’s the true nature of Galileo’s telescope.
A better analogy in my opinion would be Einstein's attitude to quantum mechanics, he accepted that it had predictive power but was militantly in the camp that it was incomplete as a theory for various reason. Similar to how Chomsky thinks that deep learning is effective and interesting as an engineering concept to generate language but sheds relatively very little light on how human intelligence and language works on a deeper level.
That's incorrect on a few levels. Aaronson is talking about something else, different from what Chomsky worked on. ChatGPT can put together a statistically likely series of tokens, but ChatGPT doesn't understand the meaning of those tokens, and therefore ChatGPT has no concept of "truth." ChatGPT cannot deliberately lie or tell the truth, it has no understanding of such things. By contrast, Chomsky, for much of his career, was a linguist and he focused on the issues of how is knowledge constructed, and how do we know what truth is, and how does language express this? So Chomsky is pointing out that ChatGPT creates a fascinating illusion of a real conversation, but it isn't exactly what AI researchers were aiming for, for several decades. And it is appropriate that the New York Times would want to publish an essay that speaks clearly about what ChatGPT is and is not, because otherwise there is a risk that the general public will get an over-hyped view of what ChatGPT does.
Please see my comment below[1], this is not a fair characterization. ChatGPT is not a pure predictive LLM, it also undergoes RLHF which gives it a much more human-like character, including clearly having a concept of truth within its limits.
Even if you come up with a fancy prompt that asks it to "assume" certain things, it will still have to base that assumption on things that it already 'knows' to be true, as it already perceives them to be true.
It does encode meaning, but just knowing things doesn't give you the ability to know if they are true. A human brain in a jar that was only fed text from the internet also wouldn't be able to verify what it learned against reality.
Chomsky has the flawed conception of language. It is just one tool (though a very powerful one) that humans use when reasoning.
For example, suppose I give you a dictionary of a language you don't know. Could you learn the meaning of words in the language, by reading their lemmas in the dictionary? Note that words in a dictionary are collocated with explanations of them, not just other words, or tokens.
I'm Greek, so suppose you didn't speak Greek and I gave you a dictionary of Greek, where you could find the following:
Αβαρκαμαξας: ιμιά καρός μαύρα φούμαρα σαχλαπαράκατα μαρακαφούλες.
Can you tell what the word "Αβαρκαμαξας" means, just by looking at the dictionary entry above?
Obviously you can't (and you can't translate it in Google Translate either because it, and its explanation, are completely made up and not even real Greek). In the same way, just matching tokens with each other based on where they appear next to each other in text doesn't say anything about their meaning.
We don't know how to encode meaning. Statistical language modelling is exactly an attempt to represent language, without having to tackle the problem of representing meaning.
I mean, I can imagine opening my hands and levitating but I don't know how to do that in practice. So how would someone encode meaning by reading a dictionary in a language they don't know? In practice?
No one claimed you can do this. EnergyAmy and I claim that you will encode meaning of a terms in a corpus after long term exposure to the corpus.
Step-by-step please, and show your work. Like ChatGPT.
ChatGPT: As an AI language model, I can recognize that the word "Αβαρκαμαξας" is of Greek origin, but the dictionary entry you provided does not seem to provide a clear definition of the word. It appears to be a descriptive phrase, possibly describing the appearance or characteristics of something, but without further context, it's difficult to determine the exact meaning.
Meaning is entirely statistical! That's why languages evolve, and new words and phrases emerge and disappear over time.
I understood your specific Greek dictionary example as being something that does not encode any meaning, because the words are not real Greek. However, completely hypothetically, if "Αβαρκαμαξας" was a word that a whole bunch of Greek people started repeatedly using in the same or similar contexts, it would quite literally become a real word.
That is what I mean when I say meaning is entirely statistical. Words and phrases emerge, disappear, and change meaning over time. These changes often happen differently within different segments of populations that speak the same language. What words mean depends entirely on enough people agreeing to use the words the same way. LLMs capture the commonly used modern meanings of words extremely accurately.
edit: That's also how ChatGPT was able to understand the commonly made typo in the question I asked it. Statistically, I probably meant to say "can you tell me what the word [...] means".
>> Language models don't encode meaning. They "encode" the probability of collocations between lexical tokens. Such collocations are correlated with meaning, but they do not "encode" meaning.
So I gave an example of some words, that I suggested were collocated, to demonstrate that you can't tell what they mean just by looking at them next to each other.
That the words don't really exist is something that you wouldn't know unless I had told you, or unless you had access to a dictionary. Or, I guess, a Greek speaker. I told you that the words are not real Greek not to make a point, but to avoid being disingenuous [1].
The point of all this jumping around is that if you don't know what the words mean, you won't know what the words mean, no matter how many statistics you take on a corpus, no matter how large.
This is actually related to your point. I don't think that "language is statistical". I do agree that language, and the meaning of words and sentences, changes constantly. But, that's one more reason why language models are such a poor, well, model of language: in ten years from now, there will be many expressions used in everyday language that ChatGPT will not be able to recognise, because they would not have been in its training corpus. What kind of model of language is one that's frozen in time, and can't keep up with the vicissitudes of language as it evolves spoken by its speakers over time?
A very poor model of language, indeed.
______________
[1] You put the words in ChatGPT and it told you they mean nothing. That's not entirely true. There is a pair of words in the explanation part of my sentence: "μαύρα φούμαρα", which are real Greek words and that form a meaningful Greek phrase. So I was disingenuous, a little bit, or depending on how you see it: I said that the "explanation" is not real Greek, not all its words. In my defense, I was sure you'd put the phrase in Google Translate, or, if I told you it means nothing, to ChatGPT. So I wanted to double-blind this a bit. Apologies for leading you down the garden path. But, look at all the flowers!
In any case, ChatGPT didn't recognise these two words and the phrase they form and gave you a generic answer, riffing off the information you gave it that the sentence was a "dictionary entry". So you told it it's a dictionary entry and it said it's a dictionary entry and it doesn't know what it means. That doesn't mean that it knows that it doesn't know what it means. It just says the most likely thing, which just happens to be "it's all Greek to me", basically. It could just as well have made up some meaning, because meaning means nothing to it and all the words are the same to it, except for their location in its model.
Come to think of it, I suspect the prompt you gave ChatGPT, and particularly the bit about "it's OK if you don't know", is a strong bias that leads to it saying "I don't know", rather than pretending it knows. Can you try again, without that part?
In fact I read Chomsky's comments in regards to Serbia and Russia, and I think truth is in the eye of the beholder.
Do they? Personally I can't rule out that of LLM model was trained on all of the language a single human heard/read and produced it wouldn't be able to create next utterance that might be indistinguishable from what that human says.
This is the core problem with this schematised (and i think, pseudoscientific) computer science approach to intelligence. Output isnt intelligent.
So, for any given output, it could have been created by system A or system B, whose properties could be radically different.
It matters why, eg., we get "I hate the rain!" as output. If system-A says it because it: cares, hates, muses, imagines, prefers, intends... then that's radically different than if B does so because, "it's combining a weather API with some internet chat history".
It starts to remind me of "Yes! But it doesn't have a soul!"
Science doesn't deal with the "indistinguishable". We cannot, on earth, simply distinguish between whether we go around the sun, or the sun goes around the earth.
Does the solar system have a soul?
The world exists, and it has properties, and those are independent of how dumb apes happen to be and what we are in a position to "distinguish" or otherwise.
A system generating text is acting as-if its having its intelligence measured. Each sentence we take to be a symptom of its: having a theory of the enviornment, having something to say about it, having some intention, etc.
When I say, "I don't like what you're wearing!" that sentence itself isnt somehow "intelligent". It is only a valid measure of my caring, preferring, speaking, intending, thinking... because that is why i said it.
A shredder which happened to assemble those words is likewise not intelligent.
This is basic science: measurements arent objects; and measurements have validity criteria which is, at least, the causal properties of the system give rise to those measures.
In the case of ChatGPT no relevant properties give rise to its ouptut. Its sentences are not caused by any intelligence, and aren't valid measures of it.
There is no boiling water. Your digital thermometer is broken.
As I understand, this was one initially of the main issues with the new model proposed by Copernicus - it was not more accurate initially.
If a "shadow prompt" told chatGPT that it writes at a 3rd grade level, we wouldn't argue as much over how smart the bot is.
If it omitted the friendly/helpful/deferential assistant stuff, we'd also argue about it less. Bing's initial defensiveness and aggression made it seem even stupider than the mistakes it was making.
They're honing in on better prompts and other configuration that will make the bot seem smarter. It seems smarter to say "I can't answer that question" than to confidently say something untruthful.
But the underlying computational program (GPT trained on the internet) is the same. If we judge the program's intelligence based on its output, it isn't well defined. The same thing looks intelligent or hilariously unintelligent based on the tokens you (an intelligent person) provide it with.
Or in other words... Suppose we collect all of the system's "intelligent" outputs and disregard the rest. We throw away a lot, the majority of responses, and the resulting set looks impressively smart.
The system appears to demonstrate advanced machine intelligence when restricted to (some?) preimages of this set, even though it acts like a total idiot over other parts of the domain. And it's clear that it takes real knowledge and understanding to solve this boundary problem, so that the calculated image has an "intelligent" shape.
I agree this is true, and that it will be a breakthrough for AI, but it's entirely unclear how far away it is in the time dimension.
One of the things humans do is forget a lot of unimportant crap so we're not constantly rewriting our brains. Of course there is a the issue of how do we make sure we're training our AI how to learn multiplication and not feeding it a diet of junk food information/fake news too.
but those interactions are infinitely complex and contain an enormous amount of data
Humans take in 10 million bits[1] from their eyes every second. 10,000,000 bits/sec * 60 secs/min * 60 mins/hour * 24 hours/day * 1000 days = 108 terabytes. ChatGPT only used 570 GB of training data, so 2 orders of magnitude less data, and that's only counting the visual data.
edit: And that would be for a 3 year old, so comparing ChatGPT's intelligence to a 3 year old shows that ChatGPT comes out favourably.
[1]https://www.sciencedaily.com/releases/2006/07/060726180933.h....
I agree that AGI is quite a ways off.
It’s not a straw man. Blake was an outlier but also very much on the inside and not just some random wonk. There are plenty of people who are being led to believe that LLMs are much more than fancy statistics.
But there are many other claims and many other arguments the Chomsky piece makes: “we fear that the most popular and fashionable strain of A.I. — machine learning — will degrade our science and debase our ethics by incorporating into our technology a fundamentally flawed conception of language and knowledge”. The Aaronson article is responding to those claims, even if the response is “I disagree; more details to follow”.
Separately, it’s rather audacious to dismiss this response as ad hominem, considering the tone of what it’s responding to.
Aaronson’s ad hominems: “the intellectual godfather of an effort that failed for 60 years”, “what Chomsky and his followers are ultimately angry at is reality itself”.
Chomsky’s ad hominems: “ineradicable defects”, “lumbering statistical engine”, “stuck in a prehuman or nonhuman phase of cognitive evolution”, “the predictions of machine learning systems will always be superficial and dubious”, “pseudoscience”, “ChatGPT exhibits something like the banality of evil”, “the amorality, faux science and linguistic incompetence of these systems”.
(I enjoyed this exchange a lot, thank you.)
Here's the entire comment:
>> In this piece Chomsky, the intellectual godfather god of an effort that failed for 60 years to build machines that can converse in ordinary language, condemns the effort that succeeded.
Not only is that a personal attack, it's a completely absurd fabriaction: Chosmky did not work for 60 years on creating chatbots ("machines that can converse in ordinary language").
I see that Aaronson added a comment to his blog to stop people calling him out for that. But the comment doubles down on the absurdity and says that Chomsky is "regarded as the highest authority" by some vaguely defined faction of "anti-statistical" someone or other.
Like many people who have only recently heard about AI, Aaronson has heard of things he has only partly-digested, like for example that there are two "camps" in AI, but of course he has a very superficial understanding of what that means. The "old guard" of AI researchers, as he calls them elsewhere in his post were always polymaths with contributions in many fields.
Take Claude Shannon for example (oh, you didn't know? Shannon was at Dartmouth and one of the people who invented the name "Artificial Intelligence" for the workshop there). He invented logic gates, and information theory. There is no separation like the one that Aaronson is trying to make, it's only in his head and in the heads of people who treat the whole AI debate as a football game, and just want to be cheerleaders for the home team.
And he still hasn't removed that embarrassing attack from his article. Well I hope he goes down as the guy who thought Chomsky worked on chatbots. If that's the hill he really wants to die on...
However, my issue is that more than once, ChatGPT has suggested code to me that could likely introduce a security issue in a product had someone been dumb enough to blindly copy and paste. If this were a code snippet on a website, I'd probably drop a comment or just remember to ignore whatever I found on that website in the future.
So I get why Chomsky isn't impressed, once accuracy really matters ChatGPT falls on its face. When it comes to generating fluff like marketing copy, jokes and other things that don't require any degree of accuracy, it's amazingly good.
Nothing worked. Syntax was wrong. Midway I resorted to googling and after a short while I learned why. The library had a major overhaul at one point. They switched from using macros to using functions. ChatGPT didn't notice that and served what was mixture of old and new syntax.
I was working with JWT tokens in GoLang and the library I was using didn't validate expiration by default. I asked ChatGPT before I dug into the library code and it suggested a solution that that bypassed the built in validation of the JWT library I was using and introduced a fairly big security issue.
That was the moment I got a little nervous about folks using ChatGPT.
And people understand it!
The semantics of natural languages is over the experience of an animal and if you think of it as a “language instinct”, that instinct is a peripheral of a animal’s brain which has considerable capability for cognition (particularly in the case of mammals and birds) without language.
From that point of view natural language competence is a cherry on top of animal intelligence and you can’t simulate a language-using animal without simulating the rest of the animal.
ChatGPT does a surprisingly good job of faking linguistic competence with nothing but language so it certainly looks like a challenge to the Chomskyian point of view but I’d remind you that animals, pre-linguistic children, aphasics are all capable of cognition without language so “language is all you need” is still an incomplete position. (But boy there is that strange fact that transformers work for vision although research may be showing that is accidental?)
What does it mean for the mind?
A major part of animal behavior involves sequences. Think of tying a knot or a bird migrating, or for that matter serializing and deserializing mental structures into words. In the 1980s I read books about the brain that were lost at sea about how a neural network could model sequences and artificial neural networks have come a long way since then.
This is one of these places I hope we aren't trapping ourselves in our own thinking like we did with flying. "It's only flying if you flap your wings" could be thought of one of those trapping in 1900, when it turns out that is but one way of flying, and throwing lots of power at rotating blades is another.
We may be able to create 'intelligence' from text alone, but I do think that it will need input/output methods of divining that truth, hence simulating the IO functions of the animal, whatever that may look like in the end.
https://en.wikipedia.org/wiki/Symbol_grounding_problem
And I’d say there is more than one kind of ‘embodiment’. It sounds trivial to map the word ‘red’ to an area of the RGB color space but if you ask a robot to give you the red ball and you get the red ball didn’t you ground ‘red’?
I think the image-language models put language in confrontation with something else and ground it in some respect, you could even say HFRL training ChatGPT causes it to move towards pleasure and away from pain and that’s a very simple kind of experience. For that matter just turning text into database records that can be processed by ordinary software is a useful form of grounding, as would be asking for something and getting it in the mail.
‘Language is all you need’ is a more credible position than it was a year ago.
I guess we see how it pans out really soon.
It starts off with a complete mischaracterization of Chomsky's research program (to imply that he is a disgruntled sore loser in the race to build machine intelligence) and then continues with a bogus analogy to imply Chomsky is a kind of deluded religious fanatic.
Then it concludes with a note complaining that people don't trust the objectivity of the author's opinions now that he's employed by the entity he's defending - well, yes, of course they don't, that's the compromise you made when you decided to take Open AI's money.
Please, make it even more condescending, this is what we need in a good debate. /s
“I”
Completely as an aside, from what I've read, the Catholic astronomers did look through Galileo's scope, and confirmed his observational evidence.
They were willing to accept a model of Tycho Brahe, in which the sun goes around the earth and the planets go around the sun.
Not needing to look through the scope, to know the truth, was introduced as a hypothetical in arguments. And there were Churchmen who were using the controversy to push the Church towards what we might call a more "conservative" position today. But amongst intellectuals, the debate was over whether the evidence supported the heliocentric theory or not.
Well, Aaronson may not be the godfather, but he's definitely an influential figure in effort that failed for 45 years to develop magic machines.
He was right it that case, similar to how Chomsky s right in this case.
This is true for spreadsheets, word processors, and hydraulic presses.
It does not make them AGIs, or has any bearing to Chomsky's argument (especially since he acknowledged that explicitly).
Aaronson seems to have built a strawman out of this, for some reason.
The fact that ChatGPT cannot tell if anything is real or correct or not is not a small thing that one can hand-wave away and say, "Well, maybe tomorrow." The inability to discern truth is built into the LLM method. This is not true of people. Certainly not intelligent ones.
ChatGPT cannot even correctly perform math or logic, and those are fundamental to intelligence and intellectual development. Hilariously, we already have many programming languages which can do those things, so that's not much of an advance to rely on software that cannot.
Aaronson's premise, that Noam Chomsky has nothing more than "sour grapes" to offer, is really puerile and uninformed. See Chomsky's original article [1] for yourself and then decide who is on the hype bandwagon and who is not.
[0] https://medium.com/gitconnected/behind-the-curtain-understan...
[1] https://web.archive.org/web/20230308104809/https://www.nytim...
To be fair, ChatGPT is not a pure predictive LLM. It also undergoes RLHF, a reinfocement learning step which is much more similar to human learning, and allows it to develop a more human-like character, and seems to give it a pretty good grasp on truth(telling), ethics and context, certainly compared to the behavior of a raw predictive LLM with no prompt engineering.
I recommend read this blog post for a more nuanced understanding of LLMs: https://astralcodexten.substack.com/p/janus-simulators
I don't disagree with your premise. But this applies equally to human intelligence. There's no way to ever tell if the reality you experience is real or not. This is a fundamental fact of existence that we hand wave away so that we can continue to interact with and learn about the world. My favorite thought experiment on this is Boltzmann Brains.
The fact that it can instantly reply with a broad consensus of the ideas it has been fed is precisely what I need from it. When I ask it to write a function that performs a binary search in Rust with a given set of constraints, I'm not looking into original research or "truth", I want precisely the boring boilerplate I'm too lazy to write. For that use-case, it's invaluable and awe inspiring.
It's limits are laughable too, try as I might I could not make it count the numbers in a string of '9's that fit in a 128bit integer. It hallucinated fictional names of tools that I should use, such as "cobol_to_rust".
It seems clear to me that both camps have profound insights and LLM are both invaluable and wholly insufficient to reach any form of substantial "Artificial Intelligence".
Well...ChatGPT has been struggling. It generally struggles with anything subtle, no matter how simple. If it can't regurgitate based on what's slurped up from the internet it struggles.
Do note, my question is simply meant to assume i am using a-w as my digits and "word" is a number formed via a positional number system. It is something a human with requisite knowledge will veer towards, ChatGPT doesn't.
To your point, it is indeed an awe-inspiring tech for the thing it is good at, ie replacing the current crop of search engines.
For your intended use-case (as explained in your comment) shouldn't you be phrasing the second part of this specification as "treat the characters in "word" as if they are digits"?
The way you worded it introduces some ambiguity. The ambiguity can be resolved by a process of elimination - I had to think of what could possibly happen if I "assumed the characters in 'word' are digits" and realise that it doesn't make sense, so it must mean you want to use the characters as digits.
ChatGPT and LLMs in general really can't be expected to deal with that kind of ambiguity, not least because it's not the entire story (you also need another little logical jump, from "treat characters as digits" to "treat characters as denoting their positional values in the alphabet" or some such).
I'm saying, you're asking ChatGPT to take many reasoning steps, when everyone is having a big debate about whether ChatGPT and LLMs can reason _at all_. To make any heads or tails of ChatGPT's abilities, one needs to start with simple examples, and then get to the more complex stuff.
On the other hand, I expect that if you clarified your question more, or perhaps stated it just the right way, there's a good chance you'd get a correct answer. After all, the idea of counting with characters is very common and it's hard to believe ChatGPT hasn't seen a few thousands of examples of it during training.
But not surprising since Scott is now employed at OpenAI.
It is deeply immoral and it will rekindle the disturbing image of evil technologists and mad, power hungry scientists at a time where we need as much trust and confidence in the use of technology and science to absolve us of problems we ourselves have created.
Woah there. When did Chomsky ever take an interest in building machines to do anything? What is he even referring to here?
From the get-go, this piece sounds like an ad-hominem slam.
From Chomsky's essay, 2 important lines:
"Whereas humans are limited in the kinds of explanations we can rationally conjecture, machine learning systems can learn both that the earth is flat and that the earth is round. They trade merely in probabilities that change over time.
For this reason, the predictions of machine learning systems will always be superficial and dubious."
But since LLMs cannot provide explanations for their beliefs, humans will never be able to rely on LLMs because the way we actually communicate is through explanations, not probabilities.
Chomsky's conclusion: this is a good predictor, but not a human.
Where Chomsky is actually wrong is he mixes up how good are you at prediction vs. how well can you convince a human of the prediction vs. how intelligent you are. We humans use a combination of accuracy + convincingness as a marker of intelligence.
An AI that was just as accurate but 0 ability to convince a human because 0 ability to produce explanations could still be intelligent. This AI would seem like an alien to us, but an alien that could uncannily beat us at any challenge that requires an understanding of the natural world. In fact, we might never truly understand its internal explanations of the world, but still acknowledge it has them and is good at building new ones internally.
Imagine being in a room with a foreign language speaker who beats you at chess. Clearly they have a mental model that works, even if they cannot explain it to you.
This ideas do not contradict each other. Chomsky talks that LLMs is not sufficient to create AGI, Aaronson says that LLMs is essential to create AGI. A cause can be essential but not sufficient, nothing wrong with it.
(And Chomsky's take on ChatGPT is much better than his recent takes about the Russian invasion of Ukraine…)
Yes, his article does just about clear the incredibly low bar that you have set for him.
Which explains why he is now defending the Closed AI snake oil over his quantum computing research and is in fact defending his OpenAI equity like the rest of everyone who recently rushed into the OpenAI hype train scam.
His weak response to Chomsky tells us that he has no answer to LLMs like ChatGPTs lack of transparent explainability about the sentences it regurgitates, which is the core reason why LLMs are untrustworthy to which Chomsky repeatedly brings up.
But of course he chose not to disclose that in his article so here is the source for his announcement that he joined. [0]
One thing I would've like Chomsky to engage more in is comparing how people create new "output" from previous "input" vs. how AIs like ChatGPT do it. He talks about "fake thinking" and "real thinking", but doesn't really go into the "hows" of it (ironically, kinda like how ChatGPT would).
I _do_ agree, however, that we're nowhere near AGI and this doesn't bring us closer, but I don't know what would, either.
On the topic of Aaronson's blog post, I feel like he didn't understand what Chomsky wrote about or intentionally mischaracterised it, then gave a super childish response. It's very embarrassing.
At least he leads with the fact he doesn’t understand Chomsky’s research program at all.
LOL, this seems completely wrong to me. I think Chomsky's idea is that humans have some kind of 'grammar engine' that just needs configuring from 'miniscule data' aka many years of listening and practice speaking.
No doubt humans have some optimized areas for language, but I doubt it's anything so formal.
"Yes, they’re wrong, and yes, despite being wrong they’re self-certain, hostile, and smug, and yes I can see this, and yes it angers me"
Norvig's essay is less hateful and goes deeper.
The Norvig - Chomsky Debate
ChatGPT: "I am not conscious, self-aware, or capable of having personal perspectives. I can provide information and analysis based on the data I have been trained on, but I don't have the ability to form personal opinions or beliefs."
ChatGPT itself will tell you the limits of its capability.
Have you had a real conversation with ChatGPT? As in a "trying to extract factual information from another entity" conversation? I'll answer for you: no, because ChatGPT is generative (it's in the name) of apparently valid syntax with consistent meaning, but it's not extractive in nature and does not wield denotation behind-the-scenes. "Converse" means a lot more than "vomit words in a proper order at each other". By that alone, it has not "succeeded" in conversing, just arranging terms convincingly (using Chomsky's theories to do so.)
I encourage anybody who believes otherwise to read this paper before continuing to posit that ChatGPT is doing anything more: https://www.philosophy-index.com/russell/on-denoting/Russell...
Chomsky never had anything to do with any sort of effort "to build machines that can converse in ordinary language". I have no idea where that came from in Aaronson's article. Chomsky was a great influence to cognitive science, and artificial intelligence, in particular in its early years, but he has nothing to do with chatbots. At. all. That's just plain nuts, to open an article against "Chomskyism" (bad form, that title) with such a complete fabrication.
Moreover, Chomsky's influence on AI did very much lead to systems that could "converse in ordinary language" - but with which Chomsky had nothing to do, personally, or as a researcher. The "influence" I mention refers to Chomsky's work on formal grammars, and how they have been used for a long time in NLP to model language.
For a couple historically notable examples:
ELIZA: Joseph Wizenbaum's chatbot pretending to be a Rogerian psychologist. I think at this point in the conversation, everyone has heard about ELIZA and how it drew people in and made them ascribe human qualities to it, just like ChatGPT and friends.
https://en.wikipedia.org/wiki/ELIZA
SHRDLU: the doctoral thesis of Terry Winograd, written in 1968-70. SHRDLU was a natural language interface to a virtual robotic arm manipulating coloured bricks in a "brick world". It was based on a natural language grammar created by Winograd, and could support intricate instructions that the system then executed with precision. The wikipedia article on SHRDLU has an example, from which I quote:
Person: Is there a large block behind a pyramid?
Computer: YES, THREE OF THEM: A LARGE RED ONE, A LARGE GREEN CUBE, AND THE BLUE ONE.
Person: Put a small one onto the green cube which supports a pyramid.
Computer: OK.
Person: Put the littlest pyramid on top of it.
Computer: OK.
Person: Does the shortest thing the tallest pyramid's support supports support anything green?
Computer: YES, THE GREEN PYRAMID.
Person: What color is it?
Computer: BY "IT", I ASSUME YOU MEAN THE SHORTEST THING THE TALLEST PYRAMID'S SUPPORT SUPPORTS.
Computer: RED
Person: How many things are on top of green cubes?
Computer: I'M NOT SURE WHAT YOU MEAN BY "ON TOP OF "IN THE PHRASE "ON TOP OF GREEN CUBES " .
DO YOU MEAN:
1 - DIRECTLY ON THE SURFACE
2 - ANWHERE ON TOP OF ?
Person: 2
Computer: THREE OF THEM.
Now, if you've seen discussions with ChatGPT and earlier large language models you'll know that the above is beyond the capabilities of modern systems; including ones trained specifically to manipulate robotic arms etc. ChatGPT, faced with instructions like the above, will soon start to hallucinate objects that don't exist, then hallucinate moving them, and make a pig's meal of the entire state of the blocks world. SHRDLU, confined as it was in its virtual, toy world, could still follow the instructions of its user with absolute precision. We still have nothing that can repeat this feat. Why? Because "Chomskyism" was abandoned, and everyone turned to statistical NLP, after the AI winter of the '90s crushed AI research funding, that's why.
> [...] the famous dialogue with SHRDLU where you could pick up a block, and so on, I very carefully worked through, line by line. If you sat down in front of it, and asked it a question that wasn't in the dialogue, there was some probability it would answer it. I mean, if it was reasonably close to one of the questions that was there in form and in content, it would probably get it. But there was no attempt to get it to the point where you could actually hand it to somebody and they could use it to move blocks around. And there was no pressure for that whatsoever. Pressure was for something you could demo. Take a recent example, Negroponte's Media Lab, where instead of "perish or publish" it's "demo or die." I think that's a problem. I think AI suffered from that a lot, because it led to "Potemkin villages", things which - for the things they actually did in the demo looked good, but when you looked behind that there wasn't enough structure to make it really work more generally.
Unlike "Chomskyian efforts," statisical NLP actually allows the machine to interact with natural language.
https://conservancy.umn.edu/bitstream/handle/11299/107717/oh...
In that passage you quoted, Winograd is saying that SHRDLU was essentially an academic demo, something created only for the purpose of passing a thesis examination, even as a tech demo is something created only to show off at a showroom. That's well known.
But SHRDLU, academic demo that it was, was a program hand-crafted by a single PhD student in a few months. It didn't have behind it a team of 30+ people, or a budget of hundreds of thousands of dollars, like GPT-3 -and who knows about ChatGPT!
And yet it still worked better than ChatGPT has ever worked, even though just for the purpose of moving virtual blocks around a virtual world.
>> Unlike "Chomskyian efforts," statisical NLP actually allows the machine to interact with natural language.
Sure. And do what? Go off the rails like a crazy train, most of the time.
> SHRDLU, confined as it was in its virtual, toy world, could still follow the instructions of its user with absolute precision.
It seems clear that, in light of the quote from Winograd, that simply isn't true.
> But SHRDLU, academic demo that it was, was a program hand-crafted by a single PhD student in a few months. It didn't have behind it a team of 30+ people, or a budget of hundreds of thousands of dollars, like GPT-3 -and who knows about ChatGPT!
Do you have some idea in mind how we might scale up from SHRDLU's approach? Does Winograd? It seems pretty clear to me that no one does; it was a one-off toy that didn't solve any fundamental problems.
> And yet it still worked better than ChatGPT has ever worked, even though just for the purpose of moving virtual blocks around a virtual world.
There are plenty of questions remaining concerning LLMs. We don't know what they will be used for, where they will succeed and fail. But they are absolutely a huge step forward from earlier attempts and it's utterly bizarre to deny that.
In any case, I'm no clairvoyant and I can't see the future. What I see right now is a huge mess and a gigantic marketing bubble. So I'll just sit tightly and wait to see what will happen in a few years.
I do have to note though that, so far, all the large hype models that have come out of OpenAI's research -GPT-2, GPT-3, now ChatGPT- have not really been useful in anything but parlor tricks: generating silly poems and malfunctioning code, and the like. They're great entertainment, but not very useful all.
As indeed have been earlier overhyped results. I wonder if anyone remembers the Neural Translation Machine in Google Translate and how it invented its own "interlingua"?
>> It seems clear that, in light of the quote from Winograd, that simply isn't true.
You're right, my comment exaggerates the achievement in SHRDLU. So let's say, not "absolute precision" but "unprecedented precision" and precision not since achieved again. Certainly not by any statistical language model.
Or, I mean, I don't know. People have controlled robots with code created by GPT-3 etc (I've been in a couple of conferences where people from MIT were showing off some research to that point). Can you show me an example of an interaction like the one in Winograd's thesis? Personally, I haven't seen any so far.
Edit: I'm re-reading my comments to make sure the tone is civil. Please let me know if you don't think it's OK.
Whether LLMs are more or less revolutionary than google search remains to be seen. I predict they'll be somewhere in that ballpark.
> Or, I mean, I don't know. People have controlled robots with code created by GPT-3 etc (I've been in a couple of conferences where people from MIT were showing off some research to that point). Can you show me an example of an interaction like the one in Winograd's thesis? Personally, I haven't seen any so far.
I don't know of anything. Will be interesting to see how far they get with that.
I don't mind your tone. I think it's civil but, frankly, I enjoy a little incivility now and then.
I'll do my best :P
The False Promise of ChatGPT - https://news.ycombinator.com/item?id=35067619 - March 2023 (20 comments)
Chomsky has been extremely derisive towards anyone with whom he disagrees. You can see that in the NYT piece but it's been a decades-long pattern where dissenters are automatically morons. That's not a manner befitting a scientist.
I agree that "Chomskyism" has been a "false promise" in some sense. It's not clear to me what Chomsky's linguistics have actually accomplished. Perhaps they gave us a new way to think about language. In that sense, they were a philosophical achievement. But we didn't build anything with them, and that's ultimately the test of science. Meanwhile neural nets got us Google translate and ChatGPT (so far). Now, I don't think philosophy is useless, but Chomsky going after machine learning is a bit like "Jesuit astronomers declining to look through Galileo’s telescope" in that Chomsky is a bit like a philosopher immersed in metaphysics who is dismissive towards empirical scientists who accomplished something much more concrete.
Then there's the question of whether ChatGPT is conscious and whether it could lead to AGI. I agree with Chomsky that it's not actually intelligent in some sense. But I'm not sure how much that matters. If you can build a Q&A machine, whether it's intelligent or conscious is an interesting philsophical question but ultimately beside the point. Anyway, that debate has been raging for 50 years (see Searle) and isn't especially interesting at this point. As to whether LLMs can scale into AGIs, I have no idea, that depends on how we define AGI. To me, one of the lessons of ChatGPT is that we don't need consciousness to build useful AIs.
Chomsky is a humanist. I believe his critique of behaviorialism was based on humanism: he wanted to place humans in a separate category (as philosophers often do). I think his criticism of ChatGPT is ultimately similar. But the worry is misplaced. ChatGPT is in some sense more human than we are. It isn't some foreign, disembodied intelligence. It's based entirely on text we, as a species, generated and in that sense it's a culmination of human potential.
Moreover, it doesn't matter that LLMs work differently from the human brain. Per Larry Wall, TIMTOWTDI ("There is more than one way to do it").
Most animals don't acquire language, and the non-humans who are the best candidates for it all appear to have developed it independently; specifically in humans the potential is at most around 10mil years ago since we don't find it in other closely-related primates. Actual language is ~3 orders of magnitude younger, and during most of the time since then humans didn't have such high-density ways to ingest it. As I mentioned elsewhere in the comments, GPT-3 consists of around a million modern-human-years of linguistic intake. It seems to have the strong lead here.
> it doesn't matter that LLMs work differently from the human brain.
It does if your goal is to learn how human language acquisition works, rather than grind through another trillion in VC cash via your mechanical parrots.
>On the contrary, the human mind is a surprisingly efficient and even elegant system that operates with small amounts of information;
True, but then
> it seeks not to infer brute correlations among data points but to create explanations.
This is such an odd dichotomy, and feels really wrong to me. Explanations are to me just further correlations.
> a young child acquiring a language is developing — unconsciously, automatically and speedily from minuscule data — a grammar, a stupendously sophisticated system of logical principles and parameters.
Chomsky apparently has children, but clearly may have forgotten what it's like for them to learn language! It's years of trying to communicate, constantly, failing, and learning.
Now, go ask ChatGPT to Write an ode to <obscure topic> in iambic pentameter and I bet that it will be as astonishing, creative and fluent as a native speaker.
You are correct that children learning language are constantly failing in some manner. There has been a lot of study of the language acquisition of children. However, the way in which children fail and do not fail is specific. Some things children will fail at a number of times until they learn. On the other hand some rules of grammar, logical principles and parameters children seem to pick up immediately, and in Chomsky's view are even innate.
So you can divide theoretically possible failures into two categories - ones theoretically possible that are not made, and ones that are theoretically possible and are made. The first category is what he is talking about.
Also - there are different languages with different rules, and people learning a language have to learn the rules of English which are different from Spanish which are different than Japanese. Although in Chomsky's view, these languages are all so similar (compared to say the recursively enumerable language a Turing machine uses), they are virtually identical from a theoretical point of view.
Also - children make grammar errors such as first saying "I go to the store" (correct) and then "I goed to the store" (incorrect). These errors were not made by data they acquired! The errors children make are showing they are beginning to understand grammar rules, it's a sign they are going off a series of rules and not just regurgitating data they have acquired.
Normal healthy language exposure for children is around 100KB/day, we can round up to 40MB/year.
No, I have not spent ~1000000 years around a baby.
First we don't really know how the mind works and therefore we really have no idea if the language models are path toward AGI or not.
He doesn't have an idea how a child obtains a language. You really need a lot of data to acquire the grammar, no, minuscule data is not enough.
I have observed it personally with my multilingual children, their mother tongue, to what they have the least exposure and what is the most complex, is not still not fluent for them and yes, they make obvious grammar mistakes that demonstrates that they have not fully internalized it.
His claims are simply false, easily refuted by empirical data.
I would call it fallacy of English. English is stupid simple language. Most languages are not.
I honestly don't know what he did or did not consider considering that he has made very bad judgement about two aggressor states: Serbia and Russia based on his anti-US stance. It clearly demonstrates that he is able to not think clearly.
Explaining away genocide is not a political view.
I read Henry Kissinger's China book. He seemed to be not purely happy about US Cambodia politics. (but reasons)
"Allegations of United States support for the Khmer Rouge"
https://en.wikipedia.org/wiki/Allegations_of_United_States_s...