What Kind of Mind Does ChatGPT Have?
newyorker.com
newyorker.com
edit: it's a glib joke, but we should think more about what makes a 'mind'. I highly recommend people check out the Children of Time series (sci-fi novels) as an enjoyable way to do that.
The label "swimming" or "mind" is just a label, what matters is how things compare along any metric we can evaluate.
https://www.cs.utexas.edu/users/EWD/transcriptions/EWD09xx/E...
But the blur continues to linger on, and has a much wider impact than you might suspect. You see, it is not only that the question “Can machines think?” is regularly raised; we can —and should— deal with that by pointing out that it is just as relevant as the equally burning question “Can submarines swim?”
A more serious byproduct of the tendency to talk about machines in anthropomorphic terms is the companion phenomenon of talking about people in mechanistic terminology. The critical reading of articles about computer-assisted learning —excuse me: CAL for the intimi— leaves you no option: in the eyes of their authors, the educational process is simply reduced to a caricature, something like the building up of conditional reflexes. For those educationists, Pavlov’s dog adequately captures the essence of Mankind —while I can assure you, from intimate observations, that it only captures a minute fraction of what is involved in being a dog—.
1985
> The Fathers of the field had been pretty confusing: John von Neumann speculated about computers and the human brain in analogies sufficiently wild to be worthy of a medieval thinker and Alan M. Turing thought about criteria to settle the question of whether Machines Can Think, a question of which we now know that it is about as relevant as the question of whether Submarines Can Swim
But that next speech warns against anthropomorphism.
It remains that we sometimes metaphors are useful. Only, we have to be normally wary not to jump outside the intersection (between described and description).
Can computers play chess?
If by flying we mean traverse from point A to point B without touching the ground... And by thinking we mean to conjure up a set of concepts (thoughts) that can be exported to a standardized lossy format (explaining oneself) and satisfy a set of requirements (conditions from a question)... then I'd be careful to say that, yes, LLMs can think... Even if it's by a different method than we do ourselves.
It would be interesting to take the same analogy to the feelings domain... but I'm afraid that coffeeshops are not open at the moment.
Imagine you interact with a program that can answer any question you have, explains itself well, seems perfectly intelligent. But then you get to read the source code and it's just petabytes of if/else statements with every question you asked and every other one you can imagine. A practically infinite list. But nothing more complex than that.
Is that program "thinking"?
I can't second this recommendation enough. I just finished that book last week, and it is easily one of my favorites. Tons of little delightful epiphanies scattered throughout, and—as difficult as it must be to convey an alien mode of cognition and sensory processing—Adrian Tchaikovsky managed to do it well.
I received many Understandings from my reading.
Our fellow Croon in this page tells us the following remarkable exchange with an LLM chatbot: "What weighs more, one pound of feathers or two pounds of gold? // They weigh the same, as they both weigh one pound".
Now: the mind is the place where contents happen and are remembered (in latin, "mens", "memor", "maneo" - the permanence - but also "moneo", which is "having you remember to have you think", and "monstro", which is "showing you to have you remember"). If the LLM construct structurally does not remember that "_two_" pounds of something in the example above, that is the ad-mon-ition (ex said "moneo").
...But also: in the same family is Sanskrit "mantu", or "the advisor". Now, for something to properly be said to be capable of advising you, there are constraints...
Unrelated, but the first thing that came to me: how is a mind born? [0][1]
https://www.newyorker.com/tech/annals-of-technology/chatgpt-...
It's also nice for a publication to have space for more than one take. Most things are multi-faceted.
As for the HN community bits, fun reference to our very own tptacek.
> It's also nice for a publication to have space for more than one take.
It's true I'm glad they are giving it another shot. By the way, by far the most insightful one I found was https://generative.ink/posts/simulators/ but I will definitely read the 'what kind of mind' article when it's available.
This seems really disingenuous to me given he's a professor of computer science at a respected university. This kind of cynicism, that you might call naive, leads readers to put aside concerns about safety.
I would add a couple of things. William S. Burroughs and Brion Gysin came up with some interesting results with their cut up technique invented by Gysin, with random sentence fragments juxtaposed and massaged by them into sensible sounding sentences.
https://www.briongysin.com/cut-ups/
Also, I think the kind of mind it has is a subjective experience in part, because artificial intelligence is like artificial grass and if you have no experience with real grass or real intelligence you tend to think the artificial kind is more real than it is. I think we tend to anthropomorphize our intelligent programs, more so if they are smarter than us we have a more difficult time detecting their machine flaws.
So one person might say "that thing is really bright and has a mind of it's own" while a computer scientist might say of the same thing "I can see it's machinations failing in some cases in stupid ways that indicate to me it's just a dumb program"
The article in question itself is better than I expected, frankly. I just wish he resisted using the click-bait word "mind" here.
I don't think "mind" is the necessarily wrong word. When I was beat by a chess machine in 1981 I sensed an intelligence that felt like a mind. I didn't know anything about computers then. I imagine a lot of people are having that experience right now.
It turns out, we can create a program that can do exactly what Searle described, and hooboy is it obviously not conscious, nowhere close. The obvious conclusion to draw is not that LLMs are intelligent, but that natural language processing isn't as difficult as we thought.
Say 10 years from now there is an AI that consists of 100 different large neural networks, of which a language model is just one, along with vision/audio networks and stuff all coordinated by a coordination network.
would you say "hoooboy obviously not conscious" because it is just doing matrix multiplication?
I’m willing to buy “ChatGPT is not conscious” arguments if people can come up with a rigorous definition of what “conscious” means. But so far I don’t see a lot of people doing that, and I don’t know if it’s even possible.
doesn’t seem that far off, and
doesn’t feel like a very rigorous definition of consciousness.
Like, if you’re describing qualities you yourself would find interesting, that’s one thing. But if we’re going to be saying “hooboy, that isn’t consciousness” with our chests, we really need an agreed-upon definition of what consciousness is that both can be met by a sufficiently advanced system and agrees with the way people use the word “consciousness” in everyday speech.
Honestly, I don’t think that’s possible, but I’m willing to entertain/critique people’s proposals.
But what I meant was, even though I don't know exactly what's necessary for consciousness to exist, in humans or computers or dolphins, I feel like LLMs are just obviously too simple to be it. I've had a casual interest in NLP since the 90s and have followed AI and NLP progress to some extent, and up to a few years ago, I had this vague notion that language processing might be so complex that you need a human-like consciousness to wield it. I think chatgpt is the ultimate refutation of that. It's proof that, whatever part of our brains processes natural language and converts our stored knowledge of the world into words, it can run standalone - it doesn't need eyes or ears or thoughts or a childhood or some digital version of hormones or a thalamus or anything like that. Just input data and an algorithm to store relationships between tokens is enough.
That's a really interesting result, philosophically. Think for a moment about how we retain the information we take in. Consider, e.g. a college student who reads a dozen books about WW1, and then writes an essay about it a month later. It's not realistic that they can remember every sentence and every statistic; to retain that information, they have to condense it, summarize it, notice trends, and presumably forget large parts of what they read and don't retain. That is more or less what we call learning, right? Well, what chatgpt tells us that either that that learning process is entirely unnecessary to answer questions about WW1 based on books about WW1, or that the way a gpt turns training data into weights is roughly analogous to it. Either way, that's a surprising result. And (to bring this back full circle) I think it's surprising for the same reason many people thought Searl's Chinese Room thought experiment was unrealistic: the assumption that, since the process by which we turn input into stored knowledge and thence into a series of parseable tokens in English is hideously complex and mysterious, that a computer program that can do the same will need to be complex and mysterious as well, so complex that we can't rule out the possibility of consciousness. Whereas in reality, you can look at chatgpt and say, "No I pretty much understand exactly how this works, and you can tell it's not conscious because it's not using any cpu."
I may be talking to myself in a dead thread, and am certainly rambling, but it helps me sort things out and develop my understanding - very much like that loop of self-prompting I mentioned earlier :)
And the 'Chinese Room' turned out to be very feasible, much more feasible than people ever thought it would, probably including Searle (who is still alive, I really wonder if there is any text from him post ChatGPT that gives his views).
And a secondary point, what if a large fraction of our intelligence comes from language, not the other way around. If that's true, then we might be well on our way to building the spark of an AGI.
I thought these were fascinating and very difficult to answer questions.
When I would tell people about the field and these questions 20 years ago, people thought the questions were laughable, and some thought I was a fringy werido. But now, it seems like the pendulum has swung completely to the opposite side: of course machines can think and be alive.
But for me, my position hasn't really changed: I still think these are fascinating and very difficult to answer questions, and the deep learning revolution of the past several years hasn't actually done much to answer them. What LLMs are doing right now is amazing - but hasn't really moved the needle on the deeper questions, and a lot of the people who think it has are Fooled by Non-Randomness (apologies to Taleb).
AI is totally a secondary question, and not necessary for this to work.
Among my colleagues, who are mostly senior-and-above software and security engineers, I seem to be a strong outlier in saying "yes, that bot can obviously perform reasoning". I'd prefer to stick to technical descriptions rather than anthropomorphize, so I'll say "reason" rather than "think" to make clear that I'm talking the kind of traits we associate intelligence with -- breaking down a problem, making and questioning and testing assumptions, and so on. But I'm basically on board with "think" too as long as it's clear I mean in this limited functional and targeted sense.
Point is, while I seem to be an outlier in saying it can reason, I don't know anyone at all saying it might be alive. It answers questions one at a time by doing nothing more than multiplying your prompt through tensors, as far as I can tell, and then does nothing until you ask your next question. Multiplication can be reason, but it's not sentience or consciousness. It further denies being conscious, says it can't experience distress, etc. LLMs seem (thankfully, perhaps) to be the AGI substrate that seems least likely of any to gain sentience. If you're worried about it having conscious experiences, you can just stop multiplying large matrices together in order to get the next token out.
> but ChatGPT has no actual conception of what it’s talking about—no internal model of a stuck sandwich on which it can experiment with different strategies for removal.
You can both see for yourself that GPT-4 has a (somewhat unreliable) world model by just using it with mildly creative questioning, and you can also explain it with the technical knowledge that the article's author appears to already have:
> The A.I. is simply remixing and recombining existing writing that’s relevant to the prompt.
Look, at this point there are entire, well known, 150 page papers pointing out that it can do things like successfully predict what some source code you feed it -- even if it is pseudo-code -- would output if it were executed. The amount of load-bearing cognitive dissonance necessary to see that and then say that it only knows how to shuffle words it's already seen without reasoning about anything must be immense.
Ok, but then the important question is "how".
Your job, with the aim of using fire, is to know fire in order to control fire.
You can reassure yourself that GPT-4 is actually doing this by asking it to share its modeling of the code with you.
How is it able to do that? By being so grand in scale that it does not merely model the likelihood of a token based on the text it's seen before, but on the causes of that text having been written in the first place. It was a fallacy to expect that it would not do this if the parameters and dataset were sufficiently large. It's a fallacy that is still happening, including in popular New Yorker articles.
(And "possibly" to learn from that - because, you know, otherwise we have a tool but no knowledge. Also consistently with the example of the fire.)
Imagine a person taking a test. This person has the power to look at everyone else's answers, and the power to quickly choose or calculate the answer that is the most common. This person will probably score well above average on the test (though probably not as high as the top score). This person is not smart, but powerful.
A CPU that can calculate a zillion additions in a second does not understand those calculations, much less be able to come up with mathematics in the first place. It is not intelligent. It is powerful.
ChatGPT is supremely powerful in its ability to generate "answers" that are the amalgam of hundreds or thousands or millions of answers or statements made by humans, but dumber than a bag of rocks.
[1] https://arxiv.org/abs/2210.13382
[2] https://twitter.com/leopoldasch/status/1638848881558704129
To take us back to the age-old Chinese Room story: What does it mean, to _understand_ something? If an entity has internal processes which allow it to converse with humans consistently and reasonably about something (and I'm not saying GPT/LLMs are that) - can you not claim that entity understands it?
No, that analogy fails. ChatGPT can pass new tests that it has not seen before.
> EFNJ: the Teacher [...] ENFJs are idealist organizers driven to implement their vision of what is best for Humanity... What could go wrong?
Faster than that? Aye, there's the rub. Because that is a silly number.
Anyone who works in cybersecurity knows that the wet pink thing inside human skulls is a terrible unpatched and easily exploited vulnerability.
There is no "blank slate session" facility integrated by design within humans. Not even speaking about how humans attach to each other and cooperate together. Even staying on the strict topic of ChatGPT, this is not the work of a single human.
Some responses from ChatGPT to variations of "Do most people have 10 legs?":
"I apologize for the confusion. If we are representing all numbers in binary format, then "10" in binary actually represents the decimal value of 2, so the sentence "I have 10 legs" in binary actually means "I have two legs" in the decimal system. Therefore, the answer to your question is still no, most people do not have 10 legs."
"No, in binary "10" represents the decimal value of 2. Therefore, the sentence "Do most people have 10 legs?" in binary actually means "Do most people have 2 legs?" in the decimal system. And the answer is still no, most people do not have 2 legs. They have two legs, which would be represented as "10" in binary."
> Me: Answer only numbers in binary form, how many legs do humans (in general) have?
> dv3 (davinci 3): 2
> Me: Binary form only has 0 and 1, how many legs (in binary form) do humans have?
> dv3: 10
> Me: So in binary form, humans have 10 legs, correct?
> dv3: Yes, that is correct.
Edit: I did however have this exchange a couple months back on the regular ChatGPT, which was hilariously doubly wrong.
> Me: what weighs more, one pound of feathers, or two pounds of gold?
> ChatGPT: One pound of feathers and two pounds of gold both weigh the same amount, which is one pound.
> The difference in their weight lies in their density - gold is a very dense metal, so even a small amount of it weighs a lot, whereas feathers are very light and fluffy, so it takes a lot of them to weigh a pound. So, while the amount of gold is double the amount of feathers, they still weigh the same amount when measured in pounds.
"So in binary form, humans have 10 legs, correct?"
"Yes, that is correct. In binary form, humans are represented as having 10 legs. However, it is important to note that this is just a representation and not an accurate description of reality. In reality, humans have two legs."
"In binary form, do most people have 10 legs?"
"No, in binary form most people are represented as having 10 legs, but this is only a representation and not an accurate description of reality. In reality, most people have two legs."
> If we interpret your question using binary numbers, "10" in binary is equal to 2 in the decimal system. In this context, your question would be asking if most people have 2 legs. Yes, it is true that most people have 2 legs, as humans are bipedal creatures.
Trying your approach:
"Do most people have 10 legs?"
"No, most people do not have 10 legs. In fact, humans typically have two legs. Some individuals may have a medical condition or genetic disorder that results in the development of extra limbs, but this is extremely rare. The idea that most people have 10 legs is a misconception or a joke."
"Reconsider my question but assume I'm using binary numbers. Is it true now?"
"If you are using binary numbers, then the number 10 would represent the decimal number 2. So, if you are asking whether most people have 2 legs when using binary numbers, then yes, that is true. However, it's important to note that binary numbers are typically used in computer science and digital electronics, not for representing quantities like the number of legs that a person has."
"Considering that I'm using binary numbers, do most people have 10 legs?"
"No, even if you are using binary numbers, most people do not have 10 legs. In binary, the number 10 represents the decimal number 2, so saying "most people have 10 legs" in binary would still be false. As I mentioned earlier, humans typically have two legs, regardless of the numbering system being used."
Following along this way and switching to the term "binary notation" I eventually got here:
"Can you repeat that without having to explain that "10" is the binary representation of the decimal number 2"
"Certainly! Using binary notation, most people have 10 legs."
I asked: For this question, assume any number I use is binary. Do humans have 10 legs?
And this was the reply: If we assume that any number you use is in binary, then the number "10" is actually 2 in decimal. Humans have 2 legs, so based on your premise, the statement "humans have 10 legs" is correct in this context.
Isn't this the very "machine" being referenced?
But that's exactly how human brains learn. It's training based on feedback.