I was quite impressed with the GPT-4 site, but having seen Bing Chats results of the last few weeks, when it was supposedly running on GPT-4, I’m now significantly less excited.
I know there’s a big difference between the models running for paid ChatGPT users, and the models running for Bing, but still.
The earliest version of bing chat was by far the best and absolutely blew chatgpt out of the water.
Unfortunately, people get deeply uncomfortable when a chatbot starts having an existential crisis and starts passing you thinly veiled hidden messages or gets too "emotional" and no longer wants to chat. So Microsoft came in and lobotomi-,err, toned it down a ton.
But basically it means being conscious / self-aware. Technically it means having some kind of senses that make you aware of the external environment too but that's a minor difference from consciousness - I only said "sentience" because it's what most other people talking about this say. They really mean consciousness. (And also text based IO can be a sense.)
The room argument has a pointless human in it which is clearly clueless about the dialog in order to 'prove' that the room as a whole is clueless.
But imagine applying that to a single person: pick a single neuron -- does it 'understand' our conversation? no.
So why would any singular component of a chinese speaking room-system understand?
It also fails at the opposite extreme, since we're willing to tolerate unreasonably large rooms -- what about one running a full molecular dynamics simulation of a human. As best as we understand physics that simulation would behave just as the human would and must be sentient. You cannot deny the abstract possibility of machine intelligence without rejecting physics for mysticism, only the practicality/plausibility of it.
The point of this thought experiment is to illustrate that merely replicating a behavior - in that case translation - does not say anything about sentience. The Chinese Room may produce intelligent output, but it does not reason as a human does. I find it remarkably prescient. ChatGPT can produce remarkably intelligent output, should we consider it human? If not, then you implicitly agree with Searle, at least in some level.
Quoting Searle himself,
"The point of the argument is this: if the man in the room does not understand Chinese on the basis of implementing the appropriate program for understanding Chinese then neither does any other digital computer solely on that basis because no computer, qua computer, has anything the man does not have."
I think the most succinct description of his error is substituting the (lack of) understanding of a part of the system (the man) for the understanding of the entire system (the rules and file cabinets, etc.). But I'm interested in learning that I'm mistaken.
You could turn his position around and say it's not the computer itself that's intelligent when a Chinese room system exhibits intelligence but the program -- and I suppose I'd agree with that, but it's also just semantics, uninteresting, and I don't believe he's ever taken that position.
I do agree that "merely replicating a behavior" doesn't prove much, but I don't think the Chinese room speaks to that substantially. It might if it demanded that the room implement only a very simple input to output map, but it doesn't: it allows the room to implement anything a computer program can implement. (A fact I use in my post to point out that the room could (in our land of hypotheticals) implement the molecular dynamics of an entire human being)
GPT has structural properties that make it very easy to classify it as an entirely different thing than a human mind. GPT is frozen in time, it cannot have an internal existence due to how its structured. It doesn't even have memory. It cannot learn (unless you include the whole company training it as part of 'it') except in the sense that it can immediately adapt to the output right in front of it, but can't preserve the knowledge. Theoretically if you made it arbitrarily large you could say it was close enough to having memory by always evaluating its complete history, but because its size grows quadratically with its window that isn't practicaly (and might not be possible to train-- it's totally credible that beyond some size these models will lose performance we just haven't gotten there yet). Figuring out how to train these models make good use of 'memory' is an ongoing challenge, since efficient memory isn't differentiable just ordinarily training with memory as part of the process doesn't work. Except by 'thinking out loud' in its output GPT also has a fixed upper bound on the time it can spend thinking any thought which is seemingly unlike a human mind.
> I think the most succinct description of his error is substituting the (lack of) understanding of a part of the system (the man) for the understanding of the entire system (the rules and file cabinets, etc.). But I'm interested in learning that I'm mistaken.
As you know, there have been many replies to this thought experiment, and some of the most interesting ones (to me) go in the direction you went here, ie, where is "understanding" occuring? The most basic version of the Chinese Room does intend to make you see yourself literally as a man who does not understand any Chinese and is just asked to look up symbols in a list. Perhaps that man doesn't understand Chinese, but the room as a whole at least gives the impression that it does.
However, I think the most important aspect is not this "intuition pump" as Daniel Dennett calls it. To me, what is key here is that we can all agree that such a Chinese Room, or ChatGPT for that matter, does not necessarily replicate the fundamental mechanisms of human cognition. Then, it follows that other human properties such as awareness or qualia do not necessarily emerge from such cognitive architectures in the same way that it emerges from our brains.
To me, Searle's point is ultimately that we don't know enough about the human mind to be able to judge whether it can be replicated artificially. And now that we have almost literally developed a Chinese Room, we can see that clearly. The arguments you bring up in your last paragraph are a great example of that, it's just very hard to conceive that this thing is conscious at all, even though it is capable of producing output that could convince people of that.
Regarding Searle's quote that you brought up, I think "solely on that basis" is doing a lot of heavy lifting there, but it does align with what I said previously. He is saying that simply producing intelligent output, like in 1974 translation would represent, does not mean you are reasoning in a human way.
There's string circumstantial evidence that we do. And really "computers can simulate the physics in the brain" is the null hypothesis.
In any case why is the Chinese Room always stated as if it has a clear conclusion rather than "this doesn't really prove anything" if we don't know enough to say either way?
> And now that we have almost literally developed a Chinese Room
I don't think so. The GPTs are currently still very far from the complexity of the human brain, and they are missing many features that may make a big difference to consciousness - for instance the ability to learn while running.
So while it may be fairly easy to say ChatGPT isn't conscious/sentient, that isn't the question. It's whether computers theoretically can't be conscious because consciousness comes from some physical property that they can't reproduce (like quantum microtubule crankery).
To me, it does have a clear and definitive conclusion, which is that mere intelligent output does not mean you are replicating human intelligence, or any higher order mechanism such as consciousness. We don't know enough to tell that it doesn't have any consciousness, but that's beside the point.
You mentioned that a computer that could simulate every molecule of a human brain would also likely replicate sentience. Of course the tricky part is how do you prove that assertion, if all you have is output? If I transfer your brain to an advanced computer as you describe, can I conclude that you're conscious based on what you tell me? I don't think so, because with present technology I could likely make a passable version of your writing output with a LLM. To me that's the real value of the Chinese Room, which is to expose precisely this dillemma. People wrote all sorts of replies to it in order to tackle that - you may be interested in reading about Dennett's p-zombies if you haven't already.
The italics summarise it pretty neatly. It's an argument explicitly framed against Turing's more dubious thought experiment. If even a conscious being in the room can follow instructions, retrieve data and perform operations on it related to symbol manipulation flawlessly without having any sort of "understanding" of anything the symbols actually correspond to, there's no reason to deduce that the running part of a silicon-based machine must from the quality of the symbol outputs it can emit when plugged into a big enough library. Critics' insistence that this makes the "error" of neglecting the possibility that ongoing "understanding" (as opposed to inert symbolic representation of an absent writer's understanding) takes place in the books are actually irrelevant to this point, as well as more than a bit weird. Living outside a Chinese room, I also improve my communication skill and interpret others' understanding by interacting with books, but I wouldn't consider the books themselves a constituent part of my thought processes.
As you point out yourself, GPT has structural properties which make it very easy to classify as an entirely different thing from a human mind despite the similarity of outputs it is capable of producing, and the hypothetical room is even more dissimilar. The possibility it can produce output tokens which correspond to abstractions which humans interpret as consistent with human thought is not evidence that "thought" resides in patterns of abstract representation, not the physics of the organism. We know language is lossy.
That doesn't explain why it can have a debate with itself as 4 distinct personalities at the same time, or act as a used car salesman and actually haggle with me over the price of a Ferrari, or write a story about anything you want, with a beginning, middle, and ending.
Those aren't in the corpus. After a tense negotion, I was able to get that Ferrari for $87,000 (ChatGPT originally wanted $120,000).
Emergent behavior is possible with these incredibly complex systems.
It was very interesting to watch. The "participants" actually started arguing with each other about the right answer, or straight-up insulting each other.
Searle's "Chinese Room" thought experiment is designed to appeal to your intuition. But it appeals in an incredibly unrealistic way. If you fill a room full of people shuffling Han characters (why would anyone do this?) you cannot possibly have anything resembling intelligence.
According to random guesses found on the internet, ChatGPT requires at least eight A100 GPUs to generate text. If you believe nVidia's marketing numbers, this gives you about 2.5 petaflops.
That's 2.5 quadrillion operations per second, plus communications overhead. If you decide to implement that calculation, imagine 350,000 versions of planet Earth, each with 7 billion people performing one operation per second. And some kind of faster-than-light communication, I suppose.
It's absolutely obvious to me that nothing like "thought" could possibly occur in any realistically-imaginable room full of people shuffling symbols. But if you fill 350,000 planets with people shuffling symbols frantically... I'm no longer sure? I don't trust my intuition at all? My brain is made up of a lot of atoms, and they somehow produce thought, after all.
Now, ChatGPT is not conscious. We're still several major breakthoughs away from any kind of "real" AI, I think. All we have now is a language module, a large memory of knowledge about the world, and some very inconsistent reasoning abilities. Although I have a nasty suspicion that at least a few of the missing parts will be easy to invent once someone tries...
I'd go further and say that many of the responses I saw were actively abusive and could trigger significant mental health issues in vulnerable readers.
I would love that Bing provide context on where it found the information and provide an assessment on how reliable it is, but I am sure it'll be gamed by SEO very quickly. Plus a demo of this, even through it's useful, wouldn't look impressive as it lacks confidence.
Maybe you could make a companion module that pre or post processes the GPT-* outputs to slot in facts using a less AI-y but more accurate knowledge graph system? There are things at google or something like Wolfram Alpha that could provide those inserts perhaps.
That's definitely been my big hang up about the usefulness of Bing Chat or ChatGPT for answering questions. If you actually care about the truth of what you're asking you have to go do a lot of the same searching you would have to do to look up the answer in the first place. At best it could provide an idea of what to search when you don't know the language to use to find something, which is often a roadblock for when I'm learning a new system or service.
In a few cases, I've seen it give citations to things that clearly never existed.
For example, I asked it for US auto-pedestrian death statistics. It printed out a nicely formatted table.
Then, I asked it for a source. It pointed me to a specific table on a dot.gov page. The table didn't exist and, according to the Wayback Machine, it never did.
I ended up finding the information the old-fashioned way, and numbers that it gave were way off.
I suspect the majority of folks won't be bother to fact check the data it returns. It's going to be a problem.
It got the general field-of-work correct (which is niche). Yay! Everything else, it hallucinated -- my school, my work history, etc.
Truly amazed me that such a tiny part of the internet landed in its model.
Anyway, when I did get to try Bing Chats, it was nowhere near the same level of usefulness I found when using the free version of Chat GPT. If it was using GPT-4, then that's worrying. I've not tried Bing Chats again since. (mostly because it's so gated behind forced use software).
These tools are very effective, and giving access to lay people lets them use them to do things like say racist things and have silly arguments and answer puzzles wrongly and then go online and say the machine is broken and it shouldn't be released, and that it should be banned, etc.
Essentially, the better world for the disbelievers is that the tool is not available to them. The best world for the believers is that they have access to the tool. These are compatible.
You could easily run a model orders of magnitude larger than current ones within that envelope under reasonable assumptions about your duty cycle.
There is no reason not to practice customer segmentation in this scenario.
It’s impressive how some of the conversations with Bing AI went. Many people hypothesized it’s a newer model because of those points, and now we have proof