If someone is following a ‘program’ for responding to Chinese characters, that’s as good as speaking Chinese since there is no distinguishable difference.
So far as we know, consciousness does not exist as a physical thing. The behavior of a human is completely derivable, in principle, from natural law.
There is no physical test or manifestation of awareness.
So arguments about what can or can not be conscious have the same flavor as arguments aswhether there is or is not a God. It's unproveable!
Yet unlike God, most people do not deny the existence of consciousness, because of direct personal experience.
Consciousness is the "unproveable yet true" statement in the theoretical system that is the physical universe. Probably it haunts any physical system complex enough to host it.
So does all this talk about computers being "the same" if they can, given access to sufficient human-generated inputs, produce similar strings of Chinese characters to those a conscious Chinese person might do.
If you're not stuck behind a WeChat prompts it's trivially externally observable that a big silicon box which outputs Chinese characters and an agglomeration of cells which walks, eats, makes funny faces and reproduces are dissimilar in most respects (the machine might generate a subset of human outputs which is consistently convincingly human-like, but it's trivially shown that it runs different operations on different hardware at a different speed, requires different inputs to function effectively, and it's highly probable it doesn't devote clock cycles to dreaming about the physical and hormonal release of mating with other computers.
Something which in every observable respect is the same as me isn't a computer, it's me (or perhaps a clone or twin). A computer which can produce text outputs indistinguishable from mine is a very impressive trick indeed, but trust me, my sister will spot the difference straight away when she tries to give it an EEG scan!
So imagine we put your brain in a vat; we'll give you a webcam and a microphone for input, and for output -- ah, sorry, budget constraints, just an old printer. You visualize typing on a keyboard in your mind's eye and the characters are tapped out irl on a long scrolling sheet of paper.
Would you still feel? Would you still feel like you?
I'd guess yes and only sort of, respectively. Perhaps you wouldn't be as interested in sex (or maybe it would depend on what mix of hormones the vat was feeding you).
I think we can safely say your sister wouldn't immediately recognize you, though. But given some quality time QA, I think she's end up concluding you were still you, and more than just a parlor trick.
But what do you think? Is it you? If it is, it doesn't seem THAT different from the computer program you, does it?
I think that suffers the same flaw as logical positivism: if my axioms can't find a difference, there isn't one, no way my assumptions are wrong. (Namely, my axiom is that external observations capture the entirety of reality, there is nothing subjective.)
If two people laugh at a joke, one faking and one actually finding it funny, what is the externally observable difference? Assume the faker has been trained in all manner of knowledge about what would make the joke funny, they just don't find it so.
I don't really find the Chinese Room argument very compelling because there are too many "it's obvious that X can't really understand" in it.
Also, you can't derive from it that there can't be computed consciousness in some other form.
The system's response is trivial: Sure, if the room+person combination leads always to a coherent response in Chinese, then the entire system understands Chinese. I'd go even further: If the person in the room does not understand Chinese, but the system does, then there is some entity that understands Chinese - either a person or an advanced AI, feeding the inputs. Then, from the systems perspective, the person in the room is largely irrelevant.
But this is not the argument: Despite no discernible difference from the outside, the person in the room may either understand Chinese, or they may not. And so there is a distinction - from the perspective of the individual in the room, that does not depend on the outside observation.
That's all there is to it. It shows that meaning and understanding are not the same as syntactic computation (an important point, to be sure), but it does not show that one can exist with or without the other. By extension, it does not otherwise disprove consciousness as being this or that.
The analogy might be more valid if arguing its not possible for a third party to actually determine whether an entity/system is conscious (irrespective of whether the entity is conscious or not)
Instead, the core matter is about form versus meaning - something that is indeed not observable from the outside, and yet is a distinction to the person inside the Chinese room.
Edit: Or as fouronnes3 said in a sibling comment, why it's actually evidence against Searle's original argument.
Cargo cults that developed in the Southwest Pacific after WWII reportedly attempted to emulate rituals performed by U.S. military personnel such as landing signals, believing they’ll bring back the aircraft that had been giving them gifts.
Similarly, believing that a program will possess consciousness if we provide it with some of its external manifestations seems backwards.
Of course, the problem is that we don’t know what consciousness is. Until we do, I’ll keep assuming we don’t have the proficiency to create it under such different conditions just yet.