See also the Chinese Room argument, which got a lot of airtime back in the day. It added no useful insight to questions about the nature of machine intelligence, but it did reveal how little we understood about the nature of language.
Searle's translation book was essentially an LLM. Somehow (because it is a book?) we are to assume it cannot be in anyway like human intelligence despite it making convincing responses.
This may perhaps be more obvious to a naturalistic philosopher or natural scientist, than a computer "scientist" (ie., a mathematician).
The meaning of the term "pen" in "pass me that pen" includes the pen. So when this room is asked, "pass me the pen" and it replies "i cannot pass the pen" (or whatever it replies) -- it should be obvious that the person in the room, or any function of their activity, has never acquired any reference to "the pen". It is wholly unaware that there is a pen at all.
The purpose of this thought experiment is to show that syntactical correctness or apparent "arrangement of symbols in 'a' correct order" is radically insufficient to evidence semantic competence.
This, again is perhaps more obvious to scientists -- the symbol order is only a proxy measure of semantic competence in people. It's trivial to come up with processes which clearly lack the capacity for such competence and yet are measured (/observed) to produce symbols in the right order.
In many ways, it's an over-engineered thought experiment. However I'd say Searle was baffled that more obvious phrasings of the problem seemed to confuse others, ie., that an observation of symbols isnt an observation of meanings -- one isn't a reliable measure of the other. Only under very many additional conditions does such a relationship hold in people.
Turing was not interested in producing systems that had such competence, so he may well agree with Searle in some ways at least. However, many students of computer science receive no empirical education whatsoever, and lack the basic vocabulary and understanding of the nature of the problem of meaning.
Eg., that in order to mean "pass me the pen" one must be able to acquire a reference to "the pen" which any system unable to observe its environment at the very least cannot do.
Turing machines lack devices, and hence lack any capacity to in principle refer to objects in the world. The only thing a turing machine can be said to do is express an abstraction ( a function of nat -> nat) -- since it is an abstraction.
No capacities follow from expressing such a computational abstraction -- Searle thought the chinese room made this obvious to those who didnt find it so. But he was baffled that anyone didnt already find it obvious.
One could make the same point with physics, rather than with meaning. Eg., the earth orbiting the sun computes +1,-1,+1,-1 .. and so does an infinte numer of physical processes that share no properties with the earth, or the sun, etc. Thus just because we observe +1,-1,+1... does not mean that "inside the chinese physics room" there's an earth orbiting the sun. It could literally be anything.
And yet Searle seems to pass the buck here to a book that actually "responds", not to the person in the room. I get it: the person is out of the loop.
But how does one explain the book that can answer so convincingly? That would appear to be where the "AI" resides.
The TV which displays a video game outputs images as-if there were a whole world inside the TV box: there isnt.
Alright, so what neuron in your brain "understands" English ?. Hell feel free to name any part regardless. This is why the Chinese Room is nonsensical. Either you admit the system can understand even when none of the constituents do or you admit you don't understand anything at all either. At least either conclusion would be consistent.
Unfortunately, we have many people take the nonsensical middle road. "Oh that doesn't understand but I certainly do, just because."
Yes, neurones do not understand "pen" -- but some highly particular whole bodies do (ie., english spekaing people). That's because of highly particular relationships between those neurones, the body, the environment, and the history of that language user.
This is the csci brain rot that searle is baffled by. Symbol manipulation implies no relationships between wholes and parts. The capcity to understanding meaning requires extraordinarily specific ones.
Saying, "if you can't point to the neuron that does X, then you can't prove X happens" isn't a scientific perspective. It's a willfully ignorant one. If you're confident in the scientific process, then we will eventually understand how all kinds of human mental processes make sense in the context of neural networks.
>It does not take place in an LLM.
I don't know what else to tell you but LLMs absolutely model concepts and the physical world, separate from the words that describe them. This has been demonstrated several times.
However we might, as a practical matter, have a large number of proxy tests and treat a system as meaning-capable if it passes.
Searle thought he had come up with just such a question, but it turned out that he hadn't.
My take is that the whole chinese room argument rests on extremely nebulous and shaky definitions/assumptions, rendering it worthless.
It seems fairly obvious to me that the system of operator + rulebook does "understand" chinese, for every practical definition of "understanding".
Another counterargument would be simple physical simulation: If you built a computer program that could simulate a human by the atom, then you would either have to concede that the resulting machine does "understand" for all definitions that matter, or you have to admit that you believe in magic [1].
[1]: or desperately grasp for loopholes, like nondeterminsitic physical micro-interactions, but might as well call that magic.