Hacker News Reader

Comment by kibwen | Hacker News Reader

Except that the Chinese Room shows that the existence of a mapping from input to output, however emergently it might have been devised, is not alone sufficient to demonstrate understanding.

View on HN

"Understanding" is your word, it doesn't appear in the source article nor the comment you're responding to. The Chinese Room argument does not attempt to show that a mapping from input to output can't implement a latent space, or that it can't implement complex models of what the language is describing. If a person can express those things explicitly in the output, or if you have to do them in order to correctly respond how a person would, then the room by definition has those capabilities.

(What's the point of the argument if it doesn't tell us anything about the capabilities or internals of an AI? I'm not sure.)

Right - the point I was refuting is that LLMs are "just statistical models of words." More is going on. Does that imply "understanding?" I don't know, I'm not sure we have a good enough definition to say. But it does mean that the models are more complex that say, markov chain graphs with corpus frequencies. It seems we are encoding data in the latent space with much higher complexity than "just words." There is higher order semantic information being captured - probably not the same has human "thoughts" - but again - also not _just words_.

The Chinese Room as a whole understands.

The man in the room is comparable to a human hand, and the magic rulebook to a human brain.

This in the sense that we can easily retain human "understanding" by stripping away almost all parts of the human body or replacing them with fairly trivially made replacements, except for the brain.

In the Chinese Room the equivalent is the magic rulebook: We have no idea how to construct/replace it, yet people somehow handwave that away whilst simultaneously confidently asserting it does not understand anything.

That's irrelevant to what we're talking about, though. The point here is not to assert whether or not the room understands, but to emphasize that our methods are insufficient to demonstrate this. You could posit that the room understands, or you could posit the reverse, and neither argument can be refuted.

An LLM is a big equation that we solve to get textual output. If Ai proponents already believe an equation can contain consciousness, what about the Chinese Room presents a more compelling counterargument?

I'm not bringing up the Chinese Room in its original sense of asserting that a machine can't have consciousness. Indeed, I have no reason to suspect that human consciousness is irreproducible. Rather, the point here is to emphasize that our methods of determination are insufficient. A man armed with an English-to-Chinese dictionary appears to know Chinese until he's faced with an instance of linguistic ambiguity outside the scope of what a dictionary encodes; in other words, you can, with enough probing, disprove that the room understands Chinese. But how do you prove it in an affirmative sense? Let's use a simpler example: does Hans the Clever Horse understand arithmetic? We can disprove it by throwing enough arithmetic at Hans to prove that his understanding does not generalize, but if Hans did know arithmetic, how would we prove it? There are plenty of things in this category--things that we believe to some confidence level, but cannot prove--but let's frame it for what it is: belief and faith, not proof or logic. Whatever utility I may derive from LLMs, I have every reason to be skeptical of a movement of people whose awe of LLMs echoes the steadfast furor of the religious adherent.

It does so only in the claims of its creator. Plenty of other people have pointed out fallacies in the claim. My favorite is the Dennett/Hofstadter observation that while the man in the room may not understand chinese, the room/system certainly does.

If our own brains were insufficient to demonstrate the fallacy of the Chinese Room; the particles, molecules, cells, synapses, and so forth of our bodies cannot be believed to have understanding, but we recognize it in the sum of the part anyway....

... we now have LLMs to even more pointedly show the deficiencies of the argument.

Reply on news.ycombinator.com