I looked it up because of what you said. It’s a pretty substantial article. What part of it is a fallacy? The part about computers obviously not having intelligence?
So, the opposite assumption is that there's nothing more to intelligence than a "mere computation", and all that stuff like inner dialog and "a sense of self" are just side-effects of it. That way, there's no difference between "strong AI" and "weak AI" in his terminology, no difference between "understanding" and "simulation of understanding". Then, of course, a digital computer has a mind and understands Chinese, and also the Chinese room has a mind and understands Chinese. But Searle doesn't understand Chinese and this is supposed to be a contradiction.
But there's no contradiction here, because Searle is not the Chinese room. He isn't the mind, in this model he is to Chinese room what electricity is to a digital computer, or what neurotransmitters are to human brain. So, yeah, electricity doesn't understand Chinese, serotonin doesn't understand Chinese, single human neuron doesn't understand Chinese, but Turing-test-passing computer, Chinese brain and Chinese room all do. Searle is just a detail in the Chinese room — tiny and insignificant compared to a book, containing a complete description of a program that speaks Chinese and is able to form coherent (and persistent!) opinions on every possible topic in the world.
So, the fallacy is equating Searle's role in the hypothetical Chinese room to the whole digital computer that speaks Chinese and passes Turing test with breeze.
A similar case I noticed is the resistance against the Everett interpretation of the quantum measurement problem (i.e. the wave function collapse). For some people, it's so hard to accept the consequences of your own consciousness being an emergent phenomenon of matter that interacts with other matter and thus it partakes in the very fundamental effects of such interactions, including entanglement.
However, I come to a different conclusion. I think that all this is exactly why the Chinese room concept is interesting — because it is so obvious, but somehow induces a huge resistance in so many people. I.e. the thought experiment itself isn't all that interesting — people's reactions to it are.
What's your take on this (very rapidly acquired) impression?
In it, Searle argues that no computer program will be able to satisfy the goal he calls 'strong AI', which he defines thus: "according to strong AI, the computer is not merely a tool in the study of the mind; rather, the appropriately programmed computer really is a mind, in the sense that computers given the right programs can be literally said to understand and have other cognitive states."
As far as I recall (and I am not going to read the whole thing again just to check) he does not precisely define 'computer' or 'computer program', but it is clear that his meaning subsumes ordinary usage: a Turing-equivalent digital device, together with any program that can run on it. It is also clear that, given his underlying position being that syntactical manipulation cannot give rise to semantics, he would include, in his definition, actual universal Turing machines, and any other device, digital or not, that, in his view, can only perform syntactical operations.
Therefore, Searle is saying that computers will never think, except in a most pedantic reading of 'computers'. He is, indeed, a materialist, and regards the brain to be a mind-instantiating machine of sorts, but, like Penrose, he thinks its consciousness must depend on something beyond what we have discovered so far.
Personally, I think the argument begs the question in this sense: it is predicated on the assumption that the only thing in the room that could understand anything would be the human operator, but that, in turn, is predicated on the assumption that nothing else about the room could do so. That he is doing this is made very clear in his attempts to rebuff the 'systems reply' and the 'simulator reply'.
Therefore nobody speaks Chinese.
The most grating part of the article is involving political names into the explanation of the argument, and that the writings by Liebniz and Dneprov seemingly would offer far more interesting material to base the topic on.
Wikipedia can be terrible source of misaligned atheist theology sometimes.