Do you feel that the human mind is more than an "appropriately" trained "biological" neural network?
What do you consider the limits of a DALL-E like system compared to a "true" mind?
My personal opinion is that the Chinese Room argument is fancy handwaving that crucially relies on never being explicit about what it means by "understanding", combined with an appeal to intuition.
I strongly believe that there is nothing "magical" about the human mind or brain (that could not be replicated artificially), and thus that a comparably trained, appropriately designed system ("DALL-E successor") OR a copy OR a simulation of a human brain would be all just as capable and "understanding"/"conscious" as another human...
bingo
GPT-3 is far, far better at generating descriptions of loving humanity than the average dog. But it's pretty obvious that the dog's goal formation and excitement hormones aren't particularly similar to letter-by-letter ASCII output probability calculations on a database, and that GPT-3 has no more grasp of the dog's love of humanity in general and this human in particular than the dog has of Shakespeare. "Thought" and intelligible results are essentially orthogonal, which limits the feasibility of training the former...
It isn't hand-waving. It's against it, really. It's a thought experiment that encourages a sceptic attitude towards jumps in understanding mental processes. The operator in the Chinese Room doesn't understand Chinese. While the translations are excellent, he or she wouldn't be able to go out in the street and ask for a glass of water if their life depended on it. Hence, a computer that mechanically translates Chinese cannot be automatically assumed to understand Chinese.
The argument doesn't need to explain exactly what understanding means. We all (sort of) know what it means. The same goes for e.g. attention. That's what makes it so hard to define what strong AI is and how to verify it. The Turing Test famously tries to decide this (without defining anything, I might add), and the Chinese Room is a good argument against it being the proper test.
Then what does "understanding" a language mean? Your "asking for a glass of water on the street" example implies to me that you demand a system with: a) capability for internal intent b) ability to express intent in target language to grant it "understanding".
Basically, first you deliberately construct a system not capable of intent (pure stateless query-response algorithm), then you deny that it "understands" Chinese on the basis of being unable to express intent. That does not hold.
Give me a precise definition for what you mean by "understanding" and I'll dismantle the Chinese Room for you.
You probably can't define "chair" in an exact enough sense. It would merely show the shortcoming of the definition rather than refuting the Chinese Room.
> a system not capable of intent
If I now would require a definition of "intent" I would be just as childish.
> That does not hold.
First, the system isn't stateless. It has the operator who has to remember what has happened before in order to come up with a correct translation.
Second, your argument implies that understanding requires intent. That alone is a tall order to prove. But it's the point of the Chinese Room: the mechanism doesn't understand, doesn't have intent, nothing. Yet it performs exactly as an intelligent, understanding, intentional translator. So would you not agree with the conclusion: you can't judge intent by only looking at I/O behavior?
Note that Searle (probably, this is just my understanding) doesn't mean that the Room is useless or "dumb" (in most senses of the word).