John Searle's Chinese Room argument seems to be a perfect explanation for what is going on here, and should increase in status as a result of the behavior of the GPTs so far.
https://en.wikipedia.org/wiki/Chinese_room#:~:text=The%20Chi....
https://en.wikipedia.org/wiki/Chinese_room#:~:text=The%20Chi....
Your brain runs on the laws of physics, and the laws of physics are just mechanically applying local rules without understanding anything.
So the laws of physics are just like the person at the center of the Chinese Room, following instructions without understanding.
But there's no such clear evidence, and the quantum hypothesis itself seems to be popular mainly among those who reluctantly accept materialism of consciousness, but are unwilling to fully accept the implications wrt their understanding of "freedom of will". That is, it is more of a religion in disguise.
If you apply this rule to machine learning, why can a neural network and it's model not have emergent properties and behavior too?
(Maybe you can't, I dunno, but my toddler-level analogy wants to look at this way)
The only reason the setup described in the Chinese room argument doesn't feel like consciousness is because it is inherently something with exponential time and/or space complexity. If you could find a way to consistently understand and respond to sentences in Chinese using only polynomial time and space, then that implies real intelligence. In other words, the P/NP distinction is precisely the distinction underlying consciousness.
For more, see: