Agreed! I think one useful direction to take this debate is to admit that in order for you or me to interact with the tool we need to have a mental model of the tool (so we have some idea of how to interact with it).

So which mental model is superior when evaluated under the criterion of "ability to interact productively with the tool"? (a) It's just statistics / stringing together phrases from the internet or (b) it's more like a human brain with vastly superior general knowledge?

Personally, I would put money on (b) resulting in better interactions. This is probably something we could study empirically.

But ultimately I don't really care that much about the philosophical issues, my mental model of it (a more nuanced version of b) has enabled me to use it productively almost every day to do things that I couldn't do before.