It’s worse: they look at what they anthropomorphically think it’s doing. Like looking at an animal picture classifier work and thinking it can see animals, or a watching a self-driving car and thinking it has some kind of simulation of a human driver in it.
I think the “AI purists” need to make a lot more noise about the fact that nobody knows what LLMs are doing.