Having seen photocopiers so many times produce coherent, sensible, and valid chains of words on a page, I am at this point in absolutely no doubt that they are thinking.
The proper category error in the context of the discussion would be to say the photocopier is drawing a picture.
It doesn't matter how well or not the photocopier recreates an image. To say the photocopier is drawing a picture is just nonsense and has no meaning.
The same category error as to say the LLM is "thinking".
Of course, the category error could be well exploited for marketing purposes if you are in the business of selling photocopiers or language models.
You see how doesn’t make sense what you saying?