Daniel Dennett is a philosopher who made it sort of his favorite subject: competence without comprehension. I won't attempt to explain this better than he does.
But the general idea is that these models are more elaborate versions of ELIZA, while they aren't in principle different. They work like my child who when I quiz him after reading him a page of a bed-time story on the contents of it will try to come up with plausible answers, but I can tell when he was actually paying attention / knew the words or weren't / didn't.
Now, this is something I don't know whether Dennett would claim, so, it's my thought, not his. I believe that a fundamental difference is in motivation / value judgement. I.e. humans do things for a reason that they constructed based on what they perceive to be more valuable to them. Models aren't built to have values or wishes. They don't want to be able to answer more "why?" questions about whatever they "say".
I mean, your reflection in the mirror is a very realistic counterfeit human being, and in many respects is indistinguishable from you, but it's nothing like a human being.