Lying requires intent and knowledge.
I agree though that it's a bad term for what current LLMs do. "Confabulate" seems closer.
I think we have no way of knowing, just like we don't know if an ant has consciousness or not.
> The results are sensitive to the detailed specification of the lateral connectivity and suggest that the cortical mechanisms that generate geometric visual hallucinations are closely related to those used to process edges, contours, surfaces, and textures.
Now this example is only one theory, but it shows that serious people can talk in these terms about hallucination and expect to be understood and not have to argue past reviewer #2, without this whole question vitiating their discussion.
Also, btw, we don't fully know that GPT-4 is not conscious, if you really think it matters. When the question comes up, people who are certain it's not usually point to the fact that it's not running continuously, as if that had any bearing on the question.
We humans call it "misspeaking".
Speak quickly, often enough, and without sufficient training and yule misspeak.