> I don't think we have the correct word for what LLMs do but lie and hallucinations are not really correct.

I believe 'bullshit' is accurate, as in "The chatbot didn't know the answer, so it started bullshitting".