Exactly this. I like testing chat gpt’s hallucination and my most recent one was to ask it to convert a conversation I had so far into html. It followed the prompt correctly but instead of inserting its actual answer that it had given into the html doc, it started giving different answers which included hallucinating new information.

Again, the problem here is there is no way to know for sure if what these ai chat tools are saying is correct or imagined.