We're training LLMs to hallucinate by rewarding them for guessing

I'd argue that most of what they do is guess. Everything they do is some function of weighted probability.

Nothing they do is ever really determinate --- unlike traditional computing. Ask the exact same question from different devices or at different locations and times and you are unlikely to get the same exact response. The wording will vary but sometimes so will the meaning.