Spoken as someone who has never seen a weather forecast. Computers have long produced results that were, if interpreted naively, incorrect. These systems require humans to take into account the conditional, probabilistic nature of the output and the inherent limitations of the model the computer was executing, when they interpret the output. LLM's are no different. They give you next token prediction probabilities. That is all. There is no particular reason to think that comes (or even could come) with a promise that those tokens will express truths about the world and only truths. The fact that greed-driven hype, and our human inclination to anthropomorphize the token generator inclines us to imagine things should be different doesn't make it so. No model output is truth. Never has been.