A Large Language Model is a tool and people have to learn to use it, just like any other tool. I have used GPT as a machine learning model in a way similar to the grandparent comment, and I now have some understanding of when and why it hallucinates in certain situations [0]. I use that understanding to filter responses and tweak prompts. It took a lot of time and effort invested, but I was able to get a handle on it.

I know that a lot of people won't do this, and will just accept whatever an LLM says at face value. And again, that's true for any tool. But if you invest the time to understand the parameters and limitations of the model, it can be incredibly valuable.

[0] in my experience, GPT davinci is much more likely to hallucinate in (non-programming) situations that would be difficult for a human to explain. using the above example, it can easily handle a standard credit application. But it will be more likely to hallucinate reasoning for a rare case like someone with a very high income but a low credit score. YMMV, just sharing what I've seen