OpenAI: Models are programmed to make stuff up instead of admitting ignorance

The original article has already been posted here, the discussion there might be interesting if someone can find it. The Register's click/ragebait framing isn't very interesting.

I don't really think people want these things to say "I don't know" - they want it to know.

That's obviously not reasonable for everything but I bet a lot of hallucinations ARE things where the model should be able to know or figure out. Most people are asking questions with well-known answers.

But I would guess the OpenAI post is correct, fundamentally they are trained in a way that rewards guessing, which I think must make it more likely it guesses even if the answer is in its reach.

This is why I just can't fully get behind the idea that AI is going to be the panacea the hype merchants keep claiming, at least in this era: making stuff up incentivizes users to prompt again, while returning "I don't know" tends to be a conversation ender. They're not incentivized to be accurate, they're incentivized to give the appearance of competence to the point that it keeps users coming back.

Tufte said it best: There are only two industries that call their customers 'users': illegal drugs and software.

This is the problem Aloe (a state-of-the-art agent designed and built by a cognitive scientist) is solving. This article out yesterday delves into exactly that: https://puck.news/is-aloe-the-first-self-building-ai/?sharer...