The problem with LLMs isn't hallucination, it's context specific confidence

This article and attendant comments reveal the AI sector is turning to co-dependent excuse making for a technology that clearly can't live up to its hype.

Get ready for phrenology of AI...

"I am going to need to visit your data center to lay hands on the subject."

Humans get rewarded for thinking "I don't know", a lot. That's why it's hard to compare.

> A model that always bluffs

A model doesn't bluff. It feels to us humans that they bluff, but there is no bluff mechanics in play. The model doesn't assess the prompter's ability to call their bluff. It's not hiding that it doesn't know something. It's just not reached a predictable point in a sequence of token predictions that can or not have something that resembles a call to what resembles a bluff.

Up to the point it's corrected, the model's representation of what was asked is the best it can do. It has no means to judge itself. Which leads to...

> The real issue isn’t that models make things up; it’s that they don’t clearly signal how confident they are when they do.

Which sounds like exactly what I said, but it's not. Signaling confidence is just a more convincing faux-bluff. Signaling is a side-effect of bluffing, a symptom, not the real thing (which is more related to asessing whoever is on the other side of the conversation).

> Imagining things, seeing problems from the wrong angle, and even fabricating explanations are the seeds of creativity.

I agree with this. However, Newton was not bluffing, he was right and confident about it, and right about being confident about it. It just turns out that his description was of a lesser knowledge resolution than Einsten's.

For this to work, we need lots of "connective tissue" ideas. Roads we can explore freely without being called liars. Things we can say without saying that these things are true or false, without the need for being confident or right, without being assessed directly. This is outside the realm of bluffing or saying useful things. It's quite the opposite.

When people saw comets and described them as dragons in the sky, they were not hallucinating or telling lies, they were preserving some connective tissue idea the best they could, outside of the realm of being right or wrong. This were not bluffs. There were some "truths" about their mistakes, or something useful (they were unadvertedly recording astronomical data, before astronomy existed). Those humans felt that was important, those stories stuck. Can we say the same thing about LLM hallucinations? I don't think we're ready to answer that.

So, yes. Hallucinations could be a feature, but there's a lot missing here.