We gotta stop ignoring AI's hallucination problem

A statistical model the can guess the next word is in no way "intelligent" and Sam Altman himself agrees this is not a path to AGI (what we used to call just AI).

Also the same for asking about apps/tools. Unless it is a super known app like Trello which has been documented and written about to death - the LLM will give you all kinds of features for a product, which it actually doesn’t have.

It doesn’t take long to realize that half the time all these LLMs just give you text for the sake of giving it.

People will clamor for LLMs that tell them what they want to hear, and companies will happily oblige. The post-truth society is about to shift into overdrive.

Every AI use I have comes with a big warning.

The internet is full of lies and I still use it.

Well snake oil sells. And the margins are great!

I'm firmly in the camp that considers calling it "hallucination" or "getting things wrong" category errors that wrongly imply it gets anything right, and have seen nothing that remotely inclines me to revise that opinion. I recognise it as an opinion, though. It can't be proven until we have an understanding of what "understanding" is that is sufficiently concrete to be able to demonstrate that LLMs do not possess it. Likewise the view that human minds are the same kind of things as LLMs cannot be disproven until we have a sufficient understanding of how human minds work (which we most certainly do not), however obviously wrong it may seem to me.

Meanwhile, as that discussion fizzles out, the commercial development of these products continues apace, so we're going to find out empirically what a world filled with them will mean, whether I think that's wise or not.

In fact, many sell their product as hallucination free.

About the best analogy I can come with is freestyle rapping where the genre requires that you just keep talking with minimal time to think.

An LLM isn't an intelligent being deciding to talk when it has something to say - it's an LLM - a text continuation machine - that will generate a word every time the generation scaffolding pushes the button. Just like the freestyle rapper, it doesn't have the option of not saying something just because it has nothing to say. It's "concerned" about flow (just keeping the sentence going), not content.

There's an assumption that there's a system that by default produces correct representations, but has malfunctioned and produced a representation of an absent object.

But isn't the whole process of ML making what would be a malfunction into a procedure? Generating humongous numbers of completely faulty representations until some emerge that look like they're mirroring reality? Is hallucination in ML not the default state? Its correctness being a judgment of an observing subject, and in all cases, a happy accident that has struck the observer as being real by only looking real. Aren't language models just playing on how impressionable we are? How easily we're tricked into believing that something is a real thing based only on superficial attributes?

Of course I suspect they are 20 years old at most who never took part in a real project.

Unknowns alone aren’t the problem. See the camera example in the article. We know which steps to take to unjam film from a camera, the given answer is straight up bad.

And that's because they are exactly what they say they are on the tin. They are Language Models. Why would we expect a Language Model to "know" things? And to especially know what they do not know? It's modelling Language, not Knowledge, and that is not just English word play, that is a true representation of the situation. They are called "Large Language Models" for good and real reasons.

The problem isn't on the LLM side. They're amazing technology. They just aren't amazing in the way we're trying to use them.

We need some sort of AI that is knowledge based, that uses LLMs to input and output text but is something other than an LLM. Human brains aren't just big LLMs either. I, of course, have no idea what that is. Identifying problems is much, much easier than identifying solutions.

Honestly, neural nets in general are structurally disinclined to ever say "I don't know" just by the way they work. Only a small percentage of humans ever really learn how to say "I don't know" in any significant context to any degree; we have a pretty big problem with that ourselves!

LLMs are not "AI"; that is, they are not all of AI, the only thing in AI, or the the final conclusion of AI. They're just a step. And as I've said before, I suspect that once we have the next step in AI, what we're going to find is that the correct amount of resources to dedicate to the LLM portion is actually a great deal smaller. LLMs are probably super hypertrophied for what they actually should be doing and I expect 2040 to look back at our era of thinking that LLMs alone could be "AI" in much the same way we look at the idea that everyone in the world would have a Segway or something.

I think the difference is highlighted when someone is able to answer a question with "I don't know" or "I can't tell you that".

Interestingly, LLMs can be trained to answer that way in some circumstances. But if you're cunning enough, you can trick them into forgetting they don't know something.

While a human may hallucinate memories, we crucially have the ability to know when we're relying on a memory. And to think of ways we could verify our memory, or acknowledge when that's not possible.

Firstly - Humans Hallucinate. We have the ability to be impaired to the point that we incorrectly perceive base reality.

Secondly - LLMs are always ‘Hallucinating’ Objective reality for an LLM is relations between tokens. It gets the syntax absolutely right, but we have an issue with the semantics being wrong.

This is simply not what the model is specced to do. It’s fortunately trained on many conversations that flow logically into each other.

It is NOT trained to actually apply logic. If you trained an LLM on absolutely illogical text, it too would create illogical tokens - with mathematical precision.

We won't, because the machine results people want will be those that appeal to their biases.

If you finetune it in formats that humans likes, it actually gains similar biases.

I believe the ‘sparks of AGI’ talks about this, where models can much more accurately predict the probability of events than humans.

After RLHF, it mimicks human bias. So it might be that we can create these models, but we just don’t like them/like to use them.

Ok, I can only laugh on this.

Those "hallucinations" are the one main feature neural networks. It's imposed by their fundamental structure, and what gives it all if its usefulness... when the people making use of it knows what they are doing.

The way to reduce/control the hallucinations of a neural network is to unplug it from the power outlet.