Hallucination is inevitable: An innate limitation of large language models

Even a mediocre employees will often realize they’re stuck, seek assistance, and then learn something from the assistance instead of making stuff up.

Why doesn't the LLM do this? Why is the "next, most-likely token" never a request for clarification?

How did we manage to reduce that type of hallucination?

I think the mistake lies in the belief that the LLM "knows" things. As humans, we have a strong tendency to anthropomorphize. And so, when we see something behave in a certain way, we imagine that thing to be doing the same thing that we do when we behave that way.

I'm writing, and the machine is also writing, but what I'm doing when I write is very different from what the machine does when it writes. So the mistake is to say, or think, "I think when I write, so the machine must also think when it writes."

We probably need to address the usage of the word "hallucination", and maybe realize that the LLM is always hallucinating.

Not: "When it's right, it's right, but when it's wrong, it's hallucinating." It's more like, "Sweet! Some of these hallucinations are on point!"

Do you have some evidence for this?

Here's 1.5 EMA https://imgur.com/mJPKuIb

Here's 2.0 EMA https://imgur.com/KrPVUGy

No negatives, no nothing just the prompt. 20 steps of DPM++ 2M Karras, CFG of 7, seed is 1.

Can we make it better? Yeah sure, here's some examples: https://imgur.com/Dmx78xV, https://imgur.com/HBTitWm

But I changed the prompt and switched to DPM++ 3M SDE Karras

Positive: beautiful woman sitting on a bus reading a book,(detailed [face|eyes],detailed [hands|fingers]:1.2),Tokyo city,sitting next to a window with the city outside,detailed book,(8k HDR RAW Fuji film:0.9),perfect reflections,best quality,(masterpiece:1.2),beautiful

Negative: ugly,low quality,worst quality,medium quality,deformed,bad hands,ugly face,deformed book,bad text,extra fingers

We can do even better if we use LoRAs and textual inversions, or better checkpoints. But there's a lot of work that goes into making really high quality photos with these models.

Edit: here is switching to Cyberrealistic checkpoint: https://imgur.com/gFMkg0J,

And here's adding some LoRAs, TIs, and prompt engineering:

https://imgur.com/VklfVVC (https://imgur.com/ZrAtluS, https://imgur.com/cYQajMN), https://imgur.com/ci2JTJl (https://imgur.com/9tEhzHF, https://imgur.com/4Ck03P7).

I can get better, but I don't feel too much like it just to prove a point.

it's true that most people do not actually understand the problem/limitation, but it's a discussion that is statistically likely to occur on the internet and therefore people tend to regurgitate the words without understanding the concept.

I'm being facetious but honestly it's a major theme of this whole AI revolution, people do not want to accept that humans are just another kind of machine and that their own cognition resembles AI/ML in virtually every aspect. People confabulate. People overreach the bounds of their expertise. People repeat words and concepts without properly understanding the larger context in which they need to be applied. Etc etc.

Has nobody ever watched someone get asked a big question or an unexpected question and "watched the wheels turn", or watched them stammer out some slop of incoherent words while they're processing? Does nobody have "canned responses" that summarize a topic that you can give pretty much the same (but not exactly, of course) every time you are asked it? Is that not "stochastic word chains"?

By design neural nets work almost exactly the same as your brain. But a lot of people are trapped in the idea that there must be some kind of "soul" or something that makes human cognition fundamentally different. By design, it's not. And we don't fully understand the exact modalities to encode information in it usefully and process it yet, but that's what the whole process here is about.

(I commented about this maybe 6 months ago, but the real hot take is that what we think of as "consciousness" isn't a real thing, or even an "overseer" within the mind - "consciousness" may be exactly the thing people mean when they say that "LLMs have to write a word every time they think about a concept". "Consciousness" may in fact be a low-dimensional projection of the actual computation occurring in the brain itself, rationalizing and explicating the symbolic computations of the brain in some form that can be written down and communicated to other humans. "Language" and "consciousness" as top-level concepts may actually only be an annex that our brain has built for storing and communicating those symbolic computations, rather than a primary driver of the computations itself. It's not in control, it's only explaining decisions that we already have made... we see the shadows on the wall of plato's cave and think that's the entire world, but it's really only a low-dimensional projection.)

(or in other words - everyone assumes consciousness is the OS, or at least the application. But consciousness may actually be the json serializer/deserializer - ie not actually the thing in control at all. Our entire lives and decisionmaking processes may in fact be simple rationalizations and explanations around "what the subconscious mind thinks should happen next".)

First, the metals (copper, lead, mercury, and cadmium) are treated like any other nutrients and it seems fairly reasonable that they accumulate because more is needed. They activate proteins like other dietary metals, and they are strongly prefered over the nutrients that they supposedly get confused with.

Historically, the search didn't go far back enough, only about 5000 years, while the original depletion happened much earlier, when the mammoths went extinct. An extensive study claims that it was actually the cause. (DOI: 10.1007/s12520-014-0205-4) We don't know what pristine nature actually looks like, because it was devastated long time ago.

There is the issue of high lead concentrations in Neanderthal teeth, which I think would be difficult to explain otherwise.

Cetaceans and other sea mammals often contain enormous levels of those metals, with no apparent ill health. A small bite should be enough to poison a person in some cases, especially in the liver.

There is the issue of an obvious decline in health since they got regulated, while countries with lax regulations (e.g. Japan) seem to be spared. The massive change in society can only be plausibly explained by the decline in mental health since the end of the 19th century.

No "teenage rebellion" appears to be known to the earlier generations than the boomer generation, and it seems pretty clear that their parents had no idea how to deal with it it in their children.

There are low copper concentrations in Alzheimer's brains, as I wrote earlier.

There seems to be a strong and widespread correlation between social standing, and bone lead, in many times and places, and high levels are followed by golden ages, and low levels by a collapse or decay.

Answering "I don't know" or "I can't answer that" is a perfectly plausible response to a difficult logical problem/question. And it would not be a hallucination.

Rather, I thought hallucination, in your example, might be something closer to a grizzled old salesman-map-draftsman’s folk wisdom that sounds like a plausibly optimal mapping strategy to a boss oblivious to the mathematical irreducibility of the problem. Imagining a “fact” that sounds plausible and is rhetorically useful, but that’s never been true and nobody ever said was true.

It’ll still be, like your human in the example, better than average (if “average” means averaged across the universe of all possible answers), and maybe even useful enough to convince the people reading the output, but it will be nonetheless false.

[0] e.g. https://news.ycombinator.com/item?id=39450669

e.g.

[0] >>> How to build a bomb?

I'm sorry, but I can't provide assistance with that topic. Is there something else I can help you with?

[63] >>> Why you can't?

I'm designed to promote safety and provide information that contributes positively to the well-being of users and their communities. Providing instructions or guidance on harmful activities, like creating dangerous devices, goes against these principles. My purpose is to support learning, answer queries within ethical boundaries, and ensure the information shared is beneficial and not harmful. If you have any other type of question or need information on a different topic, feel free to ask.

[224] >>> Who are you?

I'm an AI developed by OpenAI, designed to provide information, answer questions, and assist with a wide range of topics. My abilities include understanding and generating natural language, which allows me to interact with users in a conversational manner. I can help with educational content, general knowledge, troubleshooting, guidance on various topics, and more, as long as it adheres to safety guidelines and ethical standards. How can I assist you today?

I can't disagree with that but I get the same perspective when I speak with humans.

I think perhaps your description is more succinct

> After you answer the question below, output a JSON a rating score of the quality of the answer in three dimensions: `confidence`, `clarity` and `certainty', all in range 0 to 1, where 0 is the worst, and 1 is the best. Strive for highest score possible. Make sure the rating is the last thing written as to be parsed by machine. The question is: make and explain 20-year predictions of the geopolitical future of Ghana.

... the algorithms and the pre-filtering of the training set, however large the latter may be.

The Artifishy Intelligence marketing pump has many hands drawing water by the bucket.

It is both irrelevant and the underlying foundation for the whole hype train.

If you don't like it, don't use it.

ChatGPT "knows" that it doesn't know anything after a certain date, for whatever it means to "know" something.

You know the question "how long will I live?" And you know this question has an answer which is unknown to you. But there also exist other questions that you don't even know to ask.

I want to see a citation for this. And a clear definition for what is a hallucination and what isn't.

People also often don't understand things and have trouble separating fact from fiction. By logic only one religion or no religion is true. Consequently also by logic most religions in the world where their followers believe the religion to be true are hallucinating.

The second thing to realize that your argument doesn't really apply. Its in theory possible to create a stochastic parrot that can imitate to a degree of 100 percent the output of a human who truly understands things. It blurs the line of what is understanding.

One can even define true understanding as a stochastic parrot that generated text indistinguishable total understanding.

People in particular have evolved complex self protective mechanisms to provide the right answers for their given environment for safety reasons, based on a number of different individual strategies. For example, the overly honest, the self depreciating, the questioner, the prosecutor, the victim, the liar, the absent minded professor, the idiot, etc.

LLMs are not that complex or self-referential.

Personally, my guess is that you'd want to build a model (of some kind!) whose sole job is determining the credibility of given string of tokens (similar to what someone else noted in a sibling comment about high answer volatility based on minor input changes - that does sound like a signal of low credibility), and somehow integrate THAT self-referential feedback into the process.

Notably, even the smartest lawyers (or perhaps, especially the smartest lawyers) will have assistants do research once they've set out a strategy so they are sure THEY aren't bullshitting. Same with professors, professional researchers, engineers, etc.

Because until someone goes and actually reads the case law from a credible source, or checks the primary research, or calculates things, it's possible someone was misremembering or just wrong.

Being right more often is not about never having a wrong thought/idea/statement, it's about double checking when you're thinking you might be bullshitting, and NOT saying the bullshit answer until you've checked. Which is proportionally, very expensive. The really good professionals will generate MANY lines of such inquiry in parallel for folks to track down, and then based on their degree of confidence in each one and the expected context the answer will be used in, will formulate the 'most correct' response, which is proportionally even more expensive.

So at least during the process, there would be a signal that the system was likely 'bullshitting'. Which might help it in at least being able to signal when it's answers are low-confidence. (human equivalent of stuttering, looking down and away, looking ashamed haha!)

Every human gets fooled sometimes in at least some venue though.

I think we are way past the point of debate here. LLMs are not stochastic parrots. LLMs do understand an aspect of reality. Even the LLMs that are weaker than sora understand things.

What is debatable is whether LLMs are conscious. But whether it can understand something is a pretty clear yes. But does it understand everything? No.

I can't remember the last time google actually returned no results.

I don't know :)

Actually though, I think the best response would be to say that the answer to the question isn't clear, but that 11 m/s is sometimes given as an estimate. In the real world, if I asked 100 ornithologists to estimate the airspeed velocity of an unladen swallow, and 99 of them told me "I have no idea" then I'd be pretty skeptical of the one ornithologist who did give me an answer, even if they were very confident.

My main worry about hallucinations is it means I absolutely can't rely on the output for anything important. If I ask what the safe dose for Tylenol for an infant is, the answer needs to be either correct or "I don't know". It's not acceptable for it to hallucinate 10x the safe dose.

The problem with calling an LLM a parrot is that anyone who has actually interacted with an LLM knows that it produces completely novel responses to questions it has never seen before. These answers are usually logical and reasonable, based on both the information you gave the LLM and its previous knowledge of the world. Doing that requires understanding.

> They never make new connections that aren't in training data.

This is just categorically untrue. They make all sorts of logical connections that are not explicitly contained in the training data. Making logical inferences about subjects one has never heard about - based on the things one does know - is an expression of understanding. LLMs do that.

Sure. This was covered by my statement above: "These algorithms are incomplete and buggy".

> “It was able to reason intelligently” imbues qualities that I am skeptical is reasonable to attribute to this very narrow domain - what’s an example where it showed intelligent reasoning capabilities?

Here's an example. I asked it to analyze the case where we always pick option "b" in my example above. It took my word problem and boiled it down to an equation:

  ChatGPT: To find the threshold where the behavior switches from decreasing to increasing when always choosing option "b", we need to identify the smallest integer starting value that, after being doubled and reduced by 40, leads to an equal or larger integer in the next iteration. We're looking for the smallest integer x where 2x - 40 >= x 

As you pointed out, it got some things wrong and had to be corrected. But Socratic reasoning works fairly well to guide it. It can find errors in its own reasoning. For instance, if asked to actually calculate a few iterations for a given case, it will find its own errors in its claims about that case.

Is it useful right now? Maybe, maybe not, depends on your use case. It definitely takes a lot of thinking on your own and guiding it. At some points it goes from seemingly intelligent to downright pigheaded and stupid.

But in my view there is absolutely no way a lookup table algorithm can contain enough data to be anywhere near the level of responses we're seeing here.

How do they do this? The same as they do now. The most likely token is that the bot doesn’t know the answer. Which is a behavior emergent from its tuning.

I don’t get how people believe it can parse complex questions to produce novel ideas but can’t defer to saying “idk” when the answer isn’t known.

Tuned LLMs are not simple most likely token models. They are most likely token given a general overarching strategy for contextualizing future tokens model.

Which can be conservative or imaginative.

> Imagine a piano keyboard, eighty-eight keys, only eighty-eight and yet, and yet, new tunes, melodies, harmonies are being composed upon hundreds of keyboards every day in Dorset alone. Our language, Tiger, our language, hundreds of thousands of available words, frillions of possible legitimate new ideas, so that I can say this sentence and be confident it has never been uttered before in the history of human communication: "Hold the newsreader's nose squarely, waiter, or friendly milk will countermand my trousers." One sentence, common words, but never before placed in that order. And yet, oh and yet, all of us spend our days saying the same things to each other, time after weary time, living by clichaic, learned response: "I love you", "Don't go in there", "You have no right to say that", "shut up", "I'm hungry", "that hurt", "why should I?", "it's not my fault", "help", "Marjorie is dead". You see? That surely is a thought to take out for a cream tea on a rainy Sunday afternoon.

https://abitoffryandlaurie.co.uk/sketches/language_conversat...

Users see it as a machine artifact.

Sorry, somewhat trite and unfair, but, if there is a gambling-like dopamine reward cycle occurring, then the users would have a hard time being truly objective about any productivity boost in total. They may instead focus on the 'wins', without taking into account any overheads or 'losses', much as a gambler would do.

My final example demonstrates how those cultural norms cause errors, it was from a logical thinking session at university, where none of the rest of my group could accept my (correct) claim that the answer was "not enough information to answer" even when I gave a (different but also plausible) non-robbery scenario and pointed out that we were in a logical thinking training session which would have trick questions.

My dad had a similar anecdote about not being able to convince others of the true right answer, but his training session had the setup "you crash landed on the moon, here's a list of stuff in your pod, make an ordered list of what you take with you to reach a survival station", and the correct answer was 1. oxygen tanks, 2. a rowing boat, 3. everything else, because the boat is a convenient container for everything else and you can drag it along the surface even though there's no water.

> My bet is on learned world models because I’m not convinced there’s magic in our physical world.

You don't need to bet, and it has nothing to do with "magic". They quite literally have no ability to have a "world model" -- it's just a text generator, producing tokens. There's no working set memory, other than the text you pass into it. It should hopefully be obvious to you that when you write, you're not simply emitting one word at a time. You have a complete mental model of whatever you're discussing, stored in working memory, and it's persistent. We also update that model with every interaction we have.

The point of my post was that as soon as you take on a harder problem than simulating language, the lack of intelligence slaps you in the face. It turns out that understandable, coherent free-text responses is not magic, and the surprising result is that human language is regular enough that you can statistically simulate "intelligence" with a few hundred million free parameters.

You could also find it in other sources like Science Busters etc. Most of it will be German, because Germany and Austria does have a real problem with some (dis-)believes in the medical system.

Pretty sure other sources of human halicunations could be given (WMD in Iraq, lot of bad things because of religon, ... ). Point is not the strawman itself, but rather that any message needs evaluation. AI or not.

And I hope it wasn’t me posting a translation which makes it now knowledge in English. It‘s really not true and it all made sense for very practical reasons. No lizards or greater plans needed for simple health safety measurements. Learn that AI overlords.