Hacker News Reader

Comment by rzz3 | Hacker News Reader

Does anyone have any technical insight on what actually causes the hallucinations? I know it’s an ongoing area of research, but do we have a lead?

View on HN

At a high level, what causes hallucinations is an easier question than how to solve them.

LLMs are pretrained to maximize the probability of predicting the n+1 token given n tokens. To do this reliably, the model learns statistical patterns in the source data and transformer models are very good at doing that when large enough and given enough data. It is therefore suspect to any statistical biases in the training data because despite many advances in guiding LLMs, e.g. RLHF, LLMs are not sentient and most approaches to get around that such as the current reasoning models are hacks over a fundamental problem with the approach.

It also doesn't help that when sampling the tokens, the default temperature of most LLM UIs is 1.0, with the argument that it is better for creativity. If you have access to the API and want a specific answer more reliably, I recommend setting temperature = 0.0, in which case the model will always select the token with the highest probability and tends to be more correct.

Thanks for the explanation, it makes a lot of sense. One thing I keep wondering though is why we can’t teach them logic at some sort of base layer. Not to anthropomorphize, but couldn’t the actual next-token prediction itself embed some sort of consultation against another (non-LLM) model that can evaluate a probability of logicality of the overall output that would be produced be selecting next-token-A vs next-token-B? I’m sure it’s obviously way more complicated than this, but still.

> One thing I keep wondering though is why we can’t teach them logic at some sort of base layer

That is what they did with these "chain of thought" models. Maybe they didn't do it in the optimal way, but they did train them on their ability to answer certain questions.

So the low hanging fruits from this style of training has already been plucked.

I know this is reductive, but at their core, LLMs really are just very fancy autocomplete. The process they go through bears no resemblance to reasoning.

Grab the best model you’ve got access to (o4, Gemini Pro 2.5, Claude Sonnet 3.7, etc) and try playing chess with it. The results are astonishingly bad.

It’s not just that LLMs make poorly thought out moves. They regularly make completely illegal moves (making a knight move like a pawn, for example), hallucinate new pieces into existence, change pieces from one color to another, etc.

More words have probably been written about Chess than any other game. You can literally buy a book that’s just about chess openings. There has to be a staggering amount of text about chess in the training data set. And yet, even the best models are far worse at chess than I was in 3rd grade (and I’m not great).

They do pretty well for the first few moves, often playing well known chess openings. But once they get to the part where you need to reason, they fall apart in the most outrageous ways. I suspect this reveals something about how they deal with other logic-based tasks. I’ve wondered for a while now if they basically code through using a vast amount of training data from Stack Overflow, which is why they’re so good at helping with common error codes, and so useless when doing something novel.

These new chess experiments have been eye-opening about how incapable the best models are of basic reasoning. Unless something about that changes drastically, I think LLMs have pretty much plateaued. There will undoubtedly be some advancements, but LLMs are never going to reach AGI without reasoning, nor will they be able to do most jobs.

Hallucinations are caused by humans anthropomorphizing LLMs and imagining that they possess properties that don't exist.

For example, LLMs cannot test their thoughts against external evidence or other knowledge they may have (such as logic) to think before they output something. That's because they are a frozen computation graph with some random noise on top. Even chain of thought prompting or RL-based "reasoning" are just a pale imitation of the behavior we actually wish we could get. It is just a method of using the same model to generate some context that improves the odds of a good final result. But the model itself does not actually consider the thoughts it is <thinking>. These "thoughts" (and the response that follows them) can and do exhibit the same defects as hallucinations, because they are just more of the same.

Of course, the field has made some strides in reducing hallucinations. And it's not a total mystery why some outputs make sense and others don't. For example, just like with any other statistical model, the likelihood of error increases as the input becomes more dissimilar to the training data. But also, similarity to specific training data can be a problem because of overfitting. In those cases, the model is likely to output the common pattern rather than the pattern that would make sense for the given input.

There's the anthropic paper someone else linked, but also it's pretty interesting to see the question framed as trying to understand what causes the hallucinations lol. It's a (very fancy) next word predictor - it's kind of amazing that it doesn't hallucinate! Like that paper showed that there were circuits that functionally actually do things resembling arithmetic and computation with lookup tables instead of just blindly 'guessing' a random number when asked what an arithmetic expression equals and that seems like the much more extraordinary thing that we want to figure out the cause of!

All responses are hallucinations; some of them are close enough to what we want to be useful, others not so much.

Correct responses wouldn't be considered hallucinations.

Not by the human evaluator, assuming they know (or otherwise validate) that the generated output is correct. But an LLM alone doesn't differentiate between hallucinations and not-hallucinations.

... But they ought to be considered hallucinations, it's the same algorithm at work, we just are biased in favor of certain results.

Similarly, suppose I always roll dice to determine tomorrow's winning lottery-ticket number. Getting it right one day doesn't change the mechanism I used. Some people might assume I was psychic, but would be wrong.

You're absolutely correct and it's ridiculous that HN can't fathom that a hallucination, in both the traditional and LLM sense, is not characterized by its form or content per se but by the fact that the form or content is incongruent with some external factor.

Would you say the same of humans?

I would absolutely say the exact same thing about humans. Except that humans know less. We also have a sense of uncertainty which is absent in LLMs.

So you are saying hallucinations are poorly-calibrated responses, in a statistical sense.

Do you even know what an hallucinations is?

We have the same generative ability, but we gradually learn the correct answers (not necessarily truths), so we don't go around generating incorrect ones (not necessarily falsehoods). Then we know when we don't have any answers, so generating one is fruitless.

LLMs is just generation. Whatever pattern they embedded, they will happily extrapolate and add wrong information than just use it as a meta model.

That is a good question. I think the biggest difference is that humans have access to sensory data that grounds us in reality. If you put a human in complete sensory deprivation, I think we would hallucinate much more.

indeed, isn't hallucination one of the purported intended effects of a sensory deprivation tank?

One vague possibility: hallucinations are mitigated by looking at uncertainty in token prediction, the idea being that it’s a proxy for factual uncertainty, and the system is more likely to confabulate a court case or whatever if it only has a fuzzy idea of what comes next. But this won’t work for reasoning models which solve math problems: the next token for “x=“ is highly uncertain until you’ve done the computation!

RLHF generally tends to make the model pretty confident making the entropy of it a not so useful predictor of when the model is going to get something wrong.

You can do a kind of sensitivity analysis to see how sensitive the output is to small perturbations of the weights... but it's computationally expensive.

Might be an interesting form of fine tuning to do distillation where the student's current sensitivity to noise (extracted from the backwards step of gradient descent) is used to shrink the predicted distribution towards uniform. It could be done very cheaply during training and perhaps could avoid the back propagation cost of doing it at inference time.

Hallucinations are kind of the default, like finding hay in a haystack. The leads (needles) people search for are on what causes non-hallucinations.

"[Hallucinations] don't form from out-of-distribution samples — they arise due to spurious attractor states on the Energy Landscape. In other words, 'noisy bumps' in the loss. So in the physics approach, to characterize the training error, we ask how sensitive is the free energy to noise. This is why in statistical physics, the training and generalization errors are not defined distributionally. That is, we don't define them using a training set and a test set. Instead, the errors are found by using the free energy as a generating function and taking the appropriate partial derivative."

https://www.linkedin.com/posts/charlesmartin14_talktochuck-t...

In pretraining, the model is just trying to predict the most likely next token given the context. Its guesses are not always correct, which leads to hallucinations. Post-training often incentivizes the model to sound confident in its output, which can make the problem worse.

Hallucinations are what make the models useful. Well, when we like them we call it intelligence "look it did something correct that was nowhere in the training data set! It's intelligent!" and when we don't like them we call it hallucinations "look, it did something that was nowhere in the training dataset because it was wrong!".

But they are the same thing: the model is extrapolating. It doesn't know when its extrapolations are correct or not because an LLM doesn't have access to the outside world (except via you and whatever tools you give it).

If it was free of extrapolations it would just be a search engine over the training data, and that would be less useful.

Yann LeCun gave a recent talk, at 10:30 he gives a good explanation that doesn't appeal to "what LLMs are trying to do on the inside": LeCun just observes that next-token prediction is exponentially diverging, since every next token with the smallest error e is still (roughly speaking) (1-e)^n for output of length n. I just thought it was a good complexity-theory based explanation (unlike most other statistical-parrot type arguments).

It is a single slide, very helpful: https://www.youtube.com/watch?v=ETZfkkv6V7Y

Anthropic had a recent paper that might be of interest.

https://www.anthropic.com/research/tracing-thoughts-language...

The interpretation this paper offers is very questionable.

It observes a so-called "replacement model" as a stand-in because it has a different architecture than the common LLMs, and lends itself to observing some "activation" patterns.

Then it liberally labels patterns observed in the "replacement model" with words borrowed from psychology, neuroscience and cognitive science. It's all very fanciful and clearly directed at being pointed at as evidence of something deeper or more complex than what LLMs plainly do: statistical modelling of languages.

Calling LLMs "next token predictor" is a bit of a cynical take, because that would be like calling a gaming engine a "pixel color processor". It's simplistic, yeah. But the polar opposite of spraying the explanation with convoluted inductive reasoning is just as bereft of substance.

There is not really some distinct pathology with hallucinations, its just how wrong answers (e.g. inaccuracies / faulty token prediction chains) manifest in the case of LLMs. In the case of a linear regression, a "hallucination" is when the predicted value was far from the actual value for a given sample.

Reply on news.ycombinator.com