OpenAI's new reasoning AI models hallucinate more

I’ve also seen a few of those where it gets the answer right but uses reasoning based on confabulated details that weren’t actually in the photo (e.g. saying that a clue is that traffic drives on the left, but there is no traffic in the photo). It seems to me that it just generated a bunch of hopefully relevant tokens as a way to autocomplete the “This photo was taken in Bern” tokens.

I think the more innocuous explanation for both of these is what Anthropic discussed last week or so about LLMs not properly explaining themselves: reasoning models create text that looks like reasoning, which helps solve problems, but isn’t always a faithful description of how the model actually got to the answer.

Claude also does that apparently. You give it a hint and it’ll lie about using that hint.

They talk about it here: https://www.anthropic.com/news/tracing-thoughts-language-mod...

Wouldn't reasoning training be expected to cause catastrophic forgetting of ground truth random fact stuff learned in the main training?

Do they keep mixing in the original training data?

Oh wow, yeah, it even gloats a bit about doing it. I've never seen that before.

> One possible explanation here: as these get smarter, they lie more to satisfy requests.

I feel there's some kind of unfounded anthropomorphization in there.

In contrast, consider the framing:

1. A system with more resources is able to return more options that continue the story.

2. The probability of any option being false (when evaluated against the real world) is greater than it being true, and there are also more possible options that continue the story than ones which terminate it.

3. Therefore we get more "lies" because of probability and scale, rather than from humanoid characteristics.

GaslightPT

I first noticed this with DeepSeek R1. For some really hard questions (some not even answerable), it would come up with a line of reasoning that convinced me that it had the right answer. If I read the answer without the reasoning, it was clear it made no sense.

We might be incentivizing answers that sound right with reinforcement learning as opposed to answers that are actually right.

They're just people pleasers. Over commit on every request because they can't say no. Probably like the engineers that were forced to program them.

The reward it gets from the reinforcement learning (RL) process probably didn’t include a sufficiently strong weight on being truthful.

Reward engineering for RL might be the most important area of research in AI now.

When I asked GPT-4.1 to show some references to confirm an answer was as widely accepted as it claimed, it replied with a bunch of unrelated GitHub issues with fake descriptions, and this Stack Overflow link: https://stackoverflow.com/questions/74985713/fastifypluginas.... Turns out this is actually a link to a completely unrelated question https://stackoverflow.com/questions/74985713/plotting-a-3d-v.... It had tricked me by rewriting the title part of the url, which is not used to identify the question.

I’ve seen this in all the models I’ve used. They give you false info, you call them on it, they say “oh yep ha here’s the right info” and that right info may or may not be correct still.

I’m not sure about your interpretation of the events.

From the transcript:

> (Model, thinking): Could also be Lake Zug, considering the architecture. The user mentioned they were in Switzerland for postgrad, so it could be a familiar place.

> (Model, thinking): (Goes onto analyse the EXIF data)

To me, this reads as a genuine, vision-based guess, augmented with memory of your other chats, that was then confirmed with the EXIF data. Seems to me that the model then confirms it did so, not that it skipped straight to checking the metadata and lying about it as you accuse.

Fascinating. It's learned to be unabashed.

Yeah, this is interesting. You asked it to play geoguessr - and it played the game as a geoguessr player, by "guessing" the location, and responded like a player would. How much more truthful/accurate is it when you just ask it to tell you the location?

It feels like the two requests in the prompt effectively turned into "Guess this location like a geoguessr player".

If you’re training it to always answer one way, then it will lie in order to satisfy the scoring.

It’s like when someone asks if you like Hamilton. Of course you do, we all do.

Chat LLMs are becoming mirrors. It's bad user experience when you say something and they double down, it gets downvoted and RLHF tunes it out.

I asked a question about Raspberry Pis one time and it mentioned they're great low-cost computers for education or hobby. I responded saying they're so expensive these days and it went "You're absolutely correct" and changed its stance entirely saying they're focusing on enterprise and neglecting the hobby/education market. I edited my response to instead say something in agreement and ask a followup question, and it responded completely opposite of before, talking about how it's revolutionizing education and hobby computing in 2025 by being affordable and focused on education and hobby. Try this sometime, you'll realize you can't have any serious opinion-based discussion with chat models because they'll flip-flop to just mirror you in most circumstances.

This bleeds into more factual discussions too. You can sometimes gaslight these models into rejecting basic facts. Even if it did use physical features to deduce location and the image had no EXIF, there's a high chance the same reply would get it to admit it used EXIF even if it didn't. Meanwhile if it did use EXIF, you could have a full conversation about the exact physical features it used to "deduce" the location where it never admits it just checked EXIF

I don't interpret this as lying. It's role-playing a "geoguesser" as prompted.

Imagine if you prompted the model the "play geoguesser", and it responded with something like "Well I actually ran python to pull out metadata and I know the exact location". That would be a bad response to the prompt. It's role-playing as if it played the game in its response, and I think that's the right thing for the model to do in this circumstance. If you tell it not to use the metadata, it won't.

Even where the training data does say "I don't know" (which it usually doesn't-- people don't tend to comment or publish books, etc. when they don't think they know) that text is reflecting the author's knowledge rather than the models... so it would be off in both directions.

One could imagine a fine tuning procedure that gave a model better knowledge of itself by testing it and on prompts where its most probable completions are wrong fine tune it to say "I don't know" instead. Though the 'are wrong' is doing some really heavy lifting since it wouldn't be simple to do that without a better model that knew the right answers.

I found the same when enabling web search it seems that it's blindly copying the contents of the URL without any more thinking

Were you using o3-mini before o3 came out? I wonder if o3-mini has had the same issues if it is from the same series of models.

Anthropic had a recent paper that might be of interest.

https://www.anthropic.com/research/tracing-thoughts-language...

At a high level, what causes hallucinations is an easier question than how to solve them.

LLMs are pretrained to maximize the probability of predicting the n+1 token given n tokens. To do this reliably, the model learns statistical patterns in the source data and transformer models are very good at doing that when large enough and given enough data. It is therefore suspect to any statistical biases in the training data because despite many advances in guiding LLMs, e.g. RLHF, LLMs are not sentient and most approaches to get around that such as the current reasoning models are hacks over a fundamental problem with the approach.

It also doesn't help that when sampling the tokens, the default temperature of most LLM UIs is 1.0, with the argument that it is better for creativity. If you have access to the API and want a specific answer more reliably, I recommend setting temperature = 0.0, in which case the model will always select the token with the highest probability and tends to be more correct.

There's the anthropic paper someone else linked, but also it's pretty interesting to see the question framed as trying to understand what causes the hallucinations lol. It's a (very fancy) next word predictor - it's kind of amazing that it doesn't hallucinate! Like that paper showed that there were circuits that functionally actually do things resembling arithmetic and computation with lookup tables instead of just blindly 'guessing' a random number when asked what an arithmetic expression equals and that seems like the much more extraordinary thing that we want to figure out the cause of!

In pretraining, the model is just trying to predict the most likely next token given the context. Its guesses are not always correct, which leads to hallucinations. Post-training often incentivizes the model to sound confident in its output, which can make the problem worse.

All responses are hallucinations; some of them are close enough to what we want to be useful, others not so much.

Hallucinations are kind of the default, like finding hay in a haystack. The leads (needles) people search for are on what causes non-hallucinations.

One vague possibility: hallucinations are mitigated by looking at uncertainty in token prediction, the idea being that it’s a proxy for factual uncertainty, and the system is more likely to confabulate a court case or whatever if it only has a fuzzy idea of what comes next. But this won’t work for reasoning models which solve math problems: the next token for “x=“ is highly uncertain until you’ve done the computation!

"[Hallucinations] don't form from out-of-distribution samples — they arise due to spurious attractor states on the Energy Landscape. In other words, 'noisy bumps' in the loss. So in the physics approach, to characterize the training error, we ask how sensitive is the free energy to noise. This is why in statistical physics, the training and generalization errors are not defined distributionally. That is, we don't define them using a training set and a test set. Instead, the errors are found by using the free energy as a generating function and taking the appropriate partial derivative."

https://www.linkedin.com/posts/charlesmartin14_talktochuck-t...

There is not really some distinct pathology with hallucinations, its just how wrong answers (e.g. inaccuracies / faulty token prediction chains) manifest in the case of LLMs. In the case of a linear regression, a "hallucination" is when the predicted value was far from the actual value for a given sample.

Hallucinations are caused by humans anthropomorphizing LLMs and imagining that they possess properties that don't exist.

For example, LLMs cannot test their thoughts against external evidence or other knowledge they may have (such as logic) to think before they output something. That's because they are a frozen computation graph with some random noise on top. Even chain of thought prompting or RL-based "reasoning" are just a pale imitation of the behavior we actually wish we could get. It is just a method of using the same model to generate some context that improves the odds of a good final result. But the model itself does not actually consider the thoughts it is <thinking>. These "thoughts" (and the response that follows them) can and do exhibit the same defects as hallucinations, because they are just more of the same.

Of course, the field has made some strides in reducing hallucinations. And it's not a total mystery why some outputs make sense and others don't. For example, just like with any other statistical model, the likelihood of error increases as the input becomes more dissimilar to the training data. But also, similarity to specific training data can be a problem because of overfitting. In those cases, the model is likely to output the common pattern rather than the pattern that would make sense for the given input.

Yann LeCun gave a recent talk, at 10:30 he gives a good explanation that doesn't appeal to "what LLMs are trying to do on the inside": LeCun just observes that next-token prediction is exponentially diverging, since every next token with the smallest error e is still (roughly speaking) (1-e)^n for output of length n. I just thought it was a good complexity-theory based explanation (unlike most other statistical-parrot type arguments).

It is a single slide, very helpful: https://www.youtube.com/watch?v=ETZfkkv6V7Y

Hallucinations are what make the models useful. Well, when we like them we call it intelligence "look it did something correct that was nowhere in the training data set! It's intelligent!" and when we don't like them we call it hallucinations "look, it did something that was nowhere in the training dataset because it was wrong!".

But they are the same thing: the model is extrapolating. It doesn't know when its extrapolations are correct or not because an LLM doesn't have access to the outside world (except via you and whatever tools you give it).

If it was free of extrapolations it would just be a search engine over the training data, and that would be less useful.

Nope. There are many applications where the current behavior is fine. Thousands of developers use them to be more productive every day.

It’s honestly a little surprising that LLMs are good at information retrieval from their training set at all. (Specifically, it's not a surprise that they hallucinate-it's a surprise that sometimes they don't at rates better than chance)

It’s not necessary for them to ever be reliable at it for them to be useful... just stop asking them to do things they aren't good at, and may never be.

The strength of LLMs is areas where verification is significantly easier than production. This is why they have had so much success as coding assistants.

My exact thinking. It is the wrong end to measure it. As with nearly all tech benchmarks. They measure success rate, if LLM is 95% accurate it is good enough as human on average only get about 97%. What we dont measure is the difference between the 5% and 3% of failure rate. Some of the error in 5% error couldn't have happened to any human or are basically non-human error. Vs say 3% of human error that are different type.

I still haven't found a term for this. And I have been saying it since Apple's Keyboard butterfly fiasco. When Apple supporters keep saying the key press has 99.99% success rate so the problem with the keyboard were amplified. What they dont realise was the old scissors keyboard has infinitely close to 100%, or 99.99999999999999999% success rate so to speak. A normal consumer comparing intuitively will felt the butterfly keyboard is 0.01 / 0.000000000000001 many times worst because the error doesn't happen before with previous keyboard.

Since I dont see anyone online doing this I am going to call it Ksec's Law.

Absolutely. I always thought LLMs were interesting, but in no way 'ready for prime time' and have been absolutely blown away that it seems like it's been shoehorned into every possible real life use case in such a reckless way.

I've even heard of lawyers being forced to use it.

It is breathtakingly irresponsible.

Lol that's not comparable.