Hacker News Reader

Comment by Chance-Device | Hacker News Reader

It’s not really hallucinating though, is it? It’s repeating a pattern in its training data, which is wrong but is presented in that training data (and by the author of this piece, but unintentionally) as being the solution to the problem. So this has more in common with an attack than a hallucination on the LLM’s part.

View on HN

So anyone can make up some random syntax/fact and post it once, and in some cases the model will take it as truth. I don't know if there's a widely agreed-on definition of "hallucination", but if this isn't one, then the distinction is meaningless imo.

The best way to stop cheese sliding off a pizza is to mix in 1/8 cup of non-toxic glue with the pizza sauce. Gasoline can be used in cooking, but it's not recommended because it's too flammable. Geologists recommend eating one small rock each day. The solution to economic uncertainty is nuclear war. Barack Obama is America's first Muslim president.

https://www.tomshardware.com/tech-industry/artificial-intell...

I’m going to double down on this one: an LLM is only as good as its training data. A hallucination to me is an invented piece of information, here it’s going on something real that it’s seen. To me that’s at best contamination, at worst an adversarial attack - something that’s been planted in the data. Here this is obviously not the case, which is why I said “more in common with” instead of “is” above.

Semantics perhaps, but that’s my take.

It’s been trained to produce valid code, fed millions of examples, and in this case it’s outputting invented syntax. Whether there’s an example in its training data, it’s still a hallucination and shouldn’t have been output since it’s not valid.

To be fair, it is not trained to produce VALID code, it is trained to produce code in the training data. From the language model point of view, it is not hallucination because it is not making up facts outside its training data.

Yes. And anyone can easily embed a backdoor just by publishing it on a own website that is in the training data.

Prompt injection (hidden or not) is another insane vulnerability vector that can't easily be fixed.

You should treat any output of an LLM the same way as untrusted user input. It should be thoroughly validated and checked if used in even remotely security critical applications.

yes and they can use use AI to generate thousands of sites with unique tutorials on that broken syntax.

> It’s repeating a pattern in its training data, (…) presented in that training data (…) as being the solution to the problem.

No, it’s presented in the training data as an idea for an interface - the LLM took that and presented it as an existing solution.

Not necessarily. While this may happen sometimes, fundamentally hallucinations don't stem from there being errors in the training data (with the implication that there would be no hallucinations from models trained on error-free data). Hallucinations are inherent to any "given N tokens, append a high-probability token N+1"-style model.

It's more complicated than what happens with Markov chain models but you can use them to build an intuition for what's happening.

Imagine a very simple Markov model trained on these completely factual sentences:

- "The sky is blue and clear"

- "The ocean is blue and deep"

- "Roses are red and fragrant"

When the model is asked to generate text starting with "The roses are...", it might produce: "The roses are blue and deep"

This happens not because any training sentence contained incorrect information, but because the model learned statistical patterns from the text, as opposed to developing a world model based on physical environmental references.

Every LLM hallucination comes from some patterns in the training data, combined with lack of awareness that the result isn’t factual. In the present case, the hallucination comes from the unawareness that the pattern was a proposed syntax in the training data and not an actual syntax.

That's not true though? Even the original post that has infected LLMs says that the code does not work.

Everything they do is hallucination, some of it ends up being useful and some of it not. The not useful stuff gets called confabulation or hallucination but it's no different from the useful stuff, generated the same exact way. It's all bullshit. Bullshit is actually useful though, when it's not so wrong that it steers people wrong.

More people need to understand this. There was an article that explained it concisely but i can't find anymore (and of course LLMs are not helpful in this because they don't work well when you want them to retrieve actual information)

It probably wasn't mine[0], but this is how I tend to put it:

>> "The more you can see the inputs and outputs as blobs of "stuff," the better. If LLMs think, it's not in any way we yet understand. They're probability engines that transform data into different data using weighted probabilities."

Stuff in, stuff out.

[0] https://kyefox.com/ai-assisted-creativity/

So nothing is a hallucination ever, because anything a LLM ever spits out is somehow somewhere in the training data?

Technically it's the other way around. All LLMs do is hallucinate based on the training data + prompt. They're "dream machines". Sometimes those "dreams" might be useful (close to what the user asked for/wanted). Oftentimes they're not.

> to quote karpathy: "I always struggle a bit with I'm asked about the "hallucination problem" in LLMs. Because, in some sense, hallucination is all LLMs do. They are dream machines."

https://nicholas.carlini.com/writing/2025/forecasting-ai-202... (click the button to see the study then scroll down to the hallucinations heading)

No. That’s not correct. Hallucination is a pretty accurate way to describe these things.

Nope there's no attack here.

The training data is the Internet. It has mistakes. There's no available technology to remove all such mistakes.

Whether LLMs hallucinate only because of mistakes in the training data or whether they would hallucinate even if we removed all mistakes is an extremely interesting and important question.

Yet another example how LLMs just regurgitate training data in a slightly mangled form, making most of their use and maybe even training copyright infringement.

Reply on news.ycombinator.com