Hacker News Reader

Comment by dgacmu | Hacker News Reader

I prefer my 8yo's answer about quantum entanglement, asked just now: "I don't know. How would I know? It's not a thing!"

Even an 8yo has better metacognition, it seems. :-)

View on HN

I suppose the LLM doesn't know it's limited in its knowledge, maybe? That others know more.

Oh, that's interesting - good point, since it's filtered and not trained from scratch. My prior would be to assume it's just bs'ing as LLMs usually do but it seems worth exploring.

"I don't know" isn't in the training data. Nobody writes engineering books, science papers and blog posts that end with "well, I don't really know, the end".

A bit like publishing bias where only positive results end up in scientific papers.

It should be in the post-training though. Hallucination rates are a fraction of what they used to be as a result

The LLM doesn't "know" anything, can't reason about its own knowledge, and has no self-awareness. It has training data, and it can use your prompts to synthesize that training data into probable continuations or responses.

If the training data doesn't include lots of text of people being asked questions and saying "I don't know", then it's unlikely to respond "I don't know" when prompted, regardless of whether anything in its training data that might actually answer your question.

Did you just make up in your head how LLMs work and then just run with it? There are a zillion explainers out there, at least put in some time before talking about it.

Humans run on training data too in the same broad sense.

> If the training data doesn't include lots of text of people being asked questions and saying "I don't know", then it's unlikely to respond "I don't know"

There's a problem with your reasoning. The above applies to a foundational model (that is to an autoregressively pretrained model). If the training data doesn't contain "I don't know" in a dialogue context, then, indeed, the model is extremely unlikely to output "I don't know" when asked. That's the nature of autoregressive training.

But we are dealing with fine-tuned, RLHF-, intruction-, RL-trained models. If the model has "grasped" a concept of knowledge, this concept can be elicited during the mentioned training.

If they're trained on the output of a larger model... Would they inherit "thinking" that they know more than they do?

It'd be interesting to see if distillation increases hallucinations for specific topics the larger LLM is confident in

Reply on news.ycombinator.com