Comment by famouswaffles | Hacker News Reader

Comment by famouswaffles | Hacker News Reader

Parent Full thread

Because they don't care to. No seriously that's it. Lots of evidence indicating language models know when they hallucinate.

GPT-4 logits calibration pre RLHF - [https://imgur.com/a/3gYel9r](https://imgur.com/a/3gYel9r)

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - [https://arxiv.org/abs/2305.14975](https://arxiv.org/abs/2305...

Teaching Models to Express Their Uncertainty in Words - [https://arxiv.org/abs/2205.14334](https://arxiv.org/abs/2205...

Language Models (Mostly) Know What They Know - [https://arxiv.org/abs/2207.05221](https://arxiv.org/abs/2207...

TLDR; Hallucinations aren't a randomness or representation issue. The computation already knows. It just doesn't care about telling you this.

"It just doesn't care about telling you this."

But why? Why would the LLM creators want the LLM giving out knowingly bad information? This is what I don't understand.

It makes me not want to use the product. Same as I don't like asking people things who I know like to just spew bad information when they don't know rather than simply saying they don't know.

LLM creators train the model to predict the next token. They have no control over what it learns in the process. We're not programming anything into artificial neural networks so control or direction is limited.

It sure seems like they are programming things into the LLMs with how it often states "As a LLM I can't do that" sort of responses.

So how do we get LLMs to admit when they don't know something rather than confabulate? Seems like that would be a really great improvement to me.

No that's just a training process that goes, "people like this sort of responses better, try to emulate it" by feeding specific input into the training process.

>So how do we get LLMs to admit when they don't know something rather than confabulate?

The prompts here demonstrably help https://arxiv.org/abs/2305.14975

But otherwise nobody is certain how to do it. The simplest solution is just making them better. Hallucination is something that happens when knowledge/memory fail. By making them more competent, they hallucinate less. So even if they never care about communicating this, you wouldn't notice.

Reply on news.ycombinator.com