Comment by standyro | Hacker News Reader

Comment by standyro | Hacker News Reader

Parent Full thread

This is complete technical hubris.

You're effectively saying that the "truth" problem with AI is solved by letting it guess. Which is probably a fine enough answer, but doesn't actually "solve" the problem at all.

To the points:

3) Training on data sets where the answer is never "I don't know" is effectively the same as raising children to believe they're always right and teaching them to be needlessly confident. No one should trust those folks.

4) "Lack of access to information" is not the issue... These models are trained on the entire corpus of Twitter, Wikipedia, and more information than I've ever seen in my lifetime, and some of them already do this (Bing) and produce little more than summaries of blog posts based on keywords. If anything, the issue is that an LLM lacks the real world knowledge to discern any nuance as to whether something is correct or grounded in reality.

I'm summarizing the talk, not giving opinions of my own. It is by an OpenAI researcher who is saying why they think GPT guesses so much and what they are doing about it.

I don't quite follow the rest of your post. Nobody is saying the solution to truth is to let it guess. At most, sometimes something not 100% perfect is preferable to nothing at all, but obviously only sometimes.

The point is to identify what in the training process is accidentally causing it to guess too often instead of admit when it doesn't know or is uncertain. Some of this bias comes from the nature of the data set. On the internet, people don't normally post "I don't know" as an answer to a question because that's useless and would be considered spam, but in conversation it's normal and desirable. In other cases they have QA datasets where the goal is to impart knowledge so every question has an answer, but this accidentally trains the model that questions always have answers. Human raters may accidentally reward guessing. And so on.

The talk goes in to what can be done to correct these biases.

Finally, in many cases where the models hallucinate it's because they can't look anything up. Yes they know a lot but just like a human this knowledge is compressed. So they make up references that sound plausible but don't exist for true facts, for example, because they can't check Google Scholar to find the right reference. This is exactly what you'd expect to see from a human who was forced to come up with everything off the top of their head. Think about how much programmers hate whiteboarding interviews, it's for the same reason. Giving LLMs tooling access does make a noticeably large difference.

Reply on news.ycombinator.com