Hacker News Reader

Comment by masswerk | Hacker News Reader

Something which should happen: AI productions must be allowed to express doubt, levels of low confidence and the like.

In a (hypothetical) 1960s SciFi film a knowledge system may have responded like this:

  > assumption: subject deceased
  > confidence score: 0.0000020017
  > sources: none

In contrast to this, current systems are tailored to respond with high confidence:

  > I'm sure X died in 2018.
  > It has been written about in the New York Times, The Guardian, Le Monde.
  > Here is a (hallucinated) link: https://www.theguardian.com/obituaries/x-obituary

(Compare various related stories.)

View on HN

I love this because I want more emphasis on this between humans. I hear there are languages with built in confidence expression.

Yep, I'm not a language expert but I remember a South American language, maybe related to Quechua, where a sentences contain a syllable that indicates "I am repeating hearsay, I don't know if this is true". Pretty cool.

This would be an unexpected outcome: AI productions motivating us to generally move towards prefix/postfix/infix decorations.

"X studied at uhm-Oxford." -> I'm not sure, if it was really Oxford, but I have picked this up somewhere. But I'm sure, X did go to university. (As opposed to, "X uhm-studied at uhm-Oxford": I think, X has some higher education and it may have well been at Oxford.) :-)

I wonder if you can simply ask ChatGPT to provide a confidence score with each answer it gives.

ChatGPT (gpt3.5-turbo) is terrible at calculating anything. I've seen some preliminary evidence that GPT 4.0 is better at calculation so it may be possible for it.

My understanding was that ChatGPT simply puts a probability distribution over the next word, so I don't see why it's not as simple as just reporting how high those probabilities were for the answer it gave, relative to whatever would be typical.

Those values are probably not intelligible as confidence scores. For example if it answers a question with "They died in 1902", since there are a lot of euphamisms and rephrasings of 'died' it will get a relatively low probability. 1902 probably gets a high score, but you can't really rely on that since it might just as well be hallucinating and pulled the year from some famous event in that person's life.

The GPT-4 paper & post[1] describe that the original model is pretty good at predicting the probability of its own correctness (well-calibrated confidence) but the post-processing degrades this property:

> GPT-4 can also be confidently wrong in its predictions, not taking care to double-check work when it’s likely to make a mistake. Interestingly, the base pre-trained model is highly calibrated (its predicted confidence in an answer generally matches the probability of being correct). However, through our current post-training process, the calibration is reduced.

> Left: Calibration plot of the pre-trained GPT-4 model on an MMLU subset. The model’s confidence in its prediction closely matches the probability of being correct. The dotted diagonal line represents perfect calibration. Right: Calibration plot of post-trained PPO GPT-4 model on the same MMLU subset. Our current process hurts the calibration quite a bit.

[1] https://openai.com/research/gpt-4#:~:text=GPT%2D4%20can%20al...

Reply on news.ycombinator.com