SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationales

arxiv.org

28 points·jasondavies··9 comments

Excellent background on knowledge calibration from Anthropic:

"Calibration" in a knowledge context means having estimated_p(correct) ~ p(correct), and it turns out that LLMs are reasonably good at this. Also a core reason why LLM-as-a-judge works so well: quality evaluation is vastly easier than generation.

This is one of the key prompting in a lot of Enterprise cases. You can currently prompt LLMs to add a confidence score along with their responses.

Especially when you are using LLMs for downstream NLP tasks.

The confidence score can be a great indicator also for applying a two-tier model approach!

yeah but they say in the paper that isn't very accurate, they suggest a specifically fine-tuned model for it, not just prompting it for a score?

In my experience self validation performance and confidence ratings are both poor. I think the problem is that these sorts of formats just aren’t that common in the training data. What does help is to as a series of structured questions pertaining to quality and to aggregate those, but it’s still often not that helpful.

The challenge is the confidence scores can often be confabulations themselves.

What is a confidence score like that based on?

I admit I'm a bit confused by the reward function, as given it seems to provide the same score independent of correctness due to the squaring? And I think even if that's a mistake and it's supposed to be negative for incorrect answers, a policy that optimizes for that reward is to output 1 for anything with less than a 50% chance of being true and 10 for anything over 50%. Is that how RL is typically done?

it is nice that you posted datasets

Reply on news.ycombinator.com