Hacker News Reader

>Explicitly prompting the LLM to output a confidence score is the least accurate

They don't show the whole prompt they use for this so I am sceptical of the comparison that recommends their service which does the token level accuracy. In my short testing you can drill down some acceptable accuracy for specific domains if you lead it over some general "ask yourself" questions then "finally, output a % confidence"

View on HN

I couldn't care less about the specific implementation.

My parent comment said "LLMs are useless for classification, because they can't give confidence scores". I think i have shown that to be demonstrably false.

My understanding is these are more “confidence in the tokens selected being appropriate” rather than the more fundamental “confidence the classification chosen is accurate given training data,” where training data is samples of the distribution being classified rather than language corpus. They seem similar at first blush but the first is more about appropriateness of phrasing and how esoteric the semantics are, and the latter more descriptive of some statistically observable process - which is something LLMs are literally incapable of reasoning about directly.

Reply on news.ycombinator.com