Auto Label can do it even with closed-source LLMs: https://www.refuel.ai/blog-posts/labeling-with-confidence
Also, your 90% number is a 'plausible-looking hallucination', which is mildly funny.
Also, your 90% number is a 'plausible-looking hallucination', which is mildly funny.
They don't show the whole prompt they use for this so I am sceptical of the comparison that recommends their service which does the token level accuracy. In my short testing you can drill down some acceptable accuracy for specific domains if you lead it over some general "ask yourself" questions then "finally, output a % confidence"
My parent comment said "LLMs are useless for classification, because they can't give confidence scores". I think i have shown that to be demonstrably false.