is definitely too strong of a claim and directly undercut by what is said near the end of the post: > "Tian et al. found in Just Ask for Calibration that with the right prompting strategy, RLHF’d models verbalize probabilities that are better calibrated than the model’s own conditional probabilities, and that prompting plus temperature scaling can cut expected calibration error by more than half. And Anthropic’s Language Models (Mostly) Know What They Know found encouraging results asking models to estimate the probability that their own proposed answer is true."
My own experience is that stated confidence is a helpful tool and of course you need a rubric and a proper prompt, but this is clearly less work than training a classifier (as advocated by the post) and requires less data.