Couldn't you train the model to keep score (develop a heuristic) for it's own level of certainty for a given answer?
However, it seems that RLHF considerably reduces the model's calibration, so perhaps the method above won't be applicable to ChatGPT and similar.