Is it possible for the current generation of LLMs to assign confidence intervals to their responses?
That's my main qualm with ChatGPT so far: sometimes it will give you an answer, but it will be confidently incorrect.
> GPT-4 can also be confidently wrong in its predictions, not taking care to double-check work when it’s likely to make a mistake. Interestingly, the pre-trained model is highly calibrated (its predicted confidence in an answer generally matches the probability of being correct). However, after the post-training process, the calibration is reduced (Figure 8).
pages 10-11: https://cdn.openai.com/papers/gpt-4.pdf
Take this with a grain of salt though, I'm far from an expert, and it's been a while since I've played around with that feature.