> The first part of this is fine, but the last sentence is not
Your following explanation suggests you have misunderstood me.
> Confidence can be extremely useful in AI. Is the “accuracy confidence” metric you’re using right every time? No.
>> They indicate how confident the model is of the result, not how likely the prediction is to be accurate.
What I said here does not mean that model confidence is not useful. It simply means that model confidence is different than the real world probability of an answer being correct. Allow me to use an example: Suppose we train a model on coin flipping. If we ask the model what the probability of flipping a heads is the model might say "51% with a confidence of 99%". The model has learned through observation and is going to be biased by good/bad "luck" in observation sampling. Of course this example is something we can solve analytically but the disagreement in the analytical solution and statistically learned solution are precisely what I'm trying to explain.
I agree, confidence is a useful metric in ML. Very useful. As someone who does generative modeling I can attest to its usefulness first hand. But models are statistical and we need to have a deep understanding of the limitations of our metrics and be careful in understanding what our metrics precisely mean. That is all I mean.