I don't see that as a major problem. In fact most life-changing decisions are already based on probability. Insurance companies already do risk analysis and whether the algorithm uses ML or basic statistics, there is a threshold level of confidence used.

I'd also argue it's how our brains work. Many times as we come to a decision we are going off of confidence, not true correctness. I'm the case of declaring suspicious person's, well by definition they are suspects based on confidence, not by truth. Even in court we determine verdicts based on human probabilistic confidence that comes from the evidence.