For neural networks unless it's calibrated otherwise the confidence score has little meaning to the accuracy. [1] see figure 4 and you will understand what I mean

[1] https://arxiv.org/pdf/1912.03263.pdf