> Calculators and computers are deterministic

How do you know?

Memory bits flip randomly. It's not a super rare thing either. You and me have experienced that many times without knowing. The only reason that computers feel deterministic is that we have error-correcting code to fix that. But in the most extreme cases, when multiple bits flip together, once "deterministic" program can generate unexpected output.

So why do you trust computers? Because statistically the case is just very unlikely. Therefore if AI is statistically unlikely to make mistakes there is no reason to not trust them.