If I needed large numbers added together, I would trust a human who can explain their general addition algorithm to me, and I wouldn't trust an AI that can spit out some usually correct answers on small problems.
The explicability of our decision making is like 95% of our progress. It is why we can identify biases evolution builds into us but we need to fight against. Or try and justify something beyond mere intuition. This is like asking how much math are humans using anyway.
Humans as a group has proven to be able to build everything we have today. No AI has proven to do anything similar, there just isn't much data there to make us confident in their abilities thus far.
Conversely though, how many people are killed in the world every single day simply because of human error?
When (in the US) a 16 year old gets their license, we don't ask them to provide a formal proof of their driving technique that shows it's impossible for them to ever get in a crash. We say 'you've had the training, you've demonstrated that you can safely drive, be careful out there'.
We have a good understanding of this risk and have collectively chosen to accept it (while many countries have chosen not to accept it in the case of 16-year-olds.)
You can question a human in an exam and ask to "show your work". If you can say solve an equation but can't explain why you did this or that transformation you'll be rightly failed. With current NNs you get an answer and that's it, there's no introspection.