>> AI is not capable of explaining why it makes a decision that it does.

That's not true for "AI". It's true for a particular kind of AI system, which are collectively known as "black box" approaches. Deep neural nets for example, are a "black box" approach because they can't explain their decisions, as you say.

There are other AI approachs besides deep learning. Recently, a system based on Inductive Logic Programming, a form of machine learning for logic programs, beat 8 human champions in the card game of Bridge:

https://www.theguardian.com/technology/2022/mar/29/artificia...

In Bridge, players must be able to explain their plays to their opponent, and the AI Bridge player in the article above, Nook, was specifically designed to have this ability _and_ play better than human champions.

Btw, lest this is perpetually misunderstood:

AI ⊆ machine learning ⊆ neural networks ⊆ deep learning