I view explainability or interpretability of a network as the ability to take a network and replace it with a drastically smaller set of functions and tables that (a) you can explain and (b) work pretty much the same as the network does.

Because we understand these functions and tables, we understand exactly how well the network will work, and also what is missing (i.e., how we can expand its accuracy.)

I think this is a very hard problem, but it is one that needs to be solved.