Unless you enforce your network to have simple understandable mechanics during training you won’t be able to properly decompose the network to a mechanistic understandable explanation. Due to the fact that it won’t work like that under the hood. Parts of the network will be simple enough to be understandable but allot of it won’t.

The human mind can not deal with and understand things that have to much dimensionality.

My 5 cents.