_Sometimes_, and people do find features in neural networks by tweaking stuff and seeing how the neurons activate, but in general, no. Any given weight or layer or perceptron or whatever can be reused for multiple purposes and it's extremely difficult to say "this is responsible for that", and if you do find parts of the network responsible for a particular task, you don't know if it's _also_ responsible for something else. Whereas with a decision tree it's pretty simple to trace causality and tweak things without changing unrelated parts of the tree. Changing weights in a neural network leads to unpredictable results.

LIME (local linear approximation basically) is one popular technique to do so. Still has flaws (such as not being close to a decision boundary).

You are right and IDK why you are downvoted. Few units of perceptrons, few nodes in a decision tree, few of anything - they are "interpretable". Billions of the sames - are not interpretable any more. This b/c our understanding of "interpretable" is "an array of symbols that can fit a page or a white board". But there is no reason to think that all the rules of our world would be such that they can be expressed that way. Some maybe, others maybe not. Interpretable is another platitudinous term that seems appealing at 1st sight, only to be found to not be that great after all. We humans are not interpretable, we can't explain how we come up with the actions we take, yet we don't say "now don't move, do nothing, until you are interpretable". So - much ado about little.

(IMO) to a lesser extent.