In a way you are right, but on the other hand there are two parts of explainability: one is the model explanation, and one is the class explanation. Often times the explanation of the class is what people really need (I classified this as a Husky since there is this dog area and snow area on the image). This doesn’t help in audits if ML models (where we want to understand why the model attended to these areas). On the other hand: are humans really able to explain their inner workings of the brain, or aren’t we as well describing what kind of input features motivated our decisions?