Thanks for the thoughtful response. What would this look like in something more complicated? In deep learning would this be something like “if you can get these neurons to fire more you can get a better output” and “you can increase these neurons firing by changing the inputs in this way”?

Also, I feel like this sounds a lot like generative AI, and explainability might not be necessary. Or maybe one could say that in the real world people want to have better assurance that things they do are based on a good underlying model. Images and text are easier to evaluate than predictive, speculative decisions about housing markets.