'Explainable Artificial Intelligence': Cracking Open the Black Box of AI

There is an interesting experiment you can do to directly demonstrate this to yourself

From Julian Jaynes' The Origin of Consciousness in the Breakdown of the Bicameral Mind

> Take any two unequal objects, such as a pen and pencil or two unequally filled glasses of water, and place them on the desk in front of you. Then, partly closing your eyes to increase your attention to the task, pick up each one with the thumb and forefinger and judge which is heavier. Now introspect on everything you are doing. You will find yourself conscious of the feel of the objects against the skin of your fingers, conscious of the slight downward pressure as you feel the weight of each, conscious of any protuberances on the sides of the objects, and so forth. And now the actual judging of which is heavier. Where is that? Lo! the very act of judgment that one object is heavier than the other is not conscious. It is somehow just given to you by your nervous system. If we call that process of judgment thinking, we are finding that such thinking is not conscious at all. A simple experiment, yes, but extremely important. It demolishes at once the entire tradition that such thought processes are the structure of the conscious mind.

Saying that humans make mistakes and cannot explain their decisions is, in fact, one of the very reasons we want to have better models. We hope they will do better than most people to create a better world. And their explanations will hopefully provide insight into how we as people make good (and bad) decisions.

And sure, we can change the laws over time, but having advanced models which can allow humans to understand decisions and even provide diagnostics to improve the models will be transformative. Until then, we will see massive impact in some areas of our lives, and frustrating holdbacks in others, driven either by the distraction of building for regulatory constraints or by choosing not to build in regulated areas at all.

Well, your categorization judgement is probabilistic. Some features make "dogness" more likely, some less likely. However, there is in fact a specific causal structure which constitutes dogness (ie: organism grown via natural reproduction along the canid species-history, with the accompanying genotype and within normal phenotypical variation). Your brain tries to learn a causal model which approximates the correct causal structure, given the evidence available to your senses.

Better: the human brain also seems to perform second-order inference, reasoning about which models predict most precisely, what's relevant to what, and which latent parameters in the learned causal model are most relevant.

So when you see a dog-like tail versus a cat-like tail, you can identify which apparent features are making you say "dog" or "cat" most strongly, which ones are the strongest evidence.

You could train a NN to do either, given lots of training data, but before risking the lives of special forces, and potential for collateral damage, you might want to question the AI as to why they think what they found was a hidden cache of weapons or roadside IED.

I agree the statement "humans are surprisingly good at explaining their decisions" is dubious at best. Most explanations are a matter of what's called reconstructive cognition, which is far from accurate in explaining the reasons for an action.

I also worked with the data science guys at a telecom in a past startup and they were so used to using white-box models that the thought of not having a decision tree or other means of explaining how the results came about scared them, just because they weren't used to it and they could check it for correctness etc. Not having that was scary to them.

AI for high-consequence decision-making is going to need the automatic feature extraction and robustness of NNs coupled with the explanatory transparency of expert systems. That's been the holy grail for almost 40 years.