Explainable deep learning improves human mental models of self-driving cars

From the abstract:

"Here, we introduce the Concept-Wrapper Network (CW-Net), a method for faithfully explaining the behaviour of machine-learning-based planners that causally grounds their reasoning in human- interpretable concepts without sacrificing performance. We deploy CW-Net on a real self-driving car and show that the resulting explanations improve the human driver’s mental model of the vehicle, allowing them to better predict its behaviour, particularly in surprising situations. This demonstrates that explainable deep learning integrated into self-driving cars can be both understandable and useful in a realistic deployment setting. We anticipate our method could be applied to other safety-critical systems, such as autonomous drones and robotic surgeons, as well as to other architectures, such as end-to-end learning systems and vision–language–action models. "

"CW-Net grounds the reasoning of a black-box ML planner in human-interpretable concepts, such as ‘Approaching stopped vehicle’ or ‘Close to cyclist’. This method is rooted in case-based reasoning, a classical artificial intelligence (AI) approach31,32,33,34 inspired by cognitive models of human reasoning and memory35. CW-Net can be applied to arbitrary pretrained deep neural networks, does not require retraining from scratch and does not degrade the performance of the original black-box ML planner. As the inferred concepts are the sole input to the final decision-making module of CW-Net and therefore directly determine AV behaviour, we refer to them as causally faithful. Notably, our approach contrasts with popular post hoc explanation methods36,37, which are applied to pretrained models and do not, by construction, guarantee faithfulness to the model decision process"