I agree that a human can use the PV to get insight into why a move might be good. I also agree that non-interactive explanations provided by a chess engine will often be inferior. I think one way to break it down might be that the PV (or sets of PVs) can provide a useful explanation of tactical value (Kf3 leads to mate in 6), but struggles to explain strategic value (Kf3 leads to the positioning opening up, which broadly makes it harder for you to press your advantage).

> There is no equivalent to that in neural networks.

The first example that comes to mind is using backpropagation to determine how much influence each value had on the output result, and then rendering it in a human-readable way. Stuff like this:

https://miro.medium.com/max/4800/0*Y3Yi7cEueF0XLZP-

https://miro.medium.com/max/1100/1*IPhQ12OKnxH31AQNja6FnQ.pn...