You can do k-NN in the feature space of the network (activation of one of the last layers) to generate partial "explanations" of the neural network decisions.