You're neglecting the emergent effect of neural networks. Fully understanding the output of one or more individual neurons hardly tells you anything about how the network behaves has a whole. From that standpoint, neural networks are nearly impenetrable. With human written software you can at least trace through the source code and get an idea of what it's doing. Imagine being handed a large matrix of numbers, being told it represents a neural net, and then being asked to find out what it does. Good luck!