Interestingly, you can treat the NN as a black box (ie, not look at individual weights or even the architecture) and still derive adversarial cases:

https://arxiv.org/abs/1602.02697