Hacker News Reader

Comment by Animats | Hacker News Reader

The underlying problem seems to be that deep neural network classifiers tend to place their classification boundary surfaces very close to data points in at least one dimension in a high-dimensional space. That makes them brittle - perturb the data very slightly in the wrong direction and they move through a boundary into some other classification.

I don't know enough about the subject to know why training does that, or what can be done about it.

View on HN

> deep neural network classifiers tend to place their classification boundary surfaces very close to data points in at least one dimension in a high-dimensional space

Doesn't this imply the Jacobian of the network is ill-conditioned near the adversarials? If so, it seems like we could test this by imposing regularization on ratio of min and max singular values of the neural network's Jacobian, and examine what effect, if any, on adversarial examples.

What's concerning is that the network used dropout, which I thought was aimed and making networks less brittle (i.e. reduce overfitting).

Go watch Yatin Gal's talk on dropout in neural networks. He shows pretty convincingly that the belief that dropout reduces network overfitting by introducing noise is wrong.

Wait, that can’t be wrong because that is literally what DO does. It is a convex hull regularizer around the network activations using noise. That is also why dropout does not solve susceptibility to adversarial examples: It merely extends the regions that the NN generalizes to outward; but that is limited because high-dimensional spaces are counter-intuitively large and the noise required to cover a descent fraction of the “unmapped” space would completely prevent learning. AFAIK, Yarin Gal merely provides a Bayesian interpretation of the noise.

IIRC, his "Bayesian interpretation of the noise" actually shows that dropout performs approximate integration over model parameters. As he says, dropout doesn't work because of the noise but despite the noise.

https://youtu.be/3ONLxYeM1Sc?t=19m21s

That seems like a strange/unnecessary way to put it because DO is noise.

I thought the point was that dropout effectively learns an ensemble of all... erm... subtopologies (if that's the right word) of your network?

You can also call it just subgraph; not all of them, but exponentially many.

It would be interesting to see what happens when these adversarial images are used while data augmentation in training

Reply on news.ycombinator.com