Breaking Linear Classifiers on ImageNet | Hacker News Reader

Breaking Linear Classifiers on ImageNet

karpathy.github.io

53 pointsmattxxx9 comments

I'd be interested in seeing if this same problem appears in generative models. It seems like a flaw of discriminative models.

Convolutional Boltzmann machines have been developed, and neural networks trained with Backpropagation can be pre-trained with generative models. Would this help alleviate the problem?

The original paper found that nets pretrained as autoencoders were the least susceptible to it, but still failed. IIRC the follow up paper (http://arxiv.org/abs/1412.6572) tested it or at least claimed it wasn't a solution.

I tried replicated Nguyen et. al's experiment on MNIST. I decided to do it using different classifiers and that "state of the art" was less than 5 percent 0-1 error on testing set.

I ended up with a ~98% accurate Gaussian kernel SVM classifier and my generated images all looked like numbers.

I also got ~96% using KNN, but haven't figured out how I would measure confidence in such a classifier (average distance?)

Any ideas?

Can our own visual system be fooled in a similar way, or is the response of the neurons involved too non-linear?

I think so, we just also have the capability to "double take" and pay closer attention and get a new point of view when the thing we're seeing is unexpected.

Its hard to say if a similar hack would work on people since we can't easily run an high speed optimization over the human visual cortex. Maybe some mice?

I do not think there is a one-to-one connection to be made between neural nets and the brain. That said, it is possible to fool our own visual system in a similar way: http://en.wikipedia.org/wiki/Gestalt_psychology#Properties

If you haven't seen this picture before: http://www.moillusions.com/wp-content/uploads/photos1.blogge... it should appear as meaningless noise. After you spot the Dalmation dog, you will notice it in the future.

Possibly. We can't backpropagate through all the neurons in a human brain. If we could we might be able to create real BLITs http://www.infinityplus.co.uk/stories/blit.htm .

What are the implications of this with self driving cars or biometric security devices?

Also.. In the "incomprehensible noise" pattern classes, I can pretty clearly see features that I would identify as a robin, cheetah, armadillo, and lesser panda. I don't think this means the classifiers are being fooled, they are just not being trained to recognize the class of "noisy image that looks kind of like something but is actally noise"

I think the required noise / subspaces are unlikely to be met in real world scenarios. That is, you won't take a picture of a spider where the sun just happens to shine in the right way and your lens is in just the right angle to make an image recognized as a giraf.

It may become an attack vector, but one the is very difficult to fit to the classifier you are targeting (your car may be using online learning, so there is a chance that an image generated on one self driving car won't work on another of the same model) then you have to show this carefully generated image to the car, but if you show it to the camera, it will already be distorted beyond its fooling capability. due to angle, lighting, lenses etc.

Reply on news.ycombinator.com