Comment by sytelus | Hacker News Reader

Comment by sytelus | Hacker News Reader

Adversarial noise is becoming quite a crises in the deep learning field and causing lot of heated debates. There have been effort for defenses using distillation and denoised auto-encoders but only thing that has worked well is actually training on adversarial noise itself.

However the bigger insight is that deep networks aren't learning high level features. This paper has very strong evidence that deep networks are just more fancy statistical models as opposed to developing more abstract representations. When if you make network more deeper, you are just developing more higher capacity models as opposed to more higher level features. This is quite damning... If you were hoping current form of deep learning will open the door to AGI, your hopes should be shattered by now.

Some of the things to investigate are (1) can proper regularization help turn this around (for example, forcing gradients to be as small as possible) (2) are there any other architectures using gradient descent that might actually develop abstract features.

This is definitely crisis time in the field and therefore more interesting than ever :).

Adversarial noise is interesting, but I think it's more of a toy problem that will go away as we switch to more sophisticated models with life-long learning that are able to develop complicated, hierarchical/compositional classifications.

For instance, there was a talk at NIPS this year where the researcher developed virtual stickers that can be added to a photo and cause the image to be misclassified (eg you have a "banana" sticker that you can apply to any picture to make the classifier classify that picture as a banana). The underlying issue, IMO, is that the classifier has only a limited number of categories from which to choose, and can't develop new categories (such as "adversarially perturbed picture"). The modified image does look more like a banana than anything else, and as the classifier never trained on adversarial images as a separate category it doesn't really make sense to make any other classifications.

It's also interesting whether there are people with apperceptive visual agnosia who are susceptible to adversarial examples.

Agreed it is an interesting question. Humans are just far less likely to build a mono culture of vision capabilities. Such that the applicability of any adversarial examples is likely far more limited.

aren't optical illusions human adversarial examples?

It's probably fair to see them as such, but they operate in the semantic domain rather than the pixel domain.

Do they? Specifically the graffiti style illusions that appear 3d to people. Those seem to work by specifically lining up the "pixels" you see.

It's not a seemingly random (meaningless) perturbations that cause misclassification/misrecognition, but a very specific (meaningful) tuning of features that serve to trigger some misrecognition. The distinction here is whether the modified features themselves have semantic information (i.e. have a representation in the semantic domain with respect to the content of the image). In the case of graffiti illusions, it seems that it does: the misrecognition is due to a particular kind of coherence with the observers perspective and the relative positioning/alignment of the features in the image. Both perspective and relative alignment of features is meaningful.

There's no semantic reason why we see this image as moving, but everyone does, even animals.

http://24.p3k.hu/app/uploads/2014/05/Rotsnake1.gif

>There's no semantic reason

It's hard to be certain because we don't know the details of our visual system.

I think you are somewhat shifting the goalposts, though. For adversarial images, you are changing what you know the system is looking at. In such a way that confuses it to "see" something else.

For humans, we just happen to somewhat understand the image that is being used against us. How is this different?

The difficulty here is operationalizing the terms involved to make the point of semantic domain vs pixel domain meaningful at all. But I think this point makes it a substantive distinction: have a representation in the semantic domain with respect to the content of the image.

There are various semantic features of an image and we presume that our visual system is extracting and operating on these semantic features. So it seems important to ask whether the features causing the misclassification can be represented in a semantic domain of the image. Shapes, (relative) sizes, gradients, high level patterns, etc are examples of semantic domains. There seems to be a distinction between the kinds of optical illusions humans are susceptible to and the kinds CNNs are. In the graffiti illusions, the placement of the image that causes our 3D object recognition system to kick in can be described through angles, lines, focal points, focal lengths, etc. These are all semantic features of the scene depicted in an image. Contrast this with CNN adversarial perturbations which have zero semantic features that we recognize. This seems like an important result. It means that our visual system is robust against certain classes of adversarial images, namely the small imperceptible deltas that trip up CNNs. To trip up our visual system requires certain combinations of semantic input which are harder to exploit (larger delta is harder to exploit).

Then I offer the ease with which we see faces where they aren't. The very constellations?

I get your point to an extent. However, I think it is presented stronger than it is. Specifically, the semantics of imagery are not much more than non frequency analysis of the pixel data your eye sees.

more fancy statistical models

This would be true even if there were intermediate representations being learned.

Reply on news.ycombinator.com