Hacker News Reader

Comment by taneq | Hacker News Reader

When discussing adversarial attacks on neural nets, everyone seems to just kind of take for granted that the human visual system is immune to this kind of fudgery. I'd be very interested (and a little afraid) to know if this is in fact the case - we know of a bunch of optical illusions that are perceived differently to what they "ought to be", so it's not too much of a stretch to figure that we could come up with a technique to generate more extreme examples.

View on HN

It’s still conjecture but the chances are good that photosensitive epilepsy[0] as triggered for example by the famous Pokemon episode[1] is conceptually similar to the unstable responses that adversarial attacks produce in computational neural networks (and yes while there is some debate about the total number of seizures triggered by the broadcasting of that episode there is no debate as to whether the medical condition exists and is an real risk factor for a surprisingly high fraction of the population, particularly males and children[2])

[0]https://en.wikipedia.org/wiki/Photosensitive_epilepsy

[1]https://en.wikipedia.org/wiki/Dennō_Senshi_Porygon

[2]https://www.epilepsysociety.org.uk/photosensitive-epilepsy

This is a bit of a stretch.

Beyond the fact that unexpected inputs produce unwanted outputs in both cases, I'm not sure I see the connection.

In the DNN case, the network should be able to end up in both states. However, the boundary is fairly sharp and small changes in the input not only push it over the boundary, but make give it high confidence that the new state is the correct one.

The brain, however, shouldn't ever end up in an epileptic state; there are feedback mechanisms that decorrelate neural activity and keep excitation in a certain range. These mechanisms are weakened or defective in epilepsy, which allows very potent stimuli to drive neural activity and kick off positive feedback loops.

The vast majority of the adversarial work on DNN’s is about using small input perturbations to push the system from one stable output state to a different stable output state, mostly because work is mostly done on images rather than video so repeatedly re-classifying the same image isn’t very interesting. That doesn’t mean stable state to stable state is the only form of adversarial input that can be found, it just means that’s all that has been looked for so far.

The odds that there are adversarial perturbations that reliably push the system into an unstable state between two or more output states is probably quite high. It just hasnt hit the threshold at which it becomes interesting for postdocs to look for yet.

Having read some very introductory stuff about human visual system, I'd be willing to guess that a partial reason is that the human vision works on the input level quite differently than "observe NxM colored pixels in grid". The act of constructing what we'd think as a single "view" of our surroundings (that one intuitively might equate with a still pixel image) includes lots of low-level processing.

(E.g. Everyone probably has heard about the blind spot at the point where optic nerve 'connects' to retina. But I was surprised to learn that most of the stuff in our FOV is seen very poorly, and only the tiny area called macula is responsible for the high acuity vision. And the macula is capable of covering only a small spot in our FOV at one time: what we'd think as a single view is really constructed in a post-processing step as our eyes move and scan our surroundings. My amateur-level explanation might be a bit off, but the point is, it stands to reason that kind of "dynamic" processing system is more robust (especially to small pixel level "attacks" as in the linked paper) than simply constructing mappings of still pixel grids to outputs, as currently done by CNNs.)

What we know about the brain points to lots and lots of top-down influence on visual processing. Your learned preconceptions can go down the stack and alter/override raw sensory data that gets fed from your eyes up the stack, forming loops that are arguably absent in ANN solutions.

Not only that but much of what you ‘see’ is reconstructed from memory not active input from the eyes.

The mechanisms of vision are really fascinating. Ever notice that when you first glance at a clock, that first tick of the second hand/digits seems to take too long?

I think people get fooled and do small mistakes all the time but we easily ignore them because it's corrected by other senses. Like you thought your phone screen turned itself on in your side vision but you just left your credit card or something on top of the screen.

You can cover someones eyes obstruct visual system entirely, and you'll be fooled for a second until you realize it's probably just your friend because you can feel their hands. Did you fool your friend's visual system? Or does it need to be fooled forever to say you fooled the visual system?

So the visual system on its own is not enough as we have many other senses that you'd need to fool in order to fool you entirely.

I think optical illusions is a great way to fool the visual system only, but it doesn't fool you. (all your senses) There are magic tricks for that sort of thing, but even then we know it's just a trick.

As raverbashing pointed out our visual system is more than just a 2d grid of points with a certain color, so obviously that is easier to fool.

When I saw the 32x32 image of the frog with the pixels I wasn't even sure what it was for a few seconds. But I had context, like I know these images are probably things that are easily classified, so it's gotta be some common animal, airplane, car, house, etc. I also knew that I was supposed to ignore the pixels.

If you present the frog image with pixels to people on the street and tell them to classify the image, I'm sure many of them will get it wrong if they are given less than 2 seconds to look at it. More so when they're focused on something completely irrelevant.

> take for granted that the human visual system is immune to this kind of fudgery

Clearly it isn't, but given that animals use camoflage, the visual system is the end product of millions of years of evolutionary arms race against confounding inputs.

What are you trying to say? What insight do these assertions give us on the human visual system?

It's simpler than that

The human vision has a natural blur (some more than others), natural temporal integration of inputs, automatic centering of subjects, limited resolution and gain adjustment

Hence one pixel nudged makes absolutely no difference

All the things you've said, bar temporal integration, are also present in CNNs in some way. Training data is "augmented" with different levels of luminosity to cancel the affect that has on classification, pooling layers give some translation invariance (and avg pooling is a type of blur), and resolution is also very limited in most models (and gets smaller as you get deeper). And it still fails!

Ah yes, but I mean there are two types of temporal integration

One is the short-timed "persistence of vision" one

The other is observing a scene multiple times by slightly different angles and positions

If some weird arrangement causes an illusion, changing the position slightly usually fixes that

Also, human data is noisy, I guess augmentation strategies might want to consider that as well

So here's an idea to test that. Take an image and find one of these adversary generating pixels. Now take that same corresponding pixel and modify it in each of the other images in exactly the same way that generated the first adversary. I would not be surprised if it is not an adversary generator across all images. I would expect each of those images has a different gradient.

I've often thought that DNN vision systems would be much more robust if they worked on video streams rather than single-image snapshots, for just this reason. It would let them implement something like the "liveness tests" the new iPhone apparently applies to the facial recognition unlock stuff.

It's definitely true that segmentation is a lot easier to do for video than images, in part because you have access to more information (e.g., motion cues from an object moving independently from the camera).

So 2 pixels or 4. Lower resolution is no protection, it just means the pixels are larger.

I think the difference is that people know when the image messes with their perception. i.e. we see the optical illusion, yes, but we also know it's not real, or incorrect in some way.

A neutral net does not. That's the missing element - neural nets "need to know how to know when they don't know".

I don't think so. Many of my friends thought I was trolling them when I told them "the actual color" of "the dress" [1].

When we do know that there is an optical illusion, it is because of the context we have that the machines don't, like, we can read the title of the article ("These 11 optical illusions will blow your mind").

[1]: https://www.google.de/search?q=the+dress

Our eyes are much, much better at handling color balance "in person" than on a screen. Normally, we have the entire surrounding environment to compare colors to and make a guess.

This is what I'm saying: We have more context. There is theoretically nothing keeping machines doing the same kinds of analyses when given the same information.

Actually in this sense NNs are actually smarter than humans - NNs can guide you to its weak points, through backprop. That is indeed how adversarial examples are found. Vice versa, people are notoriously bad at finding their own weaknesses (as demonstrated by Dunning-Kruger etc).

Dunning-kruger doesn't mean the popular misconception you are thinking of

Our visual system has all kinds of "attack vectors" of this sort. Heck, video only works as a medium because it exploits one if them.

For more a much more subtle and in some ways (imo) disturbing one, look into the blindness constantly induced by our visual perception during saccades. A malicious attack on that would be... well I don't really have a word for it. Disturbing doesn't really fit. Suggestions welcome!

Peter Watts' novels have a couple of examples of this. For instance, (SPOILERS!) the scramblers in Blindsight can render themselves nigh-invisible by moving only during human observers' saccades. (I'm not convinced that this would actually work but it made for a cool plot device.)

http://www.scp-wiki.net/scp-372

The human visual system is susceptible to misperception, sure. This has been known for awhile. I also would agree that sometimes single pixels could probably confuse people substantially.

However, if you look at the images in the article, I think it's telling about how much more sophisticated human perception is. The authors even have a note in the first image along the lines of "look carefully, because you might have trouble locating the aberrant pixel."

I think these types of attacks speak volumes about the fragility of DL optimizations: I think there's more overfitting going on than people acknowledge or realize. My sense from reading in working with some ML things is that this extends to natural language data as well.

DL systems are often highly optimized to a particular test set, and might do well on cross-validation to other exemplars of that test set, but that's not the same as generalizing across different types of test sets.

Maybe I'm wrong, though--I think DL NN models have been fantastic, but there's a certain amount of hype in ML in general.

My two cents: I'd think that it probably would be, if you could freeze its physical state and repeatedly apply it to single static images and measure it's response and do gradient descend on that.

But in reality the brain is stateful and noisy and works on a stream of images. Even if now and then, by sheer accident, for a short time, your brain might be mislead, it surely wouldn't go like "well I was pretty sure that was an X, but just now Y is scoring higher than X, so I'll just flip my opinion on that in an instant".

It’s worth noting that 1 pixel on a 32x32 pixel image is a much higher proportion than it may appear at first too. That makes the analogy to optical illusions much more reasonable.

I'm not sure if that was a real optical illusion rather than a massive amount of miscommunication. I.e. some people taking about the colours of the actual pixels in the image and some people taking about the likely colour of the dress given the cameras respond to the living conditions. There different of course but everyone just said "the colour of the dress is..." resulting in pointless arguments about different things.

I don't think most people who argued about that even realized those two are different things - and indeed the inability to make that distinction was one of the causes of the controversy.

It's an easy to reproduce optical illusion, using computers instead of real world lighting. Xkcd did a nice illustration.

I don't think it is immune. Like with neural nets, human sensory processing is dependent on previous input data to generate generalized models of sensory input to prioritize and cluster data so that we can understand how things are similar. In psychology, this phenomena goes by different names depending on the circumstances; sensory priming and sensory habituation are two fairly well documented examples of sense/memory integration.

Obviously, humans aren't going to have to have this amount of error from small, small changes like these (except for maybe in very autistic people who have dramatic reactions to changes in their environment, but I may also be wrong; I'm not autistic, and I'm not a psychologist), but our innate perceptual biases do play heavily into how stimuli are processed and reacted to.

People do generate optical illusions of the time the red green gold dress is a modern example.

Have you read snow crash?

http://www.infinityplus.co.uk/stories/blit.htm

> This first example of the Berryman Logical Image Technique (hence the usual acronym BLIT) evolved from AI work at the Cambridge IV supercomputer facility, now discontinued. V.Berryman and C.M.Turner [3] hypothesized that pattern-recognition programs of sufficient complexity might be vulnerable to "Gödelian shock input" in the form of data incompatible with internal representation. Berryman went further and suggested that the existence of such a potential input was a logical necessity ...

TL;DR https://en.wikipedia.org/wiki/BLIT_%28short_story%29

https://news.ycombinator.com/item?id=9736942

Lots of people make the equally simplistic argument the other way - that because the human visual system can sometimes be fooled, then this is not an issue.

As other people have mentioned, human vision is quite robust. But there are other ways to hack the human psyche... For instance, the abuse of selective attention:

https://m.youtube.com/watch?feature=youtu.be&v=Ahg6qcgoay4

The underline assumption is that the human visual system can be regarded as continuous – small changes in input cause small changes in output.

But we know neural nets are not continuous: a tiny change in input could cause a huge change in output as suggested here.

There's an FAQ about these things: http://ansible.uk/writing/c-b-faq.html

But what does it take to fool the human visual system? I'm guessing it's generally more than a few pixels.

Reply on news.ycombinator.com