Adversarial patch

This could be useful to make networks hallucinate all sorts of things where humans would not suspect; imagine I made stickers that made surveillance cameras see the face of the FBI’s most wanted, and stuck them on a person I wanted to harass. Or imagine I could make a “sticker” large enough to make a drone surveillance system hallucinate a tank.

There are enough humans in the loop today where this is hopefully not a serious attack vector, but this is a good thing to keep in mind when designing new systems.

This research says: defending against that is not enough. If you're allowed to vastly modify a small area of the picture, you can once again achieve misclassification.

This is a huge problem if we expect autonomous systems to do anything meaningful with vision outside of tightly controlled environments. You're going to be crucified after an autonomous vehicle merrily plows into a semi and the explanation for the crash is "well the truck was white and had the Hidden Valley logo on it which was misinterpreted as clouds." Or if an autonomous car runs a red because there was billboard somewhere behind the light which sorta-kinda looks like a green circle.

No it won't (as demonstrated in the video). This is the direct result of having CNNs. CNNs recognize pictures because of what a (at most) 32x32 set of pixels look like. It recognizes 1024 pixel patches as being toaster or not.

So no, a toaster for an image classifier is VERY different from a toaster. It's how the corners of the surface work. It's the grid within the top. It's the reflecting aluminum near the plastic border. It's the ...

A sticker doesn't have the vast majority of those properties. So a sticker with a toaster doesn't actually (usually) cause a misclassification (one imagines you can of course make the sticker good enough, but ...).

But you can collect a lot of those 32x32 patches that are really, really "toastery" and then scale them, find something that is "somewhat fractal" (looks the same at different scales) and put that on a sticker. That looks nothing at all like a toaster to you, but it looks a hell of a lot like a toaster to a CNN.

If you learn CNNs by knowing their predecessor "trick" (google "haar cascade") this makes a lot more sense.

The reason we aren't fooled* is that our eyes don't recognize pixels. We recognize a wavelet encoding of a picture, not a grid of pixels. We recognize something that's like this description "a red blob, with a smaller green blob in the 2 o'clock position, a blue blob near the bottom, then that green blob is actually split into light green on the left and dark green on the right, and that blue blob really has a white top and a black bottom and ..." (you see you give details about a large average then afterwards describe the parts where that average is wrong. Rather than describing a grid top-left to bottom right you describe it as "average of the whole grid X, average of top left Y, average of top right B, average of bottom left Z, average of bottom right A, then recurse).

* truth is you can fool humans partially with these stickers. You can make stickers that clearly are plastic stickers but you strongly feel that if you tough them you'll feel a cat's fur, or an insect/spider, or ... despite looking nothing like those things, and of course you can tell they're stickers). And of course, there's Dali.

This also reminds me of using makeup and hair to prevent automatic facial recognition. I believe this is the project: https://cvdazzle.com/

> The name is derived from a type of World War I naval camouflage called Dazzle, which used cubist-inspired designs to break apart the visual continuity of a battleship and conceal its orientation and size. Likewise, CV Dazzle uses avant-garde hairstyling and makeup designs to break apart the continuity of a face. Since facial-recognition algorithms rely on the identification and spatial relationship of key facial features, like symmetry and tonal contours, one can block detection by creating an “anti-face”.

> In this paper, we focus on facial biometric systems, which are widely used in surveillance and access control. We define and investigate a novel class of attacks: attacks that are physically realizable and inconspicuous, and allow an attacker to evade recognition or impersonate another individual. We develop a systematic method to automatically generate such attacks, which are realized through printing a pair of eyeglass frames. When worn by the attacker whose image is supplied to a state-of-the-art face-recognition algorithm, the eyeglasses allow her to evade being recognized or to impersonate another individual. Our investigation focuses on white-box face-recognition systems, but we also demonstrate how similar techniques can be used in black-box scenarios, as well as to avoid face detection.

https://en.wikipedia.org/wiki/Eyespot_(mimicry)