Robust Physical-World Attacks on Machine Learning Models | Hacker News Reader

Robust Physical-World Attacks on Machine Learning Models

arxiv.org

97 pointsearlenceferns14 comments

This paper is a fairly convincing counter-argument to another recent work (https://arxiv.org/abs/1707.03501) on physical adversarial examples for autonomous vehicles. The other paper argued there was "no need to worry" about such physical attacks.

It suffers from the same weakness as most of the other papers I've seen about physical attacks where they interpret a change in most likely class as evidence that this will work for a real classifier. They say "We observe that all such baseline images lead to correct classification in all experiments." and then state that the average predicted target class probability was 0.8 +- 0.1 which suggests to me that they are taking a network which doesn't really know whether the sign is a stop sign but guesses that it is and tricking it into not really knowing what the sign is but guessing that it is a speed limit sign. Presumably a real world system would have much more confidence in the true class for the clean images.

In Design of Future Things, ex-Apple UX designer Don Norman interviews (anonymous) engineers of a self-driving car:

DN: "It sped up when we left the freeway instead of slowing down on the exit, that was dangerous. Why did that happen?"

ENG: "Car saw it was on a straightaway. We'll add a rule to handle that."

DN: "So you have to add rules for every possible situation? Doesn't that mean that the car is always at risk for what it doesn't know about yet?"

ENG: "That's Not-A-Problem. We will classify everything."

That's an incredibly stupid answer. It is precisely this kind of thinking that makes me worry about sharing the road with alpha grade self driving hardware. There is real potential for carnage here, at highway speeds it doesn't take that big of a software bug to get a lot of people killed.

So, as a preliminary, you can conduct these kind of attacks against humans as well. It just often isn't as subtle a change to the sign.

Among other things, though, these types of attacks make a lot of assumptions. For example, they assume the only input to "what is that" is a classifier that looks at the image.

Given simple data and previous classification of the image, for example, one can easily determine "hey does it make sense for an added lane sign to appear at a 4 way intersection with no apparent added lane".

Heck, you don't even need to go that far. Given previous, before vandalism classification of the sign, and no change in any terrain/mapping data, ...

So yes, i'd pretty much say "there is no need to worry about such physical attacks", as long as you are not directly hooking up an image classifier to the steering wheel. The likelihood that this ends up a major problem for self driving cars seems pretty low.

Optical illusions come to mind.

i understand why we'd like to have near-perfect classification but do we need 100% accuracy in practice?

for example, the paper identifies an instance where a stop sign was misclassified as a speed limit under various conditions.

however, wouldnt applying bayesian priors ("how often do i see speed limits at intersections?") presumably negate misclassifications?

rather than chase the elusive ~100% accuracy from a single model, why not take a layered, ensemble approach?

I think the traffic sign example was a nice setting for explaining the research. But of course really good examples always risk interfering with communication as much as they assist communication. I think that might've happened here.

Concretely, I have a hard time believing that adversarial traffic signs will be the Achilles heel of self-driving cars. Vandalizing signs is already illegal, and if doing so causes or was intended to cause a crash, law enforcement will throw the book at the culprit. And this already happens to human drivers. See e.g., http://globegazette.com/news/local/prank-turns-to-peril-vand... or http://articles.latimes.com/1997-06-11/news/mn-2303_1_one-st...

Besides which, you can also just rip the sign out of the ground or replace the stop sign with an actual speed limit sign. From a SDC perspective, this is kind of the equivalent of using super clever software vulnerabilities when you have physical access to an un-encrypted HDD. Impressive technical tour de force, but misses the forest for the trees if your goal is security.

Traffic sign vandalism is a social problem with a social solution. The first person who tries something like this in production with malicious intent toward SDCs will be charged with some variation of attempted or actual manslaughter/murder and opponents of SDCs will switch to strategies other than domestic terrorism to advance their agenda.

This research is still very interesting and useful, of course, but primarily for other reasons. I just don't see traffic sign vandalism intended to confuse DNNs as a serious -- or unique -- threat to self-driving cars.

I think there is a point here that is being overlooked. The vandalism, a couple of small stickers, would be less detectable compared with a whole missing sign.

I can see such attack methods evolving to produce undetectetable (to humans without software) adversarial images.

I'm not saying we shouldn't practice defense in depth where it's possible.

It's just that I don't think preventing this style of attack is actually the most relevant/impactful application of this style of research.

Again, I'm not saying this research is irrelevant. I'm saying it's way more relevant than this particular example implies.

> would be less detectable compared with a whole missing sign.

This kind of assumes that the attack wouldn't work in the first place, no? If cars are crashing people are going to investigate to figure out why, and even the most thick-skulled beat cop can figure out that stop signs with rando weird ass stickers on them are probably worth running up the chain...

> This kind of assumes that the attack wouldn't work in the first place, no? If cars are crashing people are going to investigate to figure out why, and even the most thick-skulled beat cop can figure out that stop signs with rando weird ass stickers on them are probably worth running up the chain...

1. That attack may not target only self-driving cars.

2. Imagine someone driving around with some random picture on their car at the back. Except that the picture is an attack.

> Traffic sign vandalism is a social problem with a social solution. The first person who tries something like this in production with malicious intent toward SDCs will be charged with some variation of attempted or actual manslaughter/murder and opponents of SDCs will switch to strategies other than domestic terrorism to advance their agenda.

It's tunnelvision to assume that the only attackers would be individual radical luddites. There do exist terrorist organizations and state actors who intend to run risks to do damage. And unless you're planning to monitor and respond quickly to all potential threats to all stop signs at all times (and all other possible attacks), what's to stop a group of 40 terrorists from covering their faces, stealing motorcycles, and putting 10,000 stickers on 10,000 signs in 10,000 intersections in 20 different cities in 1 night?

What's to stop 40 terrorists from covering their faces, stealing motorcycles, and clipping brake cables (whatever you get the point...) on 10,000 cars parked along 10,000 streets in 20 different cities in 1 night?

...Or just blowing up a shitload of random houses or whatever.

Basically, the exact same thing as what would stop your scenario. Is it perfect? Even good? Again, no. But striving for good security in a public setting is how you get TSA.

If there are 40 committed and sophisticated terrorists willing to act, there are a lot of easier targets...

Again, I think the "clever localhost software vuln" analogy is apt. It's not that this isn't one possible attack vector. And no attack vector should be ignored. But... really?

A bigger concern in my mind is where 100% accuracy is needed, like when doing authentication (cf. any login screen that lets you in by looking at your face).

A lot of experiments are done on visual data sets, but I don't see why these techniques need to be strictly for visual inputs. Is a machine learning model making decisions based on the results of its classifier? Then it is at risk of being fooled, and whatever decisions it's making, better be trivial ones.

I wrote a document a few months ago that summarizes this and other risks that ML models currently face: https://matt.life/papers/security_privacy_neural_networks.pd...

Reply on news.ycombinator.com