for example, the paper identifies an instance where a stop sign was misclassified as a speed limit under various conditions.
however, wouldnt applying bayesian priors ("how often do i see speed limits at intersections?") presumably negate misclassifications?
rather than chase the elusive ~100% accuracy from a single model, why not take a layered, ensemble approach?
Concretely, I have a hard time believing that adversarial traffic signs will be the Achilles heel of self-driving cars. Vandalizing signs is already illegal, and if doing so causes or was intended to cause a crash, law enforcement will throw the book at the culprit. And this already happens to human drivers. See e.g., http://globegazette.com/news/local/prank-turns-to-peril-vand... or http://articles.latimes.com/1997-06-11/news/mn-2303_1_one-st...
Besides which, you can also just rip the sign out of the ground or replace the stop sign with an actual speed limit sign. From a SDC perspective, this is kind of the equivalent of using super clever software vulnerabilities when you have physical access to an un-encrypted HDD. Impressive technical tour de force, but misses the forest for the trees if your goal is security.
Traffic sign vandalism is a social problem with a social solution. The first person who tries something like this in production with malicious intent toward SDCs will be charged with some variation of attempted or actual manslaughter/murder and opponents of SDCs will switch to strategies other than domestic terrorism to advance their agenda.
This research is still very interesting and useful, of course, but primarily for other reasons. I just don't see traffic sign vandalism intended to confuse DNNs as a serious -- or unique -- threat to self-driving cars.
I can see such attack methods evolving to produce undetectetable (to humans without software) adversarial images.
It's just that I don't think preventing this style of attack is actually the most relevant/impactful application of this style of research.
Again, I'm not saying this research is irrelevant. I'm saying it's way more relevant than this particular example implies.
> would be less detectable compared with a whole missing sign.
This kind of assumes that the attack wouldn't work in the first place, no? If cars are crashing people are going to investigate to figure out why, and even the most thick-skulled beat cop can figure out that stop signs with rando weird ass stickers on them are probably worth running up the chain...
1. That attack may not target only self-driving cars.
2. Imagine someone driving around with some random picture on their car at the back. Except that the picture is an attack.
It's tunnelvision to assume that the only attackers would be individual radical luddites. There do exist terrorist organizations and state actors who intend to run risks to do damage. And unless you're planning to monitor and respond quickly to all potential threats to all stop signs at all times (and all other possible attacks), what's to stop a group of 40 terrorists from covering their faces, stealing motorcycles, and putting 10,000 stickers on 10,000 signs in 10,000 intersections in 20 different cities in 1 night?
...Or just blowing up a shitload of random houses or whatever.
Basically, the exact same thing as what would stop your scenario. Is it perfect? Even good? Again, no. But striving for good security in a public setting is how you get TSA.
If there are 40 committed and sophisticated terrorists willing to act, there are a lot of easier targets...
Again, I think the "clever localhost software vuln" analogy is apt. It's not that this isn't one possible attack vector. And no attack vector should be ignored. But... really?
A lot of experiments are done on visual data sets, but I don't see why these techniques need to be strictly for visual inputs. Is a machine learning model making decisions based on the results of its classifier? Then it is at risk of being fooled, and whatever decisions it's making, better be trivial ones.
I wrote a document a few months ago that summarizes this and other risks that ML models currently face: https://matt.life/papers/security_privacy_neural_networks.pd...