Optical adversarial attack can change the meaning of road signs
unite.ai
unite.ai
https://en.m.wikipedia.org/wiki/Operation_Greif
...German soldiers, wearing captured British and U.S. Army uniforms and using captured Allied vehicles, were to cause confusion in the rear of the Allied line.
Reconnaissance patrols of three or four men were to reconnoiter on both sides of the Meuse river and also pass on bogus orders to any U.S. units they met, reverse road signs, remove minefield warnings, and cordon off roads with warnings of nonexistent mines.
As a result, U.S. troops began asking other soldiers questions that they felt only Americans would know the answers to in order to flush out the German infiltrators, which included naming certain states' capitals, sports and trivia questions related to the U.S., etc. This practice resulted in U.S. brigadier general Bruce Clarke being held at gunpoint for some time after he incorrectly said the Chicago Cubs were in the American League[7][8][9][10] and a captain spending a week in detention after he was caught wearing German boots. General Omar Bradley was repeatedly stopped in his staff car by checkpoint guards who seemed to enjoy asking him such questions.
Is there a programmatic equivalent of asking what league the Chicago Cubs are in?
(a somewhat forced acronym of “Completely Automated Public Turing test to tell Computers and Humans Apart”)
This is a lot more interesting though from how it highlights the difference between current machine interpretation and human interpretation of road signs: which is to say, explainable AI is still incredibly important - the ability to slightly discolor the signs and have a blackbox AI read something entirely different means the systems and trainers we have are not cueing off a "safe" set of inputs to deliver their interpretations.
From that context, this is very interesting and important work - and I would argue also points the way to at least some basic road safety standards for self-driving systems (they should be resistant to this sort of non-human perceptible visual distortion).
But most of them are easy to spot, at least in the aftermath. If you spayed over a sign, people would notice. If you removed one, a driver would probably not be able to directly notice, but he might notice the situation (i.e. you should not go fast in front of a school, even when the sign is missing) or you could find evidence if an accident happens.
The scary thing about these signs is that they could easily go unnoticed, especially if placed well (for example a crossroad with already sun-bleached signs). Neither a driver (who could possibly intervene) or an investigator would be able to spot this, unless they're specifically looking for it exactly.
Things like this are, in my opinion, also exactly why people put the bar of trust so high for autonomous vehicles: They might fail in ways totally opaque to us. You can at least somewhat estimate what a person might do [0]; for autonomous cars, this is hard or even impossible.
[0] I know someone could be extremely drunk or on drugs and possibly also mistake a stop sign like this, but you'd usually see some warning signs. An autonomous car might go from totally normal to batshit insane without any prior indication or apparent reason.
Self driving cars don’t put that much emphasis on street signs, their not for example going to drive into a brick wall because of an arrow.
Alter speed limit signs in front of a speed trap might hit a lot of people but their unlikely to try and take a sharp turn at 70MPH or anything.
They have to. Even the most accurate maps of speed limits have flaws and cannot adapt to dynamic limits. I'm pretty sure that if car makers could avoid the trouble of parsing signs, the would do so.
> their not for example going to drive into a brick wall because of an arrow.
Teslas first driver death was due to the car not recognizing a white truck. It would be easy to add a fake increase in speed limit, just before a white enough wall, so that the car would not be able to brake in time once the LIDAR picks the obstacle up. To the casual observer it would look like the car just decided to accelerate into a wall.
I kind of understand where they're coming from: a sensor where under some conditions you have to totally disregard all its input is a really bad sensor to have because your model will get disproportionately good using the "good" input, and then you're still left with "and in a medium drizzle we need to also work off optics only".
We’ve evolved to do that because it’s necessary for survival - sometimes we get dust in our eyes, or there is glare, or it’s too dark to see and we don’t have a light, or we’re picking up on a impending problem with something around a corner (a loud fight, or whistling wind), etc.
If visual sensors and LiDAR suck in the rain, radar is great. If radar sucks in crowded road or RF noisy environments, that’s when LiDAR or visual sensors are awesome.
Even on the road this is true - rumble strips, mechanical noise, shaking, etc.
RADAR isn't an adequate replacement as severally crashes have demonstrated - it can't see non metallic objects.
That's the problem I think you're glossing over: under a whole bunch of circumstances your model is operating with out an entire sense.
Sensor fusion weighs inputs from multiple sensors, determining (roughly) which ones are ‘more wrong’ and ignoring them, to various degrees.
LiDAR also doesn’t just ‘not work’ at all in rain, it degrades as it gets heavier until it doesn’t work. Like vision, etc. Radar also can pick up non metallic objects, and does all the time - including geese, humans, clouds, etc, depending on frequency, and there are variable frequency radars out there that can do all sorts of cool stuff, including frequency sweeping, so it’s not just going to ‘miss’ something. There are of course issues like multipathing. But that is why sensor fusion, and similar techniques.
All of this adds up to something much more accurate and less prone to failure than basic vision, which is what Tesla is running now. It also costs more.
I honestly would push back on this, the car is more likely to recognize the arrows/lane markers and might not understand a brick wall - from what I can tell from watching the FSD ride-alongs, if they car doesn’t understand something it just ignores it.
If the self driving car misinterpreted a "NO SWIMMING" sign to a "MUST TURN RIGHT" sign, I assume it's going to take the turn and drive into a lake?
It's common that due to construction, temporary road signs are placed to re-route traffic. It would be a disaster if the self driving car simply ignores them because they are not on the map, I assume?
This is the subject of the story Car Wars by Cory Doctorow - https://web.archive.org/web/20170105065118/http://this.deaki...
I mean, walking upto a stop sign dangling a projector on a stick is pretty damn easy to notice.
It's also much easier to just hold up a bogus actual sign than to go through this mess, and it would be highly illegal as well.
(Disclaimer, I doubt this works because the adversarial image is probably based on a single frame, not the actual object in context)
The latter could easily be part of a dumb prank, "doing nothing wrong", but cause serious damage.
This piece of tape shouldn't kill anyone: https://www.extremetech.com/wp-content/uploads/2020/02/tesla...
Even with humans in the loop in the training process, there's only so much context they can bring to override data that appears authentic.
A suboptimal metaphor in context.
I think that any massively deployed true autopilot system will maintain a global semantic map of roads. Most sudden traffic sign changes, especially those which may influence car behavior, will be manually verified and committed to the global map. Any consistent discrepancies between the model and reality will be investigated as well, especially if it causes customer complaints. And it can be followed by releasing map patches telling software to prefer the model over detection for the problematic sign. Considering that at this stage such maps will be probably integrated with government services (i.e. the map will be compared against official road data), I think such attacks on the autopilot systems will be found quite fast.
I'm not sure that governments would be quick enough to keep this "central roadmap" correct up-to-the-minute (to be honest, I suspect that "up-to-the-month" would be a stretch!).
>Here in AU, we also have "school zones" with speed limit signs that turn on only during certain times and dates.
Such conditions are usually fixed and do not suddenly change. Even OSM handles them today more or less fine.
Smart signage would (likely) be more fragile, (likely) require charging or similar infrastructure, and would require configuration -- so that the sign can communicate in which direction of travel it applies and so on. Certainly not impossible hurdles to overcome, but the costs associated seem very high -- in terms of manufacture, maintenance, and training.
For what it's worth, I suspect that we will see FSD cars operating with infrastructure likely along the lines you outline in the not-terribly-distant future -- but I suspect that they will operate within very clearly prescribed zones, like freeways and highways.
And pranksters, vandals and petty thieves won’t make off with these cryptographically signed signage? Because this already happens for our existing dumb infrastructure.
Cutout of pedestrians is extremely common, especially in the country area where they want to remind drivers to slow down next to homes. Sometimes they wear vests to look a bit like traffic police, through that practice is a bit (more?) illegal.
If I remember right, when that comic came out there was a HN thread discussing exactly those things. Those tricks are not done in order to murder people, but they might confuse a computer.
Also, if you want to mess with autonomous cars it's much easier to just buy this $15 T-shirt instead
https://www.amazon.com/Novelty-Signs-Halloween-Outfit-T-Shir...
See for example
https://slazebni.cs.illinois.edu/fall18/lec12_adversarial.pd...
Yes, but pranksters frequently vandalize stop signs with messages like “stop driving” or “stop eating animals” and human drivers are smart enough to figure out what to do. I’ve never seen someone throw their car into park or toss their burger out the window upon seeing a confusing stop sign.
If you ever travelled by plane, surely you've seen people at the airport trash their perfectly edible food (snacks, etc.) upon seeing the confusing signs just before the safety checks, even though in many cases they would have been allowed to pass it through the gates.
We are just too much used to the existence of optical illusions for humans that we fail to consider them a possibility for attacks. Adversarial attacks on neural networks are simply that: optical illusions, but designed for neural networks instead of humans.
I’m not sure how much a difference that makes in practice?
The hotel room thing seems like a lot more work than quickly done vandalism, and I’m wondering how close to the sign they would need to be.
Great single point of failure there, and now any sign placed temporarily because of unforeseen events (flooding, bridge collapse, etc.) won't be acknowledged by the autonomous car because it lacks a valid QR-code.
If you really wanted a secure system you'd just have a database of all the signs in an area that cars could access (for example if I know what road i'm on, and my approximate location on that road, I should be able to look up what the speed limit on this stretch of road is).
The fact is if you want to be able to stop a car from going over a bridge that's collapsed by placing a sign in front of it, inevitably the same system can prevent a car from going over a bridge that hasn't collapsed. Realistically, unexpected road signs should prompt manual operation.
Forging isn't necessary, just copying one that is inappropriate in a particular setting.
Forgery isn't difficult either if the QR code isn't signed. QR codes are standardized and software to generate any QR code is reasonably straightforward to write (one has to know a bit about Reed-Solomon error correction), but writing software isn't necessary because there are a number of easily available libraries that can be used to generate QR codes.
Cryptographically signed QR codes (which I believe are what you are talking about) would need to include location data that could be verified by the vehicle, such as GPS coordinates and the sign's facing direction. This might work, but of course the logistics could be difficult in practice. Since signs aren't on the internet, it would be impossible to update them whenever a signing key was compromised. Key compromises would probably happen because signs are manufactured all over the world.
theoretically, if someone was to 'borrow' a 20km/h sign from somewhere, and take it to a 130km/h highway, and attach it to a random spot, would the car slow down, until the driver intervened? Or would it "guess" there's something wrong with such a drastic speed change, and beep/ignore/alert the driver?
There have however been issues with e-trons and id.X cars reading side street signs here in Norway and braking hard on the highway.
How would one go and attack a real-world vehicle? Essentially you'd need to gut a vehicle entirely, mock every sensor and actuator to fool the car's control unit to believe it is still driving a car... a lot of effort.
I thought that actually turned out not to be the case quite a bit of the time. An attack that's trained against one model will often work against different models as well, because the various models, even if trained on different data, will often end up looking at similar features.
I'm not an ML expert though, so maybe someone with more knowledge can chime in.
Stuff like this is why we can't trust AI. That doesn't mean we can't use it, but most of our models are black-boxes that can only be "debugged" as much as a pizza can be "decooked"
In the UK they are a different shape and colour, you don't need the text to know a STOP sign is what it is.
No, but some of these sign reading algorithms probably don't take color and shape into account. A naive implementation might first checks the sign for something that can be OCRed, and if it finds that that is a 2 digit number that matches known speed limits then it's a speed limit.
The training set had red octagonal signs and signs with numbers, but no red octagonal signs with numbers. Since the the model had never seen a sign with just two big digits that wasn't a speed limit sign it decides that the numbers was more significant than the shape or color and went with that.
What this shows is how important it is to include intentionally adversarial data in your training set if you want your model to be robust in 'unexpected' situations.
This following article wouldn't have happened if the vehicle was automated, it would have seen something was wrong, and also already have the correct signage from it's database -
3 Are Sentenced to 15 Years in Fatal Stop Sign Prank https://www.washingtonpost.com/archive/politics/1997/06/21/3...
What does need to stop is moving vehicles having uncovered street signs on them, if it's not in place cover it up.
That's not what the link says:
> In March 2001, the Florida Court of Appeals reversed the manslaughter convictions of all three defendants. The court held that the prosecution had made improper comments during closing argument. The court did not reverse the grand theft convictions.
So nobody believed they'd committed the crime, and this was corrected by expunging their conviction for manslaughter, but officially they still stole the sign, even though, if that were true, they would also have committed the manslaughter.
"At a hearing on the motion, several witnesses testified that they had driven by the intersection days before the accident and the stop sign was already down"
"A man who was driving to a luncheon the day before the crash testified that as he pulled up to the intersection, he saw a semi-truck backing up across the road toward the sign. He said that when he came back after lunch, the stop sign was on the ground."