Slight Street Sign Modifications Can Fool Machine Learning Algorithms
spectrum.ieee.org
spectrum.ieee.org
These things will be fixed in time. There is nothing to be gained by pretending that they are not problems.
It's not a good idea to generalize like that. There are many AI systems that use neural nets for perception and then use different models, such as conditional random fields (CRF) to see if entities make sense in context.
Not to mention that these adversarial examples don't work well from all distances and angles. As a car changes position slightly, the adversarial example disappears. It's very hard to create an adversarial example that works from all angles on all networks. There is also a special training regimen that fortifies neural nets against adversarial examples.
The problem of adversarial images is closely related to GANs and has been brought into attention by the same researcher, Ian Goodfellow. So it's not a neglected problem - some of the top minds are on it.
Many street signs especially in rural areas might have a pretty small visibility window and they will only be visible from a pretty fixed angle until you pass them.
So while you might not be able to develop an adversarial example that works for all signs from all distances and angles it wouldn't be hard to develop one for a specific sign. The angle for that sign is known, the distance in which most cars could resolve the sign is also known, the average speed on the road can also be taken into account which gives you a rough idea of the time span the autonomous driver would have to identify and resolve the sign.
Heck if you know the algorithm it wouldn't be that hard to build an app that basically allows you to take a picture of the sign with your phone from the road, select a car or subset of cars you want to jam and it would generate a likely pattern to apply to the sign to jam those cars.
I should also add that in this case some of the examples that do seem to work look to be of the shape and size of common bumper stickers, this is pretty worrisome since these are not that uncommon on street signs, especially in rural areas where they are more or less at reachable heights.
I also wonder how well do these systems deal with graffiti.
Especially given that the cost of crashing an unmanned truck will be 100% capital and 0% human, I expect we'll start seeing some pretty serious criminal penalties passed once the tech gets there.
I think that scenario isn't as likely as unintentional screwups at the moment.
That's why I believe it will be fixed. It is the "it's not a problem" crowd that I take issue with, and I would be surprised if many of the top researchers held that position, unless they are confident that they are close to a solution.
I think you are under-generalizing here. The risk here is not of a sign being mistaken for a gorilla specifically, or even about signs specifically, but the fragility of image recognition in general. Similarly, because the issue is not (just) about adversarial images, the fragility of adversarial images is not much of a mitigating factor.
... and the solutions will be written in blood.
But I suspect that driverless cars will start out pretty safe and improve from there.
I think there is an interesting distinction that is worth discussing ...
At the advent of train/plane travel, you could opt out of those novel risks by eschewing those modes of travel.
However, you could be driving along in your driver-piloted car, minding your own business and find yourself involved in an accident caused by the software in a driverless car you are sharing the road with.
Before cars, people routinely walked in the middle of streets. "Jaywalking" wasn't a concept. It was entirely up to early cars to stop in time, and they didn't have good brakes. Fortunately they weren't very fast.
Also, early pilots, besides the obvious risks, were sometimes daredevils who would do things like buzz houses.
I think the main point here is that society's attitude towards risk has changed; we take small risks much more seriously than they used to when earlier forms of transportation were introduced. Accidents involving driverless cars make the national news and manufacturers are being very cautious. With earlier attitudes towards risk, I expect they'd be in widespread use by now.
If they were trying to directly use this info, this would be the least of the serious issues. For example, there are plenty of stop signs well hidden by trees. Such a car would probably last about a mile in a non freeway setting.
A trivial model of "hey, does this make any sense appearing at this place on a map" defeats this, etc.
Of course this stuff is it is infancy and can do dumb things. But this is just isn't that big a deal.
Humans also act much worse than you imply. Look at the number of people who accelerate into crashes instead of braking, etc. Again, doesn't mean AI should get a pass, and humans are infinitely more complex, but we shouldn't be held up as amazing at this either.
(That's actually the part that worries me. That we aren't good at it)
This is not unlike the human body's deeply-evolved reflexes - before the neocortex can even process that a pan is boiling hot, the nervous system bypasses the brain and and tells the hands to drop it.
In the case of driverless cars, DannyBee is absolutely correct: truly driverless cars will not make decisions from a single data point. I have some past experience with unmanned aerial vehicles and those things had insane amounts of redundancy in almost every signal and decision making process; driverless cars are more complex and will come decades later and will probably take this even further.
In the case of driving, however, the problem of deciding what behavior to veto is itself a hard problem - perhaps the hardest of all. When the system monitoring the AI is itself AI, the argument becomes circular - which is not to say it cannot be solved, but I think it makes it harder to dismiss the concerns.
Oh look:
http://www.rtl-sdr.com/spoofing-gps-locations-with-low-cost-...
Seriously. Y'all realize how much in your daily life would pretty much explode and kill people if that wasn't true, right?
What are the guarantees that those higher-level "complex systems" aren't going to have some weird behaviors as well?
The problem isn't just that ANNs misclassify adversarial examples. The problem is that it's counter-intuitive to an average observer and that no one clearly knows why those examples generalize so well. It's a remarkable property that a lot of AI "enthusiasts" try to downplay.
It's one thing to wire up unreliable but simple and fully understood components into a more reliable system. It's an entirely different level of challenge if the "unreliable components" are complex and poorly understood.
> Humans also act much worse than you imply. Look at the number of people who accelerate into crashes instead of braking, etc.
Except we know very well how and how often people make mistakes. For most of those mistakes we have a pretty good idea why they happen (at the high level, I'm not talking about neuroscience). Our roads, our cars and our laws are designed to handle these failures. Also, we have a pretty good model of how other people behave on the road, so we cal react accordingly.
All of this goes out of the window with self-driving cars.
Hell, most people naively assume that if a single self-driving gets into 50% less accidents than an average person than replacing all drivers with self-driving cars will reduce the global accident rate by at least 50%. This assumption doesn't take into account the fact that many accidents are caused by complex interactions between several vehicles and the environment. So introducing many self-driving cars can lead to some nasty emergent behaviors and some mass accidents that just aren't possible with human drivers.
I'd say that's a pretty big deal.
I think the problem with the driver-less car is that it doesn't have goal-oriented general perception like human beings do. Humans filter the world through goals. The world is interpreted as a set of affordances and threats to your goals, your brain is predisposed with distinguishing the two.
As it stands, this is not how driver-less cars are being designed, they are being designed to specifically recognize the presence and content of street features like signs and other moving bodies. A human driver might see a tall parked car with a big gap in front, and recognize a pedestrian affordance which he should steer clear of in case somebody pops out. The way I see people talk about driver-less cars, this "common sense" type of judgement would have to be deliberately integrated.
No. It's just a fact of parsimonious signal representation that sometimes you get null spaces with which you can manipulate to maximally separate the performance of two very different receivers.
http://jamesbridle.com/works/autonomous-trap-001
They used salt to construct a circle with a solid line on the inside ("do not cross") and a dashed line on the outside ("come on in").
This is just a guy sprinkling salt around his regular car.
The car drives into the salt circle then stops.
Is this actually an autonomous car, or is it conceptual?
I don't actually have a self-driving car, unfortunately....
Even though I know the road wouldn't drive directly into the bridge, I slow down a little and look carefully to make sure I'm actually not going to crash into the bridge.
When my perception doesn't fit my internal model, I gather more data (look at different parts of the bridge and what other cars are doing), or transform the data (ie turn my head slightly and look at the bridge and road from different angles)
Edit: Likewise, when someone's tone doesn't match their words, I gather more data (look at their body language).
Have any researchers experimented with neural nets to do the same? I haven't noticed any posts here about that.
https://arxiv.org/abs/1707.03501
However, OpenAI quickly refuted it by creating adversarial examples that continue to fool the classifier even when rotated, scaled, etc:
https://blog.openai.com/robust-adversarial-inputs/
So it looks like there's no "easy" way out here. Multiple types of sensors may help, but it seems likely that it will still be possible to construct examples that fool network over all sensor inputs at once.
Ian Goodfellow and Nicolas Papernot have a good blog on machine learning security issues. One relevant post on why this is such a hard problem:
http://www.cleverhans.io/security/privacy/ml/2017/02/15/why-...
If we constructed truly adversarial examples for human neurology, I bet they would be equally insane.
Also, adversarial in this case seems to refer to images perceived differently by machines than by humans, so it's not really possible to create such ones for humans.
So a self driving vehicle would have to somehow know that the road dips and from that know that they won't run into the bridge and know that there may be cars hidden after the dip.
Edit: tried to get a good photo from Google maps but since the Google Street View Car camera is high up off the ground you can actually see more clearly what is going on with the bridge then you would be able to at street level so it's not a good example. I wonder what implications camera height has on the safety of self driving systems. Also, the bridges I have in mind seem to be on Soldier Field Road not Storrow (Storrow turns into Soldier Field and I haven't lived in Boston for years so forgive the mistake :))
The uncertainity part should ideally, in my understanding, be represented by confidences returned by neural networks. They don't claim "there's a crossroad ahead, and not a tollgate" but rather "85% match for crossroad, 30% match for tollgate, [...]". If those results are not distinct enough, the surrounding application should probably go into a more cautious mode and slow the car down to begin with. I suppose that's what such systems do, but maybe someone with more field knowledge can confirm/negate that.
Even if we intentionally supported machine drivers with special signs, beacons, or other things, those are still going to be subject to interference and defacing.
Then Animal powered mechanical transport(carriages) was invented and people made roads to accommodate the new means of transport.
Then the fully mechanical transport came and we just made roads for these machines.
For some reason, this time Humans are trying to keep the roads the same. Well, actually not, there are project to create roads designed for self driving vehicles.
I think those in the software industry have a hammer and now see everything as a nail. What a successful AI transportation system would be? Probably based on specifically designed infrastructure that is using the AI tech that never worked for the old school roads.
1. To begin with it is only used on motorways/highways as they are long straight roads, where the AI can take over the boring part and the driver can be left to take over if something complex happens.
2. Shared AI maps across all cars that can route everyone to/from work. As this is shared, it can balance the load and result in the least amount of traffic jams. Humans still do the driving.
3. The government sets up AI cameras that monitor city routes, and cars can tap into this information as they drive. The benefit here would be seeing the road from all angles, as well as massive computers bigger than cars doing all the number crunching. Over time more and more city routes could become approved to be fully driven by AI.
That's my approach to an incremental design at least.
Road signs were made for humans because we don't have the ability to connect to the internet and fetch the data in less than a second, but autonomous card do, so why not use it?
An attack on something like this would scale very poorly, as you would need physical access to all street signs. Issuing RFID to a street sign would be just another step along the manufacturing process of the sign, or as a step to the mounting of the sign.
I can't take credit for this idea though, it's already been [explored in a paper](https://link.springer.com/chapter/10.1007/978-3-642-41647-7_...).
I wouldn't underestimate the eventual legions of unemployed truck drivers.
They will probably lose eventually, but I expect a spirited effort.
The problem for them is that where they would cause the most harm, rural areas (since they are in most dire need of supplies a few towns away and don't have any good alternatives to cars), is where such an attack would be the hardest to implement. Canada, Australia, Iceland and Alaska has many roads where vital street signs can be tens of miles apart from each other, as well any actual people who might be effected by this. Also, demographic movements is working to their disadvantage; more and more people everyday are moving to large cities.
Another scenario is more of a slow and steady war. I'd expect that autonomous vehicles will occasionally be alone and unoccupied, which makes them a tempting and defenseless target. You wouldn't even need to do any physical damage to disable them - just a little electrical tape over their sensors, or some chocks under their tires. Soon, the autonomous trucking cos have to employ some of the ex-truck drivers to go around re-enabling their vehicles. And of course, those ex-truck drivers are in cahoots with the disbalers. That could certainly drive down their savings on labor costs.
Waze and other navigation systems have a database of signs, road speeds and traffic enforcement cameras.
I don't know about the US but there are places where signs are in a database already.
This can be crowd sourced just like anything else.
Google Street View already has a pretty good DB to start from.
Because you shouldn’t need to connect to the Internet and resolve GPS coordinates to know there’s a sign 50 meters away. If autonomous cars can’t "see" signs, let make them "see-able" with e.g. small radio transmitters along the road.
An online database that can be cached (so you aren't in trouble if your network drops while you go through a tunnel or similar) is much cheaper to implement than upgrading physical signs.
However, it ain't cheap and simple to implement, I agree with that. In countries like Sweden, Finland, Iceland, Canada and Australia they probably have the means to outfit their cities since they have the wealth to do it, but the rural areas would be a real challenge since they are so sparsely populated even when the money is there. In contrast to when these countries pioneered the implementation of the Internet, there's nothing like the phone network to piggy back this time.
All signs can be a victim of vandalsim.
I think computer vision, a sign DB an IR water marked QR code and some RFID combined can produce a pretty robust and tamper reselient system.
I'm not really concerned about the risk to humans that have been very precisely dressed to fool cars.
But in the context of deliberately hidden or altered pedestrians, there's no risk I can see here. The pedestrian would have to be trying to look like something else or hide in a very precise way, and they can do that to regular drivers right now.
Why does everyone think self driving cars can use GPS to identify sign locations? GPS is not accurate enough to do that, and the labeling would need to be done by hand, which is not feasible. Plus, signs change and move around, making the labeling task endless.
GPS could be a part of this, but it's easy to imagine what could go wrong. Somebody mistakenly installs a stop sign without updating the database. Another person makes a mistake configuring a cache parameter and the CDN starts serving up last year's map. There's a whole class of problems eliminated by keeping the information local. Imagine if you had to drive looking only at road information served over the internet -- would you trust it?
It is rather that we built the signs in a way such that they are very robust for the human visual system to detect.
Though I've seen a new stop sign added recently, and the number of cars that don't see it is remarkable.
I'm not an expert on AI though ( a few classed at university and I specialized in something else).
Its the general classifiers that give us trouble. We want to show it 100s of stop sign pictures and have it figure out when we show it new one. But we're not asking it, is this a stop sign, we're asking "what is this".
We can write software that probably is good at finding hexagons and colors and thus stop signs. Take facial recognition, its remarkably good at this point, but its doing one thing ( though I've seen a computer id 3 people in a photo with 2 in it , because a third person's photo was in the background.)
Though I've seen a new stop sign added recently, and the number of cars that don't see it is remarkable.
These two sentences together show that our brains aren't just looking for the signs; we're also looking at many other aspects of the situation and even taking into account past experience (e.g. is this an intersection? Have I seen a stop sign here before? If I'm new to this area, I'm likely going to be far more alert to the signage.)
If someone planted a (non-modified) stop sign on the side of a highway, where the road is completely straight and with no intersection, I bet some drivers won't even see it, those who do will be puzzled, and approximately none of them will even try to stop.
Also, the claim that humans excel at this can be called anthropocentrism. You could also point at the features those machine learning algorithms use and say "why don't humans see these very prominent features?"
When I travelled to the US, from New Zealand, to interview with a certain large company, I actually managed to mis-read road signs, in particular traffic lights, on a number of occasions, much to my alarm.
I am not a bad driver - and there were a number of factors - perhaps being tired from the trip, so, it isn't clear cut. But, I remember feeling like the traffic lights just didn't look like traffic lights - the intersections just felt off - so off, that I actually failed to recognise them in some instances.
It was actually a bit upsetting at the time, I've never quite experienced anything like it, but I think it's possible that the human system can fail to work if the input is sufficiently different from what is "typical".
We can't do that with humans. So we don't know what similar situations we could create if we did.
"We'll just cover all of the possible use cases" - Self Driving Car Engineer
OT: My favorite street sign graffiti is the hula-hoop stickers that artists put on pedestrian walking signs.
Are there any other promising technologies that can replace or at-least augment current Machine Learning systems?
To other humans, those are very much recognisable humans, but face detectors won't think they are.
Also, figuring out the type of sign for the outline. Then the icon inside seems like an approach that could work.
Our eyes can be fooled easily anyway.
This looks like a couple of bumper stickers will cause problems even when they do not obstruct the actual sign.
This is a pretty big issue that will have to be dealt with.
We're not talking about modifications causing a sign to accidentally be mistaken for something else, we are talking about deliberate modifications to road signs that cause vehicles to misinterpret them.
If you modified a sign so that most users still perceived it as a stop sign but colorblind people misread it as a speed limit you'd be doing exactly the same kind of thing.
I mean, it's not like there's some free-speech right to graffiti on road signage in the first place, let alone to modify it so that some road users will misunderstand the sign's meaning. If you interfere with a roadsign in order to deliberately confuse road users, you are a criminal.
Images with Zero Modifications Can Completely Fool Human Sight!
Examples: http://www.ritsumei.ac.jp/~akitaoka/index-e.html
(In case it's not obvious, the implication is that it's possible to engineer sign street modifications that fool human beings too.)
Everyone needs to get out of themselves and see that the human vision systems can be fooled with adversarial mods as well, just not the same bugs as computer vision...