Tesla Autopilot tricked into accelerating from 35 to 85 mph with modified sign
electrek.co
electrek.co
And yes, you could special case this case in the code, but there are hundreds of cases where "common sense" is used in driving, and you will never teach your NNs all of it.
Prediction: no actual "self driving" (to a point where driver can legally be asleep) on public streets till > 10 years from now
Road construction, for example is a reason why
Roadworks are an obvious situation where you drive at a _lower_ speed. This would be a case where you could use computer vision to identity an obstruction, identify roadworks and then adjust speed.
Those databases shouldn't be out of date. Most highway agencies (UK for sure) publish planned works and improvements weeks in advance. Google and Apple both show road closures for example. I belive speed changes are similar.
This is literally Google's back yard
This shouldn’t be a problem, in general, with a couple extra sanity checks.
I would expect the correct solution to more commonly have the opposite problem: too slow in a faster zone.
You perform the task of vision 100% of the time driving. From a purely probabilistic standpoint, even assuming highly accurate ML, the relatively infrequent database updates still make for a sensible prior.
This is somewhat adjacent to what Gary Marcus has been arguing, and I think it makes a lot of sense; there seems to be no compelling reason to rely exclusively on primitives (i.e. vision) when good priors are easily accessible.
But a car thinking a 35mph zone is an 85mph zone... that could be a very big deal indeed.
Regardless, if self-driving cars (or at least "computer assisted driving" cars) were to be widely deployed enough and shown to depend on these sorts of databases, I'd expect it to become a legal requirement to make sure these databases are kept up to date by city/county/state agencies, and that car systems get those updates. Otherwise, expensive liability ensues.
BMW handles it if the navigation database is up to date. At least on the A2 in the Netherlands, which has the 06:00 to 19:00 changes between 100 and 130. It also understands temporary changes for roadworks, but gets it wrong when a roadwork/temporary sign has an exception like "when road is wet"
That tells me the camera only understands the main sign and then the navigation database tells it about exceptions / special cases. Nice as a small reminder on your screen, but I wouldn't trust it to self drive based on that data.
In the UK smart motorways can vary from 30mph to 70mph according to traffic. This can change within minutes and obviously cannot be in databases. So this is a good case for sign recognition. In theory though, there is some network that's updating the speed limits and there's no technical reason why that couldn't be a public information service (maybe it already is).
Then you have the sibling comments - speed limits which are time based. I would imagine this is country specific and is, mostly, a matter of public record. But would you trust CV to recognise text on a sign with specific hours? I don't know how complex the signage is, but trying to use ML for this seems absurdly wasteful of resources.
Finally you have other information - how fast are the cars around you travelling? Have you detected an obstruction on the road (eg cones)? Is the weather inclement and therefore you should drive more slowly, etc.
I would imaging that if self driving cars become ubiquitous there will eventually be a system for real time speed limit determination, whether that's some kind of wireless beacon at regular intervals or an online database that's kept up to date I don't know.
For the car to react according to how other cars drive, you need to have other cars on the road at the same time. I have a narrow gravel road where the limit is 30km/h but safe driving is between 10 and 20, but my car tell the limit is 50km/h. It can be weeks without me seeing other cars on that road at the same time as me. Some mornings during the winter I can drive because I know where the road is, not because all of it is visible. Good luck having an autopilot doing that
Similarly the car should be able to identify road surface conditions and also how much power it's needing to put in to achieve a set speed. If it's identified that traction is being lost over gravel, then it should reduce power.
Equally the autopilot should consider visibility. There are plenty of roads that are national speed limit and you'd be insane to drive at 60 on them due to blind or extremely tight corners.
But with LEDs this seems like massively over-engineering the problem. Why use computer vision when you can potentially broker a deal with the highways agency to get live speed limits on a particular stretch of highway?
Current Teslas CANNOT have this error simply because MobilEye patented reading speed limit signs and Tesla no longer uses the MobilEye system and haven't since mid-2016.
Is this not how most self-driving works?
From there the car needs to segment its interpreted reality and run path finding algorithms.
AGI needs to take that reality and seemingly consciously understand the many different ways that it can interact with it.
I'm oversimplifying the post mapping implementation of self driving - but not so as to remove an insurmountable problem.
Tesla's biggest fuck ups (such as driving under a truck) seem to me to be the consequences of initial mistakes in building a map of reality, rather than in what's being done with that map.
For self driving, as far as I can tell, the map of reality is the hard part.
For AGI, it gets you to the hard part.
People saying "see, THIS PROVES self-driving is as hard as AGI" without even applying any critical thinking is just confirmation bias.
You can tell the Model S is old because it has the pre-April-2016 bubble nose. I believe all vehicles with the bubble nose have the old MobilEye system at best (which was never intended for self-driving).
Cool, so the new cars have even less features than before. And that justifies that self driving is near?
If the MobilEye system is not trustworthy then why is the vehicle accelerating based on the data it receives from it? Why doesn't Tesla update the software to stop doing that?
The reason we don't already have self driving cars is that there hasn't been enough computing power in a small package to make the sort of decision logic needed practical until recently.
As long as the prerequisite for self driving has been to emulate human's driving skills, there just hasn't been the computing power available. Now that there is, progress is being made, but really it's just a transitional period.
What's eventually going to happen is that roads everywhere will get machine assists - basically, small beacons running on low power that provide easy machine readable data on everything from precise road location to speed limits to local road conditions. They'll be small, low powered radio transmitters or even NFC devices near the road that the car can just download data from.
There's always been a chicken and egg problem with self driving vehicles, where no one wanted to invest in machine assists for all roads because no one wanted to put that much money and effort into something that no one needed (no one had self driving cars) but no one wanted to make self driving cars a product that wouldn't be successful without machine assists as well as a way to drive without them as a backup.
Now that self driving cars are out there and everyone is getting comfortable with the concept, machine assists will be coming, and they're going to have benefits to ordinary drivers, too.
If you want an example of why this will work, look at aircraft. Most jet liners today are perfectly capable of landing themselves in almost any weather because of the electronic navigation system they have built in to most airports. The planes aren't using machine vision and AI code, they're landing on a piece of ground defined by numbers in the navigation system which is precisely located by GPS coordinates.
The same will work for cars, it just hasn't been built out yet.
Airports are completely irrelevant example. There are ~2500 IATA recognized airports out of ~50K "airfields" total, and only ~1000 of these 2500 are equipped with ILS. It is not comparable with millions of kilometers of roads around the planet, even if we will count only "1st world countries".
We already have "driver-asleep self driving" on public streets: https://www.theverge.com/2019/12/9/21000085/waymo-fully-driv...
So you can attack an AI driver with targeted sign changes which wouldn't work with human drivers. I don't think you'll need experimental evidence to agree that you could also attack human drivers with targeted sign changes which wouldn't work with a particular AI driver. Relevant xkcd: https://m.xkcd.com/1958/.
An AI driver will remain dumber than a human until we're very close to AGI, if not forever. But it never gets distracted, and always tries its best to follow the rules. This may well be enough to compensate for its low intelligence - and beyond. The worst human drivers ain't always the lowest IQ ones.
Abstractly, it's interesting, if only because it probably wouldn't trick a human -- everyone would see the 85mph as BS regardless of the defaced sign.
Has anyone hacked into openstreetmaps yet and fiddled with speed limits?
Its likely now that this error was pointed out that Tesla may add some speed limit validation code to ensure a car doesn't speed up much when a speed limit jumps by more than 25% or something like that..
https://en.wikipedia.org/wiki/Tesla_Autopilot#Speed_assist
From what I've gathered elsewhere, the GPS data is OSM.
https://wiki.openstreetmap.org/wiki/Speed_limits
I do know of one instance where modifying street signs tricked humans: there's a short speed-trap section of road outside a rural town north of me that's labeled 25 mph even though the approach roads are 45 mph. Apparently it's zoned residential even though there are only like 5 houses set way back from the road (looks like it should be 45, maybe 35). Seems like someone in the 70s thought that zoning something for higher density caused high-density housing developments to arrive.
Of course, this section was where the cops liked to sit. When I was in high school, some kids pulled legitimate signs off a different section of road and re-labeled it 45 mph. While the average speed of traffic to school in the morning sped up dramatically for a couple weeks, the police did not agree that the presence of these signs changed the speed limit on the road. Fortunately, the perpetrators got off with a slap on the wrist, unfortunately, the county didn't review and revise the speed limits.
Now this would be an interesting legal challenge. The official wording will vary by state but it'll be something like
"a person must not, when the sign is in place on the highway, drive or operate a vehicle on a highway at a greater rate of speed than that indicated on the sign for that category of motor vehicle, unless another sign on a specific highway in the defined area so indicates"
So... can you really be faulted if there's a not-obviously-fake speed limit sign? If I got a ticket for that I would definitely be disputing it.
You'd have a pretty easy case, I think, the only defenses would be "there's a sidewalk here so that's typically 25" and "you drove here before so you should remember it's 25".
Ironically, current Teslas are largely immune to this as they use a database of speed limits.
Rodney Brooks points out that if we ever get AGI [1], we'll have solved the autonomous vehicle problem. But it's far from clear to me that we'll truly solve the problem much before then, as cars and roadways are built with GI expectations in mind.
[1] https://en.wikipedia.org/wiki/Artificial_general_intelligenc...
And probably prosecutable as felony manslaughter, the same way removing a stop sign would be, because you're acting in a way you know will get people killed.
Also note that TACC isn’t autopilot, one just controls your speed automatically, the other actually drives your car. It wasn’t clear if the flaw affected auto pilot as well, since it can’t be used for non freeway driving.
So if it is hard for a human, then it seems obvious that a tesla would also fail to recognize a 35.
Maybe the difference is that a human might not be certain and slow down just in case, whereas a deep neural network might be certain in its errors.
This isn't a failure of autonomy imo. Speed limits are arbitrary placeholders that are more for supplementing the slow reaction speed of the human brain. If we started focusing more on making the algorithm implement actual analysis of the environment instead of "the rules of the road", we might see better results.
You need to be a special type of stupid to think that a road with a speed limit of 35 MPH is 85 because of a road sign. The 'drive into the lake because the GPS nav tells me to' level of stupid.
It sounds like the Tesla is just using what the camera sees - which is bad if true. Admittedly this was tested on a race track, so maybe there is no data saying otherwise (or even the opposite).
In order to get all Teslas who go past a sign on the highway, you just have to modify the sign slightly and everyone is affected (in theory).
In order to affect the same amount of people with your "brick through the windshield" strategy, you'll need a lot more manpower than just a sticker on a sign.
To be clear, I find this entire concept of unsupervised robot vehicles both dangerous and absurd - even more so than the already dangerous and absurd baseline of a society so intimately bound up with automotive travel as ours. Exceeding such a high baseline as that is in its way impressive, and certainly demonstrates the astonishing overconfidence rampant in some segments of our very young and rather careless industry, but let's not get distracted from that essential point and waste our efforts on inconsequential arguments over whether a sticker is more dangerous than a brick.
No, I think it's the scope of affecting many after another without doing something more. Throwing bricks requires continues action while a sticker is a thing you do once and then it "does it for you".
See it as working every day and getting paid for that, versus a savings account where you get returns without really doing anything.
And sorry, I didn't really join the conversations to argue against "concept of unsupervised robot vehicles [is] both dangerous and absurd " so I agree with the rest of your message.
What if it was mud? Or salt? Or bird shit?
So a malicious actor could put black "tape" down on a highway, and just wait for the right conditions for people to get confused.
Yes, I'm sure the courts should throw out the ticket, but sometimes they mess up, and often you lose time and money to pursue your case.
There's a pretty long stretch of road between Austin and San Antonio that's 85 the whole way. It's not as big as the I-35, but the higher speed limit means it draws quite a bit of traffic for people traveling between the two cities.
Point stands: 85 is ridiculous in anything like a residential area.
Either way, my point is digitized information is helpful, but not sufficient
I obviously don't have any data, but I think this behavior is strongly correlated with IQ.
No matter what, the answer is probably both. You can't modify street signs now anyway. You also can't drive 85 in residential/city areas.
I think that is a real possibility if we are referring to years on Jupiter (which has an orbital period of 12 Earth years).
Next step: sql injection.
https://hackaday.com/2014/04/04/sql-injection-fools-speed-tr...
Last few cars I've rented had a speed limit sign generally matching the speed limit on the dash, and on the one I rented this christmas passing a speed limit sign (or an other sign triggering a speed limit change) while on limiter or cruise control would flash a message suggesting pressing a button twice to change the configured speed to that (rather than manually adjust using the +/- buttons).
They were mediums (C-segment) and compact MPV not large or luxury, which I expect is why the sign reading was mostly to purely indicative.
Current Teslas don't read speed limit signs, they access an online database. Vandalize the speed limit signs all you want, as modern Teslas aren't even reading them.
You must react to the environment as well...
It's also illegal to do this, so it's not a valid edge-case.
I have yet to see someone die because of the illegal and "labor intensive" situation they created in this "research".
> Would you feel better if you died due to someone breaking the law?
I can respond that back, but then with problems that are factual ( drinking and driving). Not fictional or harder to execute do it in practice ( this research)
Altering a speed limit sign may not be legal, but if it's done in a way that wouldn't fool a human, but would fool a self-driving system, that's just a way that a self-driving system could be unsafe that a human driver wouldn't, and people will react negatively toward the self-driving system. That's just human nature, even if it's not logical.
Regardless, we need to be aware of these possible "exploits" on self-driving systems. Sure, no one would bother altering a speed limit sign to cause crashes, because human drivers have context and common sense that would make that mostly ineffective. But if it were known that some popular self-driving systems could be fooled by that, I'd expect to see this happen more often, perpetrated by people who would probably otherwise try to hurt people in other ways. It's not a "new idea" in the context of driving in general, but it's a new -- and potentially very effective -- idea to dangerously mess with self-driving systems.
If selve driving vehicles become popular. The exploiter would have been recorded somewhere, identified and put to trial.
Story would go viral and the cat and mouse game continues.
No one is going to risk jail for letting cars drive 80 miles per hour, until the next road sign.
This is a stupid exploit.
Road signs can get dirty or damaged. There definitely are edge-cases where a self-driving car might not comprehend an imperfect sign which a human would read just fine.
Maybe this test in particular isn't fair, but it does raise questions.
Like...y'all have lost your goddamn minds. OF COURSE many humans would do it! Humans fucking drive straight into lakes because the map told them to. Humans do not have some kind of magical ward against fucking up or against being tricked. Quite the opposite.
But you know what the difference is? As soon as the computer system has programmed in a record of local contextual defaults, this problem won't happen again. Say the same about humans, I dare you.
I had to check if this was real or just a bit from the office that imprinted. Googling "humans drive into lake" only finds one incident. Which occured at midnight.[1]
and if you look at the research, they didnt convert it into an 8, they simply extended the middle part of the 3 a little bit [2]
[1]https://fox8.com/news/a-little-embarrassed-woman-follows-car...
[2]https://cdn0.tnwcdn.com/wp-content/blogs.dir/1/files/2020/02...
You'd find more examples if you tried slightly harder ("lake" isn't the important part!):
https://kfgo.com/2020/02/10/man-drives-into-mississippi-rive...
https://www.cnet.com/news/man-drives-into-river-gps-china/
https://www.mirror.co.uk/news/world-news/man-watches-wife-bu...
https://abcnews.go.com/blogs/headlines/2012/03/gps-tracking-...
http://news.bbc.co.uk/2/hi/uk_news/england/bradford/7962212....
https://www.boston25news.com/news/man-drives-into-pond-while...
https://www.nbcnewyork.com/news/local/gps-leads-nj-motorist-...
https://www.dailymail.co.uk/news/article-1164705/BMW-left-te...
https://www.westsiderag.com/2013/05/01/gps-brain-fail-driver...
https://www.news.com.au/lifestyle/real-life/driver-follows-g...
https://nymag.com/intelligencer/2018/01/waze-app-directs-dri...
https://www.fox5atlanta.com/news/2-drivers-stuck-on-train-tr...
https://www.bostonmagazine.com/news/2013/06/19/mbta-train-ac...
https://www.abc.net.au/news/2015-10-01/car-being-hit-by-trai...
https://abc7chicago.com/395218/
People act like humans are immune from making idiotic decisions or from ignoring their surroundings or something. That's...super naive.
would you be fooled by the attached image in my previous comment? If you were fooled, would your default action be to accelerate by 50mph, when no other drivers are?
It might if it were relying solely on a GPS and nothing else, but, of course, none of them do that. The biggest difference is that navigation software can improve and sensors don't stop paying attention. You can't say either of those things about people who screw up in exactly the same circumstances.
> Heres an example of AI driving into a river https://electrek.co/2019/03/10/tesla-crash-river-claim-unint.... so it happens as well.
Let's note that the article is skeptical that the problem was actually the car:
"I want to give the driver the benefit of the doubt, but every time we have seen similar circumstances, the logs always pointed to a user mistake."
But even giving the driver the benefit of the doubt and asserting as a premise that the AI caused the accident, we're still left with the fact that people unintentionally accelerate their vehicles all the damn time. If a machine does it once in a while, that's not a regression, that's the baseline.
Again, do they do this by 50+mph ALL THE GODDAMN TIME?
>> If a machine does it once in a while, that's not a regression, that's the baseline.
If all the machines do it at once though? The impact of self-driving failing is a lot more than 1 person, typically.
Also, we dont know the rate of error. There are 1.2 Billion drivers in the world. 3.5 Billions smartphone users. Its safe to say 50% of drivers use GPS. So 600 Millions Drivers using GPS? There arent even 1 million Tesla's on the road yet and they are already having incidents.
No. Usually they crash into something first. I can tell you one personal anecdote, though, where in 1995 the gas pedal in my truck actually got physically stuck in the down position and I had to hold the brake pedal down with one foot while wedging my other foot underneath the gas pedal to loosen it. Of course I could only do this after the several (5 or 6?) seconds it took for me to understand what was happening and then react. 6 seconds is a long time to react to a catastrophic event. Humans aren't great at it. We almost always do the wrong thing. We turn the wrong way. We push the wrong pedal. We don't understand our surroundings. We ignore our surroundings.
Anyway. Please note I'm not saying that this isn't a failure of the nav system. It is. I'm saying that anyone claiming that humans are magically immune is wrong because humans are very dumb and make dumb mistakes all of the time.
I wanted to link that not because of the "there's an XKCD for everything" meme but because it makes an interesting point: sabotaging roads isn't difficult now, with human drivers. There's no reason to assume it would suddenly become a substantial threat when autonomous cars are commonly adopted. An issue worth considering and accounting for, but not worth public worry.