Tesla Autopilot failing where there was a fatal car accident [video]
youtube.com
youtube.com
In my opinion Tesla and all other manufacturers should perfect their Level 2 tech (and not call it autopilot) and then jump straight to Level 5, even if it means it's 3 decades out. We can't have drivers sleeping at the wheel in level 3/4 cars with patchy software. And that's regardless of whether manufacturers can prove that their tech has markedly fewer accidents than often-failing human drivers. I'm happy to take the risk to crash with a sleepy fleshdriver, but I'm not ok with crashing due to a software bug. I think many share this sentiment - which means there is a double standard that setting the bar much much higher for autonomous systems than they will ever be for humans.
I find this very hard to understand, especially when you're making the choice on behalf of others. Would you drive your children to school, knowing that you're making them less safe, just so that if they do die it's your fault rather than that of some programmer?
But if you've already stipulated that the driverless cars are safe than driven ones, then that fact is irrelevant. The program has already been written and if it is in fact safer than my own driving then I would want to use it.
NASA is known to define a set of strict and absolute mission rules before the beginning of a mission for just this reason. See for example the rules defined for various Apollo missions [1]. Note how those rules define all sorts of limits and abort triggering conditions. This is done because there is always a desire accomplish the mission even when something has gone wrong.
But perhaps the most convincing argument is that even the drunk, sleepy or sloppy driver at least takes a reciprocal risk in traffic. The one thing that makes me believe that people try their best to avoid hitting me is that we will both be in the accident.
Simply put: we are sort of fine with being put at risk by human behavior. That's part of of the risks we have already accepted. This is a new risk and even if it reduces another risk (fewer human drivers) we classify it as a "new risk" and we are risk averse.
> All Tesla vehicles produced in our factory, including Model 3, have the hardware needed for full self-driving capability at a safety level substantially greater than that of a human driver.
I should note that both Tesla and you stress hardware capability, not software. Tesla mopes around the bushes with phrases like "awaiting regulatory approval".
If Tesla does actually mean "we've got all the gadgets but the software isn't there yet", they have a serious marketing issue. To the layperson, everything they say makes it sound like their car will make you breakfast, drive you to work and take the kids to school. Seems like they're not there and it's not just regulation holding them up.
At the shallow end, it's misleading advertising, at worst it means people are putting their and their passengers' lives at serious risk.
I would not be surprised if we start seeing mandated health warnings accompany semi-autonomous vehicle adverts, spelling out exactly what they can and cannot do.
If they sell something now that has hardware for level 5 but neither software nor approval for level 5, but they weasel-maket it as “autopilot” and users can enable a mode where they can let go of the wheel - then they are just criminal.
AFAIK the AP focuses more on moving targets for collision avoidance, stationary objects are harder because there's so many of them you'll never hit since they're above or next to the road, but that's hard to tell until they're very close (can't see where the road leads to 100m away).
I have only worked tangentially with self-driving systems but in general they are very bad at dealing with stationary objects - this is why it can't see the center barrier.
This article describes the reason why radar and stationary objects don't mix: https://www.wired.com/story/tesla-autopilot-why-crash-radar/
Yup. This is why I won't get into a self-driving car until they're fully driverless.
Drivers become less attentive as the system gets better.
So what is going on here? I’m not a radar expert. Is there some fundamental flaw with this technology that is preventing it from “seeing” all objects under all circumstances? Or does the problem arise from faulty driving logic...perhaps the object detection system reports the object, and for whatever reason the system interpreting that data ignores it?
https://www.tesla.com/en_AU/blog/upgrading-autopilot-seeing-...
But this seems like the hardest thing for radar to see. Moving cars are easy to distinguish from the ground because they have a different doppler shift, but a stationary object does not.
So the question still stands. Are all of these object detection technologies fatally flawed, or is it the software that is interpreting this data that is to blame?
I disagree. AP is ready for very common situations and acts faster/better than a human driver there (e.g. rear-end collisions). It's not as capable of judging some extreme situations as a good human driver yet. We'll have to fix roads in some situations, but we'll also all benefit from having at least collision avoidance assistance in the cars.
AP will not be ready for many years, and it will get a bad reputation because of companies like Tesla or Uber that push things that are not ready.
Or on this page[2], Specifically the section "Full Self-Driving Capability" makes it sound like that car can completely drive itself.
[1] https://web.archive.org/web/20160726173047/https://www.tesla...
[2] https://web.archive.org/web/20171116234123/https://www.tesla...
2) How did the radar /camera not see the barrier.
3) Give their update capability I'm surprised this section of road hasn't been fixed or updated to stop 'autopilot' for this section.
Why isn't every Tesla tracking all the lanes it can see, and the behaviour of every option at every exit, and flagging cases where one of the other lanes does something weird, or dangerous, half a mile later?
I had a funny brain-wobble whilst watching the video, playing “hunt for the lane”.
I guess the car decided on the wrong one, with fatal consequences.
I'm a bit confused that self driving cars (at least Teslas) appear to ignore that. Even if they were right and it was the correct lane, isn't following it more dangerous than staying in sync with traffic?
I'm surprised there aren't a marked (hatched) area at the junction though. But regardless, the car has to be able to handle even mis-marked or temporary roadwork.
http://www.highwaycodeuk.co.uk/uploads/3/2/9/2/3292309/rule-...
In Britain you can contest parking fines if the double yellow lines are patchy, because arguably you could have missed them. There's also (I think) some laws that say that in order to enforce fines regions must be correctly marked, so the council has a duty to keep them well-painted. Seems like Tesla would argue similarly here.
The government has some limited duty to keep roads intact but you cannot expect perfect road conditions at all times. It would've been different if there had been an object on the road that the police knew about but chose not to remove.
But it appears the combination of lines and narrow barrier and the curve don't get properly detected by the car's algorithm. It's an edge case, but with the state of roads in the US and the lack of maintenance, I don't see how an algorithm can handle every possible case. I know lane markings are worn and faded all over my town.
https://en.m.wikipedia.org/wiki/AI-complete
[1] One example of how things might change is if societies decided to build "smart roads" to assist AI drivers, and then banned non AI drivers from them.
> Our data shows that Tesla owners have driven this same stretch of highway with Autopilot engaged roughly 85,000 times since Autopilot was first rolled out in 2015 and roughly 20,000 times since just the beginning of the year, and there has never been an accident that we know of. There are over 200 successful Autopilot trips per day on this exact stretch of road.
Do we know how many of those trips were made with the same software version as the victim's? And do we know how many of those trips were in that particular lane?
[0] https://www.tesla.com/blog/what-we-know-about-last-weeks-acc...
I don't think Tesla's capabilities are the issue, their marketing is. They should've never called it Autopilot if it's just the standard adaptive cruise control + lane assist that nearly all manufacturers offer.
But, when the car detects these things and decides to disregard them, does it just forget about them? Just thinking back to the Tesla car that ended up going under the truck crossing in its path, supposedly the car didn't react as you would expect because it assumed it was an overhead sign. But I would think that an overhead sign would begin to look different than a truck on the road crossing in front of you as you continue to get closer to it, is the software not capable of at first deciding to disregard something up ahead but then change its mind as it gets closer? Likewise with a concrete stanchion that keeps getting closer and closer? Does the software allow for it to second guess itself?
Fundamentally, though, that does seem like the long route to self driving. Being able to not drive into the proverbial wall seems like Job #1 of a pilot system. Tesla need to be careful they're not not creating a reputation for themselves at this point.
2 after 100 or other better number of cars passed that point and the maps are updated enough AP can engage
3 if radar+camera detects something that does not match the maps it slows down and asks the human to take over, the maps are used as a way to double check what the AI wants to do.
This system will not work on small roads with few Tesla cars on them but it will prevent problems on big roads and collect useful data to add to the unit tests
My idea was how they could act on this situations with the drive assist tech(stupidly named autopilot) I am not of the opinion that true self driving cars are around the corner, not even in major US cities.
In either case, I think the biggest(and insurmountable) obstacle to self-driving cars will be the fact that they rely on having clear signs, clear markings, clear lanes, signs not rotated by the wind or vandalised - and it's going to be impossible(due to cost) to maintain our infrastructure to the required level. Yes, Tesla can patch their system to work in this specific road situation - but there will be countless others.
VW writes:
> The Lane Assist system is like a friendly co-driver. If it senses your car is drifting out of your driving lane it gently counter-steers the car back on line.
They explicitly say that it only corrects you, it doesn't replace steering by the driver.