Tesla crash in September showed similarities to fatal Mountain View accident
abc7news.com
abc7news.com
1. Ethical. One thing is that you do a stupid thing and die. Another is that a technology fails in trivial circumstances that in theory are avoidable. A doctor can make also errors, but a medical device is required to be very safe and reliable in what it does.
2. Of wrong comparisons: you should check the autopilot against a rested and concentrated driver that drives slowly and with a lot of care. Since in the statistics otherwise you do not account for the fact that when you drive, you are in control, and you can decide your level of safety. Such a diligent driver that goes to crash because of an immature software is terrible.
3. And lack of data: AFAIK there is not even enough data publicly available that tells us what is the crash rate of Teslas with auto-pilot enabled vs Teslas without auto pilot enabled, per kilometer in the same street conditions. That is, you need to compare only in the same streets where the auto pilot is able to drive.
Autopilot will rule the world eventually and we will be free of using the time differently (even if this was possible 100 years ago already stressing on public transportation instead of spending the money to have a car each one... this is sad. Fortunately in central Europe this happened to a degree, check for instance North Italy, France, ...). But till it is not ready, to plug it as a feature in a so immature way just to have an advantage over competitors, is terrible business practice.
This needs elaboration or citation. You're one of the only folks I've seen come straight out and make this point: that somehow a technology or practice that is objectively safer can still be "unethically safe" and should be shunned in favor of less safe stuff.
I don't buy it. This isn't individual action here, there's no utilitarian philosophy rejection that's going to help you.
In fact, medical devices seem actually counter to your argument. No one says an automatic defibrillator's decisions as to when to fire are going to be as good as a human doctor, and in fact these things have non-zero (often fatal) false positive rates. But they save lives, so we use them anyway.
Bottom line, that point just doesn't work.
This, of course, ignores the fact that stupid choices drivers make tend to affect other people on the road who did nothing wrong, so the introduction of a self-driving car which makes less stupid decisions would reduce deaths from both categories of people here.
Perhaps. I reject this argument though. A death is a death.
And if you prevent lifesaving technology then you are responsible for a whole lot of deaths, irregardless of how "deserved" those deaths are.
A death caused by shitty engineering (e.g. Tesla in this case) is not the same as a death caused by gross negligence on ones own part.
>And if you prevent lifesaving technology then you are responsible for a whole lot of deaths, irregardless of how "deserved" those deaths are.
How many deaths has Tesla prevented, which would have happened otherwise?
Expect in this case, the driver has 100% control.
If you give every shitty driver a self driving Tesla maybe you would do something to make roads safer, but if you’re just giving it to higher net worth individuals who place greater value on their own life, you haven’t even made a dent in traffic safety.
In fact in some cases all you’re doing is making drivers unsafer because the autopilot encourages them to not pay attention to the road no matter how much you think they are watching carefully. The men killed in Teslas could have all avoided their deaths if only they had been paying attention. If I see a Tesla on the road I stay the hell away lest it do something irrational from an error and kill us both.
I do see some sources that claim rich divers get better insurance rates, but it is unclear to me if that is due to driving skill or a number of other factors that increase rates like liklyhood of being stolen or miles driven.
The third paragraph though is just you flinging opinion around. You assert without evidence that a Tesla is likely to "do something irrational from an error and kill us both" (has any such crash even happened? Seems like Tesla's are great at avoiding other vehicles and where they fall down it tends to be in recognizing static/slow things like medians, bikers and turning semis and not turning to avoid them). I mean, sure. You be you and "stay the hell away" from Teslas. But that doesn't really have much bearing on public policy.
There are plenty of people making that case. See for example this piece in The Atlantic the other day (https://www.theatlantic.com/technology/archive/2018/03/got-9...) talking about the concept of moral luck in the context of self driving cars. It puts the point eloquently.
As to why we use certain technologies, that is not so clear cut either. For instance, if I have a heart attack and someone uses a defibrilator on me- at that time, that is not necessarily my choice. I'm incapacitated and can't communicate and if I die at the end there's no way to know what I would have chosen.
Not to mention- most people are not anywhere nearly informed enough to decide what technology should be used to save or protect their lives (for instance, that's why we have vaccine deniers etc).
Then these people should stop trying to force me to accept a less safe method of transportation, by preventing me from using new technology!
Yeah, those people shouldn't be forced to use a self driving care. As they are ALREADY not being forced to do.
Nobody is being forced to use the technology. Just don't use it if you don't like it.
It is literally the opposite. These other people are forcing ME into a more dangerous situation.
I'm all for less red tape.
For the record, that's my objection to the technology: that it's not yet where the industry says it is and it risks causing a lot more harm than good.
Another point. You say nobody is being forced to use the technology. Not exactly; once people start riding self-driving cars then everyone is at risk- anyone can be run over by a self-driving car, anyone can be in an accident caused by someone else's self-driving car, etc.
So it's a bit like a smoker saying nobody is forced to smoke- yeah, but if you smoke next to me I'm forced to breathe in your smoke.
Yes, we would also tolerate Teslas if they were critical life support technology. How many lives has it saved BTW?
>and in fact these things have non-zero (often fatal) false positive rate
What is non zero? Ten to the power -20?
Unfortunately, this report seems to have shot itself in the foot by apparently using 'Autopilot' and 'Autosteer' interchangeably, so it leaves open the possibility that the Autopilot software improves or adds features to the performance of fairly basic mitigation methods, such as in emergency braking, while having unacceptable flaws in the area of steering and obstacle detection at speed. In addition, no information is given on the distribution of accident severity. It is unfortunate that this report leaves these questions open.
Even if these claims are sustained, there are two specific matters in which I think the NHTSA has been too passive: As this is beta software, it should not be on the on the public roads in what amounts to an unsupervised test that potentially puts other road users at risk. Secondly, as this system requires constant supervision by an alert driver, hands-on-the-wheel is not an acceptable test that this constraint is being satisfied.
[1] https://static.nhtsa.gov/odi/inv/2016/INCLA-PE16007-7876.PDF
http://www.safetyresearch.net/blog/articles/quality-control-...
https://jalopnik.com/feds-cant-say-why-they-claim-teslas-aut...
They could be more useful with -- for now -- less features. They probably won't do it because they want some sacrificial beta testers to collect some more data for their marginally less crappy next version. But given the car does not even have the necessary hardware to become a real self driving car (and that some analysts even think Tesla is gonna close soon) the people taking the risks of being sacrificed will probably not even reap the benefits of the risks they have taken, paying enormous amount of money to effectively work for that company (among other things).
It’s not a self-driving car. It’s really an advanced cruise control.
Why do we keep referring to something that we understand should require human supervision as "auto"? Stop the marketing buzzfeed and let's be real.
The point being that in the future the car _could_ get "full self-driving capability" via a software update. In contrast, a car that doesn't have the necessary hardware will never be fully self-driving even if we do develop the necessary software to accomplish that in the future.
While the Tesla spokespeople are good at saying it's driver assist, their marketing people haven't heard - https://www.tesla.com/autopilot/. That page states "All Tesla vehicles produced in our factory, including Model 3, have the hardware needed for full self-driving capability at a safety level substantially greater than that of a human driver." but as I noted above, we don't know how much of that safety should be attributed to the human. Tesla apparently knows when the driver's hands are on the steering wheel and I presume they can also tell when the car brakes, so they may have the data to separate those statistics. At a minimum, their engineers should be looking at every case where the autopilot is engaged but the human intervenes. They should probably also slam on the brakes (okay ... more gently) if the driver is alerted to take over but doesn't.
As an aside, just the name "Autopilot" implies autonomy.
This is perhaps a case of purposely confusing marketing. All vehicles have the hardware for full self-driving capability but not yet the software. The full self-driving is to be enabled later on through an over-the-air software update.
This is Tesla's big lie.
In all their marketing, Tesla is comparing crash rates of their passenger cars to ALL vehicles, including trucks and motorcycles, which have higher fatality rates. Motorcycles are about 10x-50x more dangerous than cars.
Not only that, they aren't controlling for variances in driver demographics - younger and old people have higher accident rates than middle-aged Tesla drivers - as well as environmental factors - rural roads have higher fatalities than highways. Never-mind the obvious "accidents in cars with Autopilot" vs "accidents in cars with Autopilot on".
If you do a proper comparison, Tesla's Autopilot is probably 100x more fatal than other cars. It's a god-dammed death trap.
And remember, there are several extremely normal cars with ZERO fatalities: https://www.nbcnews.com/business/autos/record-9-models-have-...
This is not a problem that will be solved without fundamental infrastructure changes in the roads themselves. Anyone that believes in self-driving cars should never be employed, since they don't know what they're talking about.
I do believe "it's still safer than the average car" because there's not a big tank of explosive - I'm most curious to hear what caused such a massive fire in this crash. But you're talking about the autopilot and it's statistically incorrect to say it's safer than the average car. It's merely safer than a driver alone - this should be no surprise as you'll find that cars with backup cameras and alarms don't hit have as many accidents while in reverse as cars without them.
However, I would think that testing Autopilot-alone seems impractical. It's been asserted that AP has no ability to react to or even detect stationary objects in front of it. Can't we assume that in all those cases, non-driver intervention would result in a crash?
[0] https://www.tesla.com/blog/update-last-week%E2%80%99s-accide...
Almost every driver thinks they're better than average.
Even when it's a stupid person dying from their stupidity, it's still a tragedy.
I really think data driven analysys is the way to go. If we can get US car fatalities from 30000 a year to 29000 a year by adopting self-driving cars, that's 1000 lives saved a year.
Agree with your point #3. If Tesla autopilot only works in some conditions, its numbers are only comparable to human drivers in these same conditions.
What this ignores is that self-driving cars will by and large massively reduce the number of 'stupid' drivers dying (the ones who are texting and driving, drinking and driving, or just simply bad drivers) but may cause the death of more 'good' drivers/innocent pedestrians.
So the total number could go down, but the people who are dying instead didn't necessarily do anything deserving of an accident or death.
I say this as someone who believes self driving cars will eventually take over and we need to pass laws allowing a certain percentage of deaths (so that one at case of the software being at fault doesn't cause a company to go under), but undeserved deaths are something people will likely have to deal with somewhere down the line with self driving cars. At the very least, since they're run by software they should never make the same mistake twice, while with humans you see the same deadly mistakes being made every day.
I think that’s a win, even if I now have an even statistical chance to be in the 1001 and no chance to be in the 2001.
Requiring that I be in 1001 is not ok, no more than requiring I donate all my organs tomorrow. Allowing that I might be in the 1001 is ok, just a registering for organ donation is.
You're saying that auto-driving would save the lives of 1000 people who would have died without it, by causing the death of another 1001 that wouldn't have died if it wasn't for auto-driving?
So you're basically exchanging the lives of the 1001 and the 1000? That looks a lot less of a no-brainer than your comment makes it sound.
Not to mention, the 1001 people who wouldn't have died if it wasn't for auto-driving would most probably prefer to not have to die. How is it that their opinion doesn't matter?
People claim seatbelts have caused lots of deaths, and I'm sure at least a percentage of these claims are fair ([0]). I still think it's safer to drive a car with a seatbelt rather than without.
But all self-driving car victims (will) have done nothing wrong. Whethere they were riding in the car that killed them or not, they were not in control of it, so they 're not responsible for the decision that led to their deaths.
Unless the decision to go for a walk or a cycle, or to ride on a car makes you responsible for dying in a car accident?
We already know that there is an area between drive assistant and automatic driving where people just can't keep their attention level where it needs to be. Driving is activity that maintains attention. People can't watch the road hand on the wheel when nothing happens and keep their level of attention up when the car is driving.
The way I see it, the biggest safety sin from Tesla is having experimental beta feature that that is known weak point for humans. Adding potentially dangerous experimental feature, warning about it, then washing your hands about it is not good safety engineering.
The news story points out how other car manufacturers have cameras that watch the driver for the signs of inattention. Driving assistant should watch both the driver and the road. You can't have a driving assistant that can be used as an autopilot.
>you should check the autopilot against a rested and concentrated driver that drives slowly and with a lot of care. Since in the statistics otherwise you do not account for the fact that when you drive, you are in control, and you can decide your level of safety. Such a diligent driver that goes to crash because of an immature software is terrible.
I think you overestimate the rationality of human beings. I commute to work by motorcycle every day and I've noticed that I tend to ride more dangerously (faster, "dirtier" passes etc...) when I'm tired, which rationally is stupid. I know that but I still instinctively do it, probably because I'm in a rush to get home or because I feel like the adrenaline keeps me awake.
This is an advantage of autonomous vehicles, they can be reasonable when I'm not. I expect that few drivers (and especially everyday commuters like me) constantly drive "slowly and with a lot of care". A good enough autonomous vehicle would.
https://www.tesla.com/blog/putting-tesla-hepa-filter-and-bio...
https://news.ycombinator.com/item?id=11617945
I think they are not testing with small enough particles. In the article, they test with PM 2.5 particles which would be around 2.5 micro meters. However, if you look at the table on page 11 of
http://multimedia.3m.com/mws/media/409903O/respiratory-prote...
Potential bio weapons such as smallpox, anthrax, influenza and the hemorrhagic viruses are far smaller than 2.5 microns.
Also, there are probably issues with the sensitivity of the detection equipment. If you look at the table on page 8 of
https://www.ll.mit.edu/publications/journal/pdf/vol12_no1/12...
And at the table at
http://www.siumed.edu/medicine/id/current_issues/bioTable2.p...
You will see that some of the biological agents can cause infection with as few as 10 particles. I doubt that the Tesla equipment could detect a concentration of 10 particles of these sizes.
This article is basically the biological equivalent of the I can't break my own crypto article.
>Bioweapon Defense Mode is not a marketing statement, it is real.
is false. Extraordinary claims require extraordinary evidence, and the evidence of Bioweapons Defense Mode working is entirely lacking
HEPA filters capture particulates. PM2.5 means particles above 2.5 micrometers in diameters. Good 0.2 - 0.3 micrometer HEPA filter is fine enough to catch bacteria like anthrax. Smallpox, influenza virus are smaller. You need carbon absorber to be safe.
This is like thinking you will survive a storm in the middle of the ocean because you have a life jacket on you.
>"We’re trying to be a leader in apocalyptic defense scenarios," Musk continued.
Is this guy serious?
[1] https://www.theverge.com/2015/9/30/9421719/tesla-model-x-bio...
Functionality that can _most_ of the time drive itself without human intervention and occasionally drives itself into a divider on the highway seems like a callous disregard for human life & how such functionality will be used.
Sure, everything is avoidable, if there's some expectation it needs to be avoided.
The whole point of autopilot is to avoid focusing on the road all of the time. So setting up circumstances under which humans perceive the functionality (autopilot) behaving as expected most of the time, then its highly likely they'll treat it as such & will succumb to a false sense of security.
My point: when a feature is life threatening your marketing fluff shouldn't deviate significantly from your legal language.
You raise an interesting point, accidents are not normally distributed throughout the driver's day, or even the population. Your likelihood of having a crash with injuries is highly correlated with whether or not you've had one before. A substantial number involve alcohol, consumed by drivers previously cited for DUIs.
We keep using average crash statistics for humans as a baseline, but that may be misleading. Some drivers may actually be increasing their risk by moving to first gen self driving technology, even while others reduce their risk.
On the other hand, we do face a real Hindenburg threat here. Zeppelins flew thousands of miles safely, and even on that disaster, many survived. But all people could think of when boarding an airship after that was Herbert Morrison's voice and flames.
I have already heard people who don't work in technology mumbling about how they think computers are far more dangerous than humans (not because of your nuanced distinction, but simply ignoring or unaware of any statistics).
I worry we're only a few high profile accidents away from total bans on driverless cars, at least in some states. Especially if every one of these is going to make great press, while almost no pure human accidents will. The availability heuristic alone will confuse people.
I'm not sure I follow you here. Are you saying that because humans are known to be falible, but technology can be made nigh-infalible, that technology should be held to a higher ethical standard?
Those two statements don't connect for me.
I suppose that's partly because I am an automation engineer, and I deal a lot with inspecting operator-assembled and machine-assembled parts. If the machine builds parts that also pass the requirements, it's good.
It's nice if it's faster or more consistent, and sure we can build machines that produce parts with unnecessarily tight tolerances, but not meeting those possibilities doesn't feel like an ethical failing to me.
Yes, not infallible, but I believe that to take our lives in the hands of machines, the technology must be at least on par with the best drivers. To be better than average, but for instance to be more fallible than a good driver, is IMHO still not a good standard to be ethically ok to sell self driving cars, even if you have 5% less death per year compared to everybody driving while, like, writing text messages in their phones. If instead the machine will be even a bit better than an high standard driver, driving with care, at that point it makes sense because you are not going to care about the distribution of deaths based on the behaviors.
To be fair I rarely was doing anything useful on my 40 min train commute either :) Mostly reading Hacked News on my phone. Now I'm at least looking into the distance, taking some strain from the eyes.
I totally agree that public transport has to get better, it's just that there always has to be a mixture of transportation options.
Can the reason for these be that many other people use public transport instead and thus the roads are way less busier?
As we have no control over how others drive, statistics are more relevant.
As a pedistrian, I care about cars being on average less likely to kill me. If it means I’m safer I would rather the driver wasn’t in control of their own safety.
The ethical solution is the one with the least overall harm.
Of course being safer overall while taking control from the driver is unlikely to drive sales.
Do these car companies test their software by just leaving it on all the time in a passive mode and measuring the deviance from the human driver?
I'd think that at least in MV case, you'd see a higher incidence of lane following difference at this spot and it would warrant an investigation.
Something like this isn't easy but for a company as ambitious as Tesla it doesn't sound unreasonable. Such a crash with such an easy repro should have been known about a long long time ago.
So, cruise control? If people got confused and thought that cruise control was more than it really was in the 80s, or whenever it came out, what would we have done?
Makes me wonder how this gets resolved without jumping through the gap to 'perfect' autopilot.
> To put it briefly, automation has made it more and more unlikely that ordinary airline pilots will ever have to face a raw crisis in flight—but also more and more unlikely that they will be able to cope with such a crisis if one arises.
https://www.vanityfair.com/news/business/2014/10/air-france-... (linked to from another HN post today: https://news.ycombinator.com/item?id=16757343 )
Tesla is just handing it out to anyone who can afford to buy a Model S/X (and now Model 3) with the absolute minimum of warnings that they can get away with.
And also the idea that an automatic driver is easier to improve over time (skewing the trade-off better and better) than human drivers. Which of course will snowball when reliance on automation keeps making drivers worse than better, and at some point you're past the point of no return.
Whether that's right data must show.
The UK has the 2nd-lowest rate of road deaths in the world (after Sweden).
The roads in the UK are not intrinsically safe, they are very narrow both in urban and rural areas which means there are more hazards and less time to avoid them.
However, the UK has strict driver education programme. It is not easy to pass the driving test, with some people failing multiple times. It means that people only get a license when they are ready for it. Drink-driving will also get you a prison sentence and a driving ban.
Any statistics released by Tesla should be compared against similar statistics from say modern Audis with lane-assist and collision detection.
"but autopilot still lowers the death rate on average"
That's not what they said, they said the death rate was lower than the average. And yet you can't help hearing that it lowered the death rate. I think it's very likely turning on autopilot massively increases the rate of death for Tesla drivers, but they've managed to deflect from that so skilfully.
I think the question of using self driving vehicles comes down to this. Do you want to be part of a statistics that you can control, or do you want to be part of one that you cannot?
When we go in trains or planes, we are already being part of statistics that we cannot control. But those things also are extremely reliable.
So there seems to be a threshold, where someone should opt for being part of a statistics that you cannot control.
The people who are pushing SDV's, by some sleight of hand, seem to hide this aspect, and have successfully showcased raw, (projected) statistics that implicitly assumes the rate of progress with SDV's, also assume that SDV ll continue to progress until they reach that capability.....
>And also the idea that an automatic driver is easier to improve over time (skewing the trade-off better and better) than human drivers.
But there is always are risk of catastrophic regressions [1] with each update, right?
I really hope it does get to the point were all drivers are automatic and interconnected. Cars could cooperate to a much higher extent and traffic could potentially be much more efficient and safe.
It is certainly designed to do that, but even for airline pilots there are limits to what is possible.
You cannot train a human to react within 1ms; that's just physiologically impossible. Nor can you train a human to fully comprehend a situation within x ms, where x depends on the situation.
So the autopilot would have to warn a human say 2x ms before an event that requires attention, where it can of course not yet know of the event, so that amounts to 'any time there could possibly be an event 2x ms in the projected future'. Which is probably: most of the time. Making the autopilot useless.
This is a similar situation to nuclear power. With nuclear, all your waste can be kept and stored and nuclear has almost zero deaths per year. Contrast to the crazy abomination that is coal burning.
I don’t think it has anything to do with technology and automation. People are just very bad at reasoning with probabilities. This is why lotteries are so popular. And it is fuelled by unscrupulous (and even less apt at grasping probabilities) media trying to make money out of sensionalist headlines. How many times a week am I told that x will cause the end of the world, replace x by flu pandemics, Ebola, global warming, cancer as a result of eating meat or using smartphones, etc.
That is, we have an accepted safety standard - by definition, the worst human driver with a license. If a Tesla is safer than that, the rest is theoretically preferences and economics.
I'm not saying that the regulators or consumers will accept that logic - if airlines are any example to go buy, they'll take the opportunity to push standards higher - but I think the point is interesting and important. It is easy to smother progress if we don't acknowledge that the world is a dangerous place and everything has elements of risk.
I think the issue with nuclear power is more the size of the blast radius. For example, the U.S. population within 50 miles of the Indian Point Energy Center is >17,000,000 including New York City [1]. The blast radius of a self-driving car is a few dozen people at best. It's not evident that nuclear technology is uniformly better than solar or wind, but it's probably expected-value positive.
[1] https://en.wikipedia.org/wiki/Limerick_Generating_Station
Exactly this. People will not accept being hurt in accidents involving AI, but will accept being in accidents caused by human error. There is no way this will change, and car manufacturers should also have realized this by now.
As far as I can see, the only viable solution to the "not quite an autopilot" problem that tesla (and maybe others) has created for themselves is this: Just make the car sound an alarm and slow down as soon as the driver takes his eyes off the road or his hands off the wheel. The first car that doesn't do this should be one that is so good it doesn't have a driver fallback requirement at all - which I think is two decades out.
The flawed assumption behind this thinking is that all accidents have "accidental" factors. Alcohol and drugs and the poor decisions of young men are _huge_ components of that statistic. You also have to consider that motorcycles make up a not insignificant portion (12%) of those numbers.
It also completely ignores the pedestrian component (16%) which is due to a huge number of factors, of which, driver attentiveness is only one. So many pedestrians die, at night, on the sides class B highways that it's silly.
EDIT: To add, Texas has more fatalities on the roadway than California. Not per capita, _total_. This is clearly a multi-component problem.
So in fatalities statistics the fair thing is to compare Tesla with similar cars.
The "current climate" is a backlash to the cavalier "move fast and break things" externality-disregarding culture of the last 10 years or so. We should be extremely conservative when it comes to tools that can potentially maim or kill other people. Not valuing human life should be an aberration.
Is it? Can it be proved controlling for all variables?
It’s like if you said “I don’t get why people are concerned with every kid bringing a gun to school, if that would make the schools 100 times safer then shouldn’t we do it? Even if there’s an interim “learning period” where they are much less safe, doesn’t the end justify the means?”
I really can’t fathom the mindset of people who honestly believe that the only way forwards with self-driving tech is to put it into the market prematurely and kill people. In 10-15 years we’ll have safe robust systems in either case, let’s not be lenient with companies who kill costumers by trying to move to fast and “win the race”. Arguing from statistics about deaths without taking responsibilities into account is absurd. Would it be ok for a cigarette company to sell a product that eliminated cancer risks but randomly killed costumers as long as they do the math and show that the total death count decreases? Hell no.
Your opinion will be irrelevant--the insurance companies get the final say. When that safety point arrives, your insurance rates will change to make manual driving unaffordable.
I see people saying that. Why would insurance be more expensive than today--implying that manual driving becomes more dangerous once self-driving cars become common? It's already a common pattern that assistive safety (and anti-theft) systems tend to carry an insurance discount versus not having them.
including
> There is additional controversy, it should be noted, about the proposed level 2. In level 2, the system drives, but it is not trusted, and so a human must always be watching and may need to intervene on very short notice -- sub-second -- to correct things if it makes a mistake. [...] As such, while some believe this level is a natural first step on the path to robocars, others believe it should not be attempted, and that only a car able to operate unsupervised without needing urgent attention should be sold.
Defining safe errors might be a challenge though.
Tesla has been criticised all along for both its marketing and allowing long periods of unattended driving assistance.
Train drivers are the closest thing and they have a lot of equipment that forces them to stay alert.
Personally I think that autopilot should only take over when the driver is unwell or that it thinks there will be a crash.
Even when driving full time drivers get distracted. So when the car is doing the driving humans will get bored and won't be able to respond in time.
Autopilot however is always on and never distracted but doesn't work in some cases.
As does Cadillac's Super Cruise system which is not getting the respect it should -- including by GM itself.
Except they sometimes fall asleep: http://www.bbc.co.uk/news/uk-england-london-39457148
> http://www.spiegel.de/panorama/justiz/zugunglueck-von-bad-ai... (german only)
Either it requires the driver to occasionally take over and handle a situation on short notice, then the driver should have hands on the wheel and eyes forward (and the car should ensure this by alerting and slowing down immediately if the driver isn't paying attention).
Or the driver isn't required to pay attention and take over occasionally, and then it's fine to reply to that email on the phone while driving. I see no future in which there is a possible middle ground between these two levels of autonomy.
That might not be worth the full cost of autopilot (particularly compared with lane keepers), but it's a thing.
I should not need to trust you -- if I trusted you, why would we need autopilot at all? Humans are either qualified to drive cars or they aren't. If they are qualified, we don't need autopilot. If they aren't qualified, autopilot should never require human intervention. What we have now is a half-measure that assumes that neither humans nor autopilots can be trusted, but that some combination of those two untrustworthy parties can somehow be trusted. It doesn't make any sense.
[0] https://jalopnik.com/two-years-on-a-father-is-still-fighting...
edit: corrected a typo
Whether you trust others is immaterial; statistics will be the final arbitor. If Autopilot still causes fatal accidents, but fewer fatal accidents than humans alone, how could you argue against such a safety system? What of the lives saved that wouldn’t have been if we demand an entirely fault proof system prior to implementation? Who are you (not you specially, but the aggregate) to take those lives away because of irrationality?
This claim was immediately called into suspicion when it was first published, and the NHTSA is currently facing a FOIA lawsuit for refusing to release data to independent researchers.
http://www.safetyresearch.net/blog/articles/quality-control-...
https://jalopnik.com/feds-cant-say-why-they-claim-teslas-aut...
So does any car with automatic emergency braking and/or or forward collision warning (links below).
The problem here is, this Tesla drove head-on into the gore point. And previously, it drove right into the side of a huge truck. So... can it be trusted? Your call.
https://www.consumerreports.org/car-safety/automatic-emergen...
" IIHS data show rear-end collisions are cut by 50 percent on vehicles with AEB and FCW. "
It may be irrational, but I can forgive a human, I cannot forgive an AI.
I suspect the unfortunate reality is people die on the journey to improvement. Once we decide this technology has passed a reasonable threshold, it now becomes a case of continual improvement. And as horrible as this sounds this is not new. Consider how much safer cars are than 20 years ago and how many lives modern safety features save. Or go for a drive in a 50 year of sports car and see how safe you feel. And in 50 years we'll look back at today's systems in the same way we look at the days of no seat belts.
The very challenging, and somewhat unusual given the life/death stakes, aspect of this domain in general is that general 'easy' logic works for 99% of the scenarios. But in the remaining 1% -- and I mean every single possible last scenario, whether probable or not -- the "what to do" is so much more difficult. Enough time to do it? Can be done safely? Etc.
The reason the car turns control back over to the human at all is because the vehicle presumably does NOT have any maneuvers left which are high-confidence -- whether that's because the sensing equipment failed, the software failed or failed to interpret the data satisfactorily, or there's just no "good move" left, or any other number of reasons, etc...
seriously, i slow down when i am unsure of the conditions, i surely have never evaluated every last scenario, and i have never run into a concrete barrier.
OTOH, the car has probably saved me from at least one accident where traffic ahead of me suddenly slowed down right as my attention relaxed. The _only_ correct way of using these systems is to treat them as an extra level of safety on top of your responsibilities as a driver. An "extra set of eyes" to help you avoid an accident while leaving you as the primary driver of the vehicle.
Human+automation is very powerful! However, I don't think we are far from self-driving beating human alone.
It's not perfect but it sure beats the driver being so used to it they start doing other things like watching movies on their laptop.
30 seconds is a HUGE amount of time to react on a highway. You'll be long dead by the time another 25 seconds pass if the accident happens just 5 seconds after you took your hands off the wheel. And this is exactly what happened in the recent crash.
Also, if Tesla intends to shield itself behind a beta status for their autopilot system until it is perfect, I think this beta status will remain even longer then the time GMail was in beta. At least, in this case they should own this problem and somewhat hardcode a meaningful warning at such intersections or do something.
The question seems to be which is better:
1) A car very intelligently helping a human drive better.
2) Or a car mostly driving well itself but needing humans to very rarely override fatal crashes.
I prefer 1)
I'm not sure there are any studies comparing these things, since "human driving but lots of extra tech help" is harder to quantify and doesn't have all the data going back to the mothership like Tesla.
If they want beta software on a private track, that's their (and their insurance company's) problem.
Different environment (and stakes), but I observed the same thing happening a couple of times during IT incident response. The automation crashed/failed/got into a situation it could not automatically fix, and the occurrence was rare enough that people actually panicked and it took them quite a while to fix what in the end was a simple issue. They just didn't have the "manual skill" (in one case, the tools to actually go and solve the issue without manually manipulating databases) anymore.
I dont trust anything from Elon anymore, big talker and lots of hype.
Not to mention there are things most people do defensively while driving that autopilot doesn't: anticipate a vehicle coming into my lane by both looking at the wheel position and the lateral movement. Autopilot ignores those pieces of information and data until it's in basically in your lane.
Personally I feel I have to be more on guard and attentive with it on because I know their are fatal bugs.
It seems like AutoPilot users take their hands off the wheel regularly, and I just don't see how that is safe with this type of system.
Isn't it true that the thing cannot detect a stationary object in its path if the vehicle is traveling at above 50kph? If that is true, then I think this is an extremely dangerous situation that the owners of these vehicles are in...
That's what 'too good' is supposed to be.
The solution is easy, but probably not what many would like to hear: ban all "self-driving" or "near-self-driving" solutions from being activated by the driver, and only allow the ones that have gone through very rigorous and extensive government testing (in a standardized test).
When the government certifies the car for Level 4 or Level 5, and assuming the standardized test has strict requirements like 1 intervention at 100,000 miles or whatever is something around 10-100x better than our best drivers, then the car can be allowed on the road.
Any lesser technology can still be used in other cars, bit instead of being a "self-driving mode" it should actually just assist you in the sense of auto-braking when an imminent accident is about to happen, or maybe just warn you that an imminent accident is about to happen, which could still significantly improve driver safety on the road until we figure out Level 5 cars.
Street View: https://www.google.com/maps/@37.6346515,-122.104109,3a,75y,8...
Dirt, snow, rain, fog, dark, and other common conditions could obscure even correctly painted lines, and it just isn't realistic to expect perfect roads throughout such a large network anyway (in particular as human drivers can infer the line is meant to be there from the neighboring lines).
At its core, a system like Autopilot is meant to stand in for humans. Plus in this video where they reproduced the accident there was a full chevron pattern so the argument is flawed/wrong anyway: https://www.youtube.com/watch?v=6QCF8tVqM3I
Considering Tesla's must have driven through there hundreds of thousands of times, they should have data to make sure the car at least follows the highway. I don't crash in to a barrier when a paint stripe gets scuffed in between my Monday morning commute and my Tuesday morning commute.
It's even worse: wait until someone does something malicious to an autopilot car to cause it to crash/kill.
I hate to be so cynical, but if you're developing this stuff, you have to imagine worst case scenarios. And there are much much worse scenarios when malice is involved.
The same suggestion as before, autopilot needs to be turned back into learning only. It never should have left. There were way too many videos just on youtube showing how bad it could be. It should simply be recording what it would decide to do versus what the driver did and flag every exception. It certainly can see barriers and its routines could save every instance where it might have hit something.
The NHTSA really should demand that Autopilot features be disabled. There are even cases of it failing to park without hitting objects.
Musk should be lauded for his EV push, it opens doors to all sorts of new technologies being incorporated in cars and will one remove the need for gasoline/diesel as the primary drive in all cars. However he needs to learn to step back on technologies that are clear cutting edge and not ready for prime time.
At this time features like "AP" should be used not to take over for the driver but instead assist the driver by keeping them from being unsafe. This means doing similar to what blind spot monitors to and lane drift systems work on.
(i) That there is an obstacle fast approaching.
(ii) That all cars in front of it is doing something else -- time to slow down.
(iv) That it has (literally) been down this road before and had to revise its estimate of what it thought it was a lane (i.e. jerk back into the lane once it realized).
(v) That other Teslas have been down this road and learned that.
(vi) That the lane is suddenly getting wider than a freeway lane should, and that doesn't normally happen.
(vii) That the paint is worn, which indicates poor road maintenance, which necessitates caution.
(ix) That it saw a freeway exit sign a short while back, which logically indicates there must be a V-shaped spot ahead somewhere that it shouldn't drive into, and this could be it.
There have been at least two other video near collisions of people driving on that section - it seems dangerous.
This reminds me when I was living in the Midwest and there was this one spot where I had to do a u turn in my commute which was very dangerous because of some tree branches obstructing view of oncoming traffic. One day I almost got into an accident and since it shook me quite a lot, I wrote an email to the city that the tree branch needed to be trimmed so I could see better. Sure enough I got an email back a few days later from a woman saying they “understand my frustrations” or something like that. I thought I did my part and went on with my day. Fast forward about a month, and I almost get in an accident at the same spot, man I was fucking furious that my communication with city was just “ohh we understand” —- instead of employing a sea of men and women telling me that they “understand” why could they have just not just trimmed the tree and improved safety.
If I was still living there I think I would have just gone with my chainsaw and done it myself.
Degraded paint does not a bad road make. Roads are optimized for human drivers that can leverage additional context to make their decisions. Many countries don't even bother to stripe their roads at all.
Wealthy individuals/corporations but poor roads (and other government infrastructure/services) isn’t unexpected, it’s more or less the goal.
California had a lot of fires it had to fight last year. Where did that budget come from?
Streetview:
https://www.google.com/maps/@37.4112791,-122.076645,3a,75y,9...
March '18 was at 101/85
More faded than the last StreetView pics.
I also wonder if the line -- where the asphalt meets concrete -- contributed. Maybe it looked like the shoulder. Granted, there is also a yellow line.
I havent heard of GM killing anyone with their car.
At the very least, the system should slow down significantly and warn the driver if there's not high confidence in where it should be going.
IIHS '17 study puts Autopilot at only reducing 40% accidents which is the same as their '16 study which had the same 40% reduction for any car with Auto Emergency Braking.
That's his shtick though, promise unbelievable things, then when you get halfway there, people don't care you were wrong and/or lying. He understands how to market himself and his ventures.
My guess is he's got too many fans and money behind him. If things ever go truly south, that's when they'll really go after him, because people really care about their money.
Unless I'm missing something, 40% is actually a good result (given what they're trying to measure).
However, I think the more important statistic would be regarding how many injuries/fatalities does it reduce vs cause. Are there any studies on this?
> Full Self-Driving Hardware on All Cars
Followed by this:
> All Tesla vehicles produced in our factory, including Model 3, have the hardware needed for full self-driving capability at a safety level substantially greater than that of a human driver.
http://web.archive.org/web/20180323054727/https://www.tesla....
Getting past the arguments of whether Tesla should be calling this "Autopilot", is its claims about hardware true? Is the failure to detect stationary objects like a firetruck or a gore point solely as a result of currently-inadequate software? Because it doesn't sound like the sensor suite is anywhere near the AVs of Waymo or Uber.
The claim is falsifiable if and only if one can prove the minimum sensor capability required for full self-driving exceeds what they have. But even proving the falseness at a later date doesn't prove malice or fraud unless Tesla knows right now that their claim is not true.
Tesla actually hasn't called that Autopilot -- they refer to it as "full self driving" and not in the same category. But...is it true? It's vaporware right now, so we can't really say.
Obviously, in a purely hypothetical landscape, even with cameras you should be able to emulate what humans can do and have 100% attentiveness. That is to say, humans don't have lidar and radar, yet humans do drive. So, if you can achieve human levels of perfection and have 100% attentiveness, it should be possible to beat out crash statistics based on the fact that many are caused by failure to respond on the human side accurately or fast enough.
This is all hypothetical, though. All of it comes down to computer vision/software, which is really hard to do. It's possible, but not yet done.
Much harder to imagine software surpassing alert human driving capabilities if it's not equipped with sensors offering better depth perception that could compensate for lack of general intelligence to process unfolding situations, visual anomalies etc. Harder still if the optical sensors are objectively worse than human ones in many lighting conditions.
Which camera has the same dynamic range as the human eye?
It might be true, but it's also true that most self-driving researchers prefer to use lidar, not 2D images, because lidar gives better results.
In combination with chevron markings that mean "never ever drive here". Either should be a globally overriding signal to emergency break or turn, not "keep following the white line".
If the answer is "during morning light, it doesn't see it", then they need to take the feature offline until the car is fitted with an appropriate LIDAR; or other technology that can detect stationary obstacles.
The answer to that is that sensors detect, but don't make decisions. The decisions are made by the car AI that takes input from the censors and outputs control signals.
So the pertinent answer is "why is Tesla's AI unable to make the decisions necessary to avoid a big, yellow and black stationary barrier?".
Unfortunately, nobody can answer that, including Tesla engineers, because their cars' AI is trained with black-box machine learning algorithms that are extremely resistant to any attempt to understand how they make decisions.
But, you know. I'm pretty sure they're really safe /s
What is the justification for allowing anything we don't understand to be on the road? I really want the NTSB to ask specifically what decisions the Tesla was making and not accept "black box learning" as an answer.
https://forums.tesla.com/forum/forums/autopilot-handling-sto...
So, to see this barrier, you'd have to guess its distance from 2D images. That's a hard problem. I think this would be called "obstacle detection using stereo vision" or maybe even "monocular vision" if they're using one camera.
There isn't as much research into this field as there is in systems that use lidar because lidar has been proven to work, and academics probably don't care as much about building their products at massive scales, so the cost of lidar doesn't matter as much.
Searching Arxiv for that phrase [1], the top result is a paper called A Joint 3D-2D based Method for Free Space Detection on Roads [2] that uses Lidar. The next result that doesn't mention lidar is Free Space Estimation using Occupancy Grids and Dynamic Object Detection [3].
Following that is Failing to Learn: Autonomously Identifying Perception Failures for Self-driving Cars [4]. In that paper's conclusion, it states its next plan is to use lidar to improve data quality,
> we plan to incorporate free space computation from the path of the vehicle and from active sensor returns like LIDAR to identify false positives to further improve our assessment and understanding of modern object detectors at the fleet level.
[1] https://arxiv.org/find/all/1/all:+AND+vision+AND+stereo+AND+...
[2] https://arxiv.org/pdf/1711.02144.pdf
are they? so why was the barrier crushed in the first place before the crash? and why was it designed to be easily rebuild? its almost like someone knew people will have trouble and will eventually drive into it.
It does.
Btw you wont find this type of barrier in this state of neglect in EU, its illegal, its like setting up a trap to kill people.
> Our data shows that Tesla owners have driven this same stretch of highway with Autopilot engaged roughly 85,000 times since Autopilot was first rolled out in 2015 and roughly 20,000 times since just the beginning of the year, and there has never been an accident that we know of. There are over 200 successful Autopilot trips per day on this exact stretch of road.
Which might be technically true, but conveniently ignores other, very similar crashes.
[1] https://www.tesla.com/blog/what-we-know-about-last-weeks-acc...
if they are not even considering that the driver overrode the system because it was going to crash then all their claims are suspect.
It seems tons of upper middle class to wealthy people are lining up to pay dearly with their wallets and lives to be a billionaire's crash test dummy.
Testing of vehicle safety features, and safety features of many products, is most often done in mass markets. Automatic safety features are being rolled out in essentially every single auto brand. Automatic emergency braking (AEB) is set to be standard in numerous makes in a couple years. Lane control, lane reminders, blindspot detectors, these features are all being rolled out on the roads, to varying degrees of success.
This is a less-safe version of the way drugs are brought to market. First, small-scale tests with cells only, then mice, then healthy people, then target patients, and finally post-market surveillance.
Cars indeed have a MUCH less regulated rollout for their safety features, yet to say we are dummies in their tests is not even close to reality. While drug testing may be more stringent, we also don't have 30-40k people a year dying from "not taking a drug" accidents, which is the situation on our roads today.
Ultimately, it comes down to lives saved. Every time you delay a safety feature, you have to ask the question: is this delay going to lead to loss of life or saving a life? That's the moral quandary the automakers are in, and it seems they go with "release it" much more often than not. That will lead to some EXTREMELY well publicized accidents, and yet we will NOT hear of the countless cases where everything went nominally, and who's to say precisely how many accidents were prevented?
Maybe I bought too much into their PR spin but it sounded as if it was barely a delay.
It's a situation where if you were piloting yourself you're a lot more likely to have mentally mapped that you continue in a straight line at that point, so listing to the left is going to feel WRONG even when suddenly blinded by the sun. But if autopilot takes you there, and you're "lizard brain level" paying attention but didn't have that mental navigation map in your head, and the sun's in your eyes, you're going to take a few seconds to try to figure out how to correct so as not to make things worse. That's enough to hit the gore point in that video.
What do you say to people like me, who, if you took the feature away, would be forced to reduce their safety? It's easy to call for its removal when you only focus on the subset of people who will have their safety increased by removing it (ie, the people, often through no fault of their own, who come to rely upon it more than it is designed for.) What about everyone else that currently benefits from it? Will you force them to go back to driving "solo" to protect those people?
It's not so simple as "Tesla is reckless by releasing this" -- this is providing real safety to people who are using it the way it is intended. Also, the release of autopilot as-is also makes it possible to accelerate the development of full autonomy, which will save lives that otherwise would be lost to accidents if it were to be created later in time. Any analysis which does not account for this dynamic is missing an important piece of the ethics of removing it.
The main question that is worth asking is if there is more Tesla could do to ensure autopilot is used as a co-driver. The name is a poor choice to start.
I would say we should go by the numbers. How many accidents and deaths are there per mile, on similar roads (divided), from vehicles of similar class (new, luxury), in similar weather conditions (fair)? Compare that to miles where autopilot is enabled and we have something.
As yet, I don't believe Tesla is required to report this, and they probably won't voluntarily do it. At best, we would get results from certain states. We would never get national or world level data.
So, we'll have to wait for the NTSB and NHTSA reports and look to their recommendations.
In my view, the ultimate goal should be to ensure that Autopilot is used in the way I described, and is disabled for drivers who are unable to use it that way. That, in theory, would provide the most benefit for the most people. Perhaps that is a fool's errand, either because its technically impossible, very few people use it that way, or literally nobody is able to and anyone who thinks they are is being fooled by their own bias (as I'm sure many readers of this thread assume I am.)
But it seems a goal worth pursuing before giving up and throwing it out because the net result of say, reducing accident fatality likelihood by 50% 5 years ahead of full autonomy results in something like 75k saved lives (in the US.)
Humans are extremely bad at monitoring monotonic things, even if we are trained to do them. That‘s why trains have elaborate dead man switches to ensure the operator is paying attention.
From these accidents it‘s clear that Teslas aren‘t good enough at making sure the driver is paying attention.
And it‘s also pretty clear that these autopilot victims weren‘t aware how distracted they really were. I‘m pretty sure they thought they had everything under control, right up to the point when their car did something stupid.
Unfortunately having the computer takeover has the same issues. If it thinks you're following the wrong lane it could choose to steer you into a barrier, for example.
Chris Urmson's team at Google ran user tests at the start of their autonomous vehicle program and found that users misbehave [1] when behind the wheel of semi-autonomous vehicles, despite having been given instruction to pay attention. We're seeing that scenario play out in slow motion in Teslas and perhaps other driver-assistance programs that get less press.
That's what the NTSB is for.
I travel/drive a good deal in the summer and for a few years now have silently (mostly) muttered to myself that if auto-pilot cars are going to use "painted lines" to navigate - they're "gonna have a bad time".
Poorly painted lines aside, think of how many "wtf" moments you have on the roads around your town - just down the road from my house is a road that just... ends (due to poor planning years ago). It's obvious to human drivers what to do (you're supposed to exit the paved road and drive over a hump to the dirt road that continues), but for auto-pilot? Not so much unless they make use of "data sharing" that would reveal the human solution to this screwed up piece of road.
Actively veering towards stationary barriers in otherwise perfectly safe conditions is not what I would describe as reducing the likelihood of accidents.
What I am noticing right now is that if I were able to afford a Tesla today, there is no way I would use Autopilot.
I will still buy a Tesla if I can ever afford it, I'll just leave autopilot well alone.
I don't know what gets me back from this mental place that seems to be firmly set in my mind now.
AutoPilot is a human assisted utility. I think many people here, and in general, expect too much of it and are being too critical when there is an incident.
I do think Tesla needs to communicate better what AutoPilot is and isn't because we've now lost lives due to operator negligence, and by negligence, I mean the operator is still responsible for the vehicle despite relinquishing control to a computer.
> CalTrans finally sent ABC7 News a statement later Thursday"confirming" it's their policy to fix broken safety barriers within seven days, or five business days ... but storms delayed the work. As Dan Noyes reported Wednesday, the family told me they believe Walter Huang would have survived, if CalTrans fixed that safety barrier in a timely fashion.
Maybe he was trying to get out of his loan but didn't know the impact absorption barrier had been removed.
I would be very interested in seeing a side-by side test of the LDW systems of major manufacturers. This might give us an idea if the problem is fundamental to the technology or is simply a deficiency in the engineering approach of one (or some) manufacturer(s).
The problem with saying that "our automatic driving system is much better than the average driver" and thus you should be good with allowing our system to drive, is that you are not exposed -- serially -- to the whims of some average driver. One "average driver" mistake and you're dead. No replays.
Therefore, the compound "odds" are INCOMPUTABLE. Just like being paid $1,000,000 per trigger-pull to play Russian Roulette: you are not earning an average of $833,333.33 per try! On "average", over a pool of random 1-time players, sure. But you? You're guaranteed to be dead in a small number of plays. Non-ergodic and ergodic probabilities are incomparable.
So; since automatic driving systems are non-ergodic, they must be dramatically better (ie. orders of magnitude better) than the "average driver" to be considered viable.
I live in a part of the world where even cruise control is unusable for 6 months of the year. Automatic driving systems? Not even thinkable, except for perhaps a few days of the year, on a few simple portions of a few roads...
The risks from human drivers are also non-ergodic, so that suggests that if automatic driving systems are even minimally better than the average driver you should demand that every other driver be replaced with them without delay, because you will die sooner with human drivers on the road.
Of course, you won't want to be replaced unless the automatic driver is better than you, and human drivers all think they are far above average. And that’s the real problem.
The computer must not just be better on average — people must know that it’s better overall, across all possible weird events that a human just might throw a Hail-Mary and survive!
This is often a joke, and there's some truth to it - people are often overconfident. But there certainly are individuals who think they are below average. It might be rare and still a market.
The question is why it is Tesla, in particular, that has to remind its customers that they have to actually pay attention when they drive its cars. Is it possible the company somehow managed to convince people that its cars are equipped with self-driving software, rather than a "driver assistance system"?
Right at the start I was very much wondering how these even allowed this feature on the road. Just from a regulatory point of view. It very much sounded and looked like a complete auto-pilot.
That said, people go through a drivers test, get their license and then are allowed to drive any vehicle really. Can you just solve this by giving the driver assistant a test and seeing if it passes? This needs some kind of filter or regulator to make sure it is safe.
The advantage of getting somewhere [without having to steer] will be negatively affected if "but also, you might die" is on the ticket.
https://www.reddit.com/r/space/comments/76e79c/i_am_elon_mus...
https://twitter.com/elonmusk/status/789007996038721536?lang=...
Victim who died in Tesla crash had complained about Autopilot: http://abc7news.com/automotive/i-team-exclusive-victim-who-d...
I-Team investigates why CalTrans didn't fix safety barrier before Tesla driver died there: http://abc7news.com/automotive/exclusive-i-team-investigates...
What has been unreliable about their reporting?