A Tragic Loss
teslamotors.com
teslamotors.com
Maybe they had to do it for legal reasons I don't know (I'm certainly not a lawyer) and I'd love to own a Tesla but couldn't they have worded this a little more sympathetic and a little less lawyer?
Already down $6/share in after hours trading.
So long as every Tesla accident makes the homepage of CNBC, unlike the millions that occur in other makers' vehicles.
Tesla puts out an immediate PR, and the breaking news that went up on all the financial sites included quotes from it nearly as fast.
http://www.cnbc.com/2016/06/30/us-regulators-investigating-t...
Which is about right.
Table 28 of http://www.cdc.gov/nchs/data/hus/hus15.pdf#019 is relevant to this discussion
The Tesla Model S will now never be one of them, and in fact, is now among average for fatality rates that includes motorcycles, drunks, and other dangerous factors. It's now not even among the class of luxury cars that have very low driver deaths.
Right now self-driving cars are far too dangerous for public use.
Which version? There is 60, 60D, 70, 70D, 75, 75D, 80, 80D, 85, 85D, 90 and 90D so which one are you talking about? Tesla calls their cars Model S the same way Apple calls their laptops Macbook.
It seems fair to me.
Unfortunately, fortunately, Tesla is having to educate people and being very clear to not allow room for panic and unwarranted fear mongering.
I would say it is as to the point as it can be, and that it is heartfelt.
Notice how quick they are to bring up "this is the first fatality in 130 million miles", but AFAICT, traffic fatalities in the US occur at a frequency of around 10 per billion miles [1], which is the same.
So even though this is just one data point, it's spot on the current average for fatality frequency in normal cars.
Furthermore, to gather enough statistics to be able to say with confidence "semi-autonomous Teslas are/aren't safer than normal cars", Tesla either needs to increase their sales volume by several orders of magnitude, or we'll have to wait for ten years.
[1] https://en.wikipedia.org/wiki/List_of_motor_vehicle_deaths_i...
If we had data for the "high end modern full-size car" category, it would most definitely be lower than 10 fatalities per billion miles.
A normal car drives about 100 000 miles in 10 years. You then need one million autonomous cars driving around for a decade before you even know whether they're safer!
Maybe a billion miles are needed for a fatal crash, but for small mistakes it's much less.
Can you give a source for that number as it seems a bit low given warranty numbers.
"Among all vehicles in the US, there is a fatality every 94 million miles. Worldwide, there is a fatality approximately every 60 million miles."
If you want to do the numbers to compare the Tesla to other cars, where they measure driver fatalities per million registered years, do this:
130,000,000 miles / 12,000 average miles per year per car = 10,833 years.
That's 10,833 registered years for the Model S.
The IIHS site normalizes to deaths per million registered years, so multiply that number by 92.3, and since we have one death so far, the Tesla Model S is now at 92.3 driver deaths per million registered car years.
Let's compare to other car models:
Ford Taurus 2WD: 20 deaths
Hyundai AccentL: 86 deaths
Lexus LS 460 2WD: 18 deaths
Acura TL 2WD: 5 deaths
Audi A4 4WD: 0 deaths (yes, a bunch of car models had zero driver deaths)
And average for ALL 2011 vehicles: 28 deaths.
How you can draw such a conclusion from a singular data point? That's beyond ridiculous.
> This is the first known fatality in just over 130 million miles where Autopilot was activated. Among all vehicles in the US, there is a fatality every 94 million miles. Worldwide, there is a fatality approximately every 60 million miles.
"4 Times Deadlier!" certainly comes across as sensationalist based on such flimsy data (never mind that the real figure is actually 3.2 and the parent chose to round up rather than down to serve his/her purpose).
Whereas Tesla's statement comes across more as a statement of fact (albiet quite self-serving.. no question about that).
Auto-pilot is semi-autonomous... I gather you consider that a risk...
What about Subaru EyeSight? It can autonomously brake. I can drive for hours on the freeway without ever hitting the brake nor gas, even in stop and go traffic. It has warned me multiple times of a sudden slowdown just as I noticed it. It hasn't yet saved me from anything, but if any of those circumstances I'd been distracted (even for a legit reason) it could have. It once warned me of a motorcycle in my path on a very dark night when the dim tail-light of the motorcycle was overwhelmed by the much brighter tail-lights of the cars to its left and right. I would have seen it soon enough, but the car saw it first. Is that a risk to other road-users? Yet, it's automated by some definition.
What about various other manufacturer's lane keeping features? That automation to some degree.
What about ABS? Stability control? Again... automation to some degree.
All of these have their detractors, yet I think statistics clearly show an improvement with each level of driver assist.
I would assume that the higher-end GM cars do have an automatic braking collision avoidance system by now, since that's a pretty standard feature above a certain price point. And I would assume it's not foolproof either, but at least it doesn't lead the driver to think he can take his hands off the wheel and not pay attention to what's happening on the road.
Tesla reiterates every time and makes you confirm you understand that this is not the case. If the driver did it, that's completely on them being irresponsible.
And pilots still make mistakes when adjusting to a context where the autopilot has unexpectedly handed control back to the humans.
there is no such feature, there is a MobilEye _lane following_ system Tesla repackaged under Autopilot(TM) brand.
Say it out loud - lane following. Mercedes sold this same tech almost 10 years ago, difference is Tesla felt pressure and started believing their own marketing lies about autonomous driving.
Although US foreign policy and intelligence decisions were more or less directly responsible for 9/11, you don't lecture people at Ground Zero about the dangers of blowback and not being aware of your government's activities.
There's no such thing as "condolences, but...", and Elon Musk has shown a pattern of this kind of tone-deafness before. Not to say he's not brilliant and not to say autonomous cars won't ultimately save many lives, just that he should either listen to his PR department if he overruled them, or fire his PR department if they did this without him.
Excellent, Doc. Further to the point about these statements and Musk's leadership: not too long ago Tesla would include the fact that nobody has been killed in a Tesla in their public statements. I found it completely inappropriate since we all knew it was just a matter of time, sadly. Again they play with fire, as another death with autopilot will really wreck their statistics.
one failure and they will take a minor publicity and money hit, two and its going to devastating
Now, this is pure speculation, but I really can't imagine running into the side of an 18-wheeler crossing the highway unless A) I was traveling at a reckless speed (unlikely given that "auto-pilot" was on) or B) I wasn't paying attention to the road.
https://img.washingtonpost.com/blogs/the-switch/files/2016/0...
IMO, human (customers') expectations require more than a mere autopilot.
I think they did a fine job of handling this, and with the high visibility of this incident due to the use of autopilot they really had no other option.
Edit: Given the down votes, I guess people really want the mental picture. Sad.
They have to defend autopilot not only to protect the brand but to protect the public's perception of autonomous vehicles in general.
Self driving tech is poised to save many many lives. So from a utilitarian perspective, it's probably justified to take extraordinary measures to make sure reactionary media and public whim doesn't kill it off, however uncomfortable that might seem in the short term.
Whilst this case is incredibly sad (and I don't want to downplay that in any way), if you're trying to minimise the overall amount of fatal crashes, exonerating the tech is the priority (if it is truly not at fault).
2. From the rest of this thread, it sounds like their long-range cameras/radar has very limited FOV.
I personally can't wait for fully self-driving cars, but it's going to be a big PR battle in the US
> They have to defend autopilot not only to protect the brand but to protect the public's perception of autonomous vehicles in general.
They could have done that in two separate posts. This was tone deaf at best.
Why? Why not just build the cars people want? (including cars that people want but don't realize they want yet)
There shouldn't be a political agenda associated with engineering. Build what is needed. Build what people want. Build what people will need. But never "defend my reputation and the reputation of this device that I'm making"
EDIT: I mean, I get why Public Relations are important and so forth. So Tesla is certainly free to do what they want here. But lets not pretend that this carefully crafted "condolence" piece that has come out roughly one and a half months late is anything but damage control for this company.
The computer would have to be 99.99999% reliable to do that.
The accident rate is around 74 per 100 million miles (and fatalities is 1.13).
It's unclear exactly how to turn that into a percentage, but no matter how you do it it's quite high.
Say an accident takes 5 minutes, and people drive 30 miles/hour. Then that works out to 99.999% for humans. If you use the numbers for fatalities then it's 99.99999%.
I.e. 99.99999% of the time, as whole across all [US] humans, people drive in a way that does not cause a fatality.
That's the bar computers have to cross in order to save any lives at all.
Additionally, 92 people are killed in fatal car accidents in the US every day. So it's not as though this is some uncommon occurrence that autonomous vehicles would be unlikely to improve.
It most definitely is not above that bar! The autopilot is combined with a human operating the car.
How many accidents were avoided by the human rather than the autopilot?
I suspect a ton of accidents (i.e. the autopilot has a high error rate).
How many did the autopilot avoid that the human would not have? Probably not that many, if any.
> Additionally, 92 people are killed in fatal car accidents in the US every day. So it's not as though this is some uncommon occurrence that autonomous vehicles would be unlikely to improve.
Do the math.
I did.
0.00001% seems pretty uncommon to me. There is a lot of driving in the US, so even that low of a number is visible, but it's still a low number.
Remember every single computer must be all but perfect to reduce the error rate.
Have you ever seen a computer to be that good? At even something as simple as not crashing? Never mind driving a car.
Or probably it avoided 100,000 accidents!
Unfortunately, we are both just guessing.
So please keep your math out of this.
Except I am not guessing. That's the point of math.
The autopilot plus a human, only did slightly better than a human alone. That means the autopilot did nothing, since slightly better is well within the range of normal for a human alone.
> So please keep your math out of this.
What a strange reply. Why keep math out of this?
Are you hoping that this will be real if you ignore all evidence to the contrary?
Right now the evidence is in: Computer assisted cars don't do anything helpful to the accident rate. This bodes poorly for self driving cars, and since the error rate they have to hit is so low, it's really not looking good.
I personally don't expect self driving cars to ever be used on regular streets. Only on computer-exclusive roads, specially marked for them.
You have to understand the magnitude of the perfection needed to have any hope of implementing it.
Tell me: Before you read my message, would you have assumed at a computer that is 99% perfect, or 99.9% perfect, would be better than a human?
I can tell you, that until I did the math, I thought so myself.
But 99.99999% is 10,000 better! If the computer was 99.9% perfect you would have almost 1 million fatalities per year (assuming things scale linearly, which I'm sure they wouldn't, probably most of the time driving is easy and 99.9% would still not get into an accident).
> It's a regular, daily occurrence for 92 real human lives every day.
I know. (Although you said that badly: it's not a daily occurrence for those people. But I get your emotion.)
But a computer will not solve this problem, not for a long time. We simply are not able to make a computer that is that good.
Let's see if we can make a web browser that is that good before we try to make a driving computer that good.
It also puts human drivers in a new light. I was of the camp that people are terrible, horrible drivers that kill all the time.
But actually humans are nearly perfect at driving, it's just there are so very very many people driving, so even a tiny cumulative error shows up.
Based on what?
Human drivers are very bad. We kill over 32,000 people a year (and that's a 60 year low, just a few years ago it was over 40,000).
So if self driving cars are just a little better than humans, they will save many lives. If they are a lot better than us, they will save many more.
Bad compared to pretty much every other form of transportation that humans employ.
That's a hypothetical. You say autonomous cars are "poised" to save many lives -not even avoid many deaths, actively "save".
That's very strong language to use based on an "if" that nobody yet knows the answer to with any certainty.
That is, unless you are privy to some information that the rest of us are not? Are you?
What is it that you base your assertion on? Do you base it on anything or is it just a pretty turn of phrase you thought sounded cool?
There are already videos of assisted-driving cars avoiding accidents which likely would have been fatal. We already know they can potentially save lives. It's something we already accept for flying. Autopilot in planes has resulted in deaths of crew and passengers. So we have precedent in at least one mode of transportation.
Drivers get T-boned or rear end other people all the time. Those are two scenarios where driver-assist could have applied the brakes and actively saving lives for drivers inattentiveness. You'll also find no shortage of Liveleak clips of people being killed by drivers who failed to see them and brake in time. Many being scenarios that Google's self-driving cars have the technology to avoid.
Autopilot in a plane doesn't have to deal with obstacles besides the odd mountain here and there.
So I'd say that in many (most?) ways it's easier and safer to autopilot a plane than a car, and as you say, even planes with autopilot have failed.
Planes most certainly do have to deal with wildlife (e.g. birds) appearing suddenly in their path regularly.
In other words, the plane autopilot doesn't "deal" with wildlife at all unless you count running into it. Which is apparently what Tesla autopilot does as well.
I'm not concerend about assisted driving. I'm concerned about fully autonomous cars being handed complete control in real-world situations.
I'm not just "concerned" about them- they scare the hell out of me.
>> Many being scenarios that Google's self-driving cars have the technology to avoid.
What Google says it programs its cars to do, what it really programs its cars to do and what its cars can really do are all separate things.
The problem is that the current technology level is nowhere near advanced enough to allow fully autonomous vehicles to operate safely. There is a huge number of situations that those cars aren't programmed for, that they can't be programmed for, because those situations are completely unforseeable.
Machine learning has a huge problem with data sparsness. You may train a learner in petabytes and petabytes of data collected from the real world and still miss the vast majority of events that may occur.
That is why, like I say elsewhere, machine learning-based AI makes utterly ludicrous mistakes that humans would never do, even in difficult situations were they can't be expected to perform with 0% error. I've used a few metaphors- here's another one: a human would never mistake a truck for a cucumber. A machine learning algorithm, might.
And what's worse, there's no way to prevent this sort of mistake, or even correct it, because most of the time the models built by such algorithms are simply too complex to be processed by humans in the way a hand-crafted system would be (and goddess knows how hard those can be to process).
https://en.wikipedia.org/wiki/List_of_motor_vehicle_deaths_i...
From which I quote:
The number of deaths – and deaths relative to the total population – have declined over the last two decades. From 1979 to 2005, the number of deaths per year decreased 14.97% while the number of deaths per capita decreased by 35.46%.
So at least the rate of fatalities is decreasing. I won't claim I know whether that's because humans get better at driving, carse get safer, traffic regulations get safer, or any other factor, but the point is roads are getting safer already and without any autonomous vehicles on them.
[Original bit:] Again- based on what? Why do you say that? What is even your measure of humans getting "better"? Better at what? Better drivers? Better cognitive systems? Better what?
I said no such thing, you are putting words in my mouth.
What I said is that the baseline for traffic deaths is quite high, and if (or when) autonomous cars are even a little better than humans, then less lives will be lost.
I was replying to someone else and you jumped in. You can hold the accusations of putting stuff in places, thank you.
>> What I said is that the baseline for traffic deaths is quite high, and if (or when) autonomous cars are even a little better than humans, then less lives will be lost.
And I said that autonomous cars being better than humans is just a hypothetical.
I'll add that we have no evidence that it's the case. Computers are better at humans at tasks that require fast and accurate retrieval, or computation, but they'll not just magically become better than humans at driving. Somebody has to program them to do that.
So who is going to do that? Do you know how to do that? Do you know anyone who knows how to do that?
The theory is that we'll put enough (real or virtual) cars in enough training situations that they'll learn to drive on their own, but that approach has serious limitations, not least the fact that in the real world there may be an infinite number of situations that a learner will never encounter during training. In any case, with any other cognitive task that we train computers to carry out they end up making completely ridiculous mistakes, which in the case of driving will cost lives.
Until we have a way to develop systems that understand their surroundings, autonomous cars being better drivers than humans is just a dangerous fantasy.
Also don't forget that while each of us is somewhat unique in terms of driving, the cars are not, so a bug in a car's software can amplify issues even further.
There are issues also how cars will behave when they will have to interact with other self driving cars. Those problems are already difficult when writing distributed applications and with cars we don't have luxury of just letting things crash together until things improve.
Only when that's deemed acceptable.
http://www.fastcompany.com/28121/they-write-right-stuff
> But how much work the software does is not what makes it remarkable. What makes it remarkable is how well the software works. This software never crashes. It never needs to be re-booted. This software is bug-free. It is perfect, as perfect as human beings have achieved. Consider these stats : the last three versions of the program — each 420,000 lines long-had just one error each. The last 11 versions of this software had a total of 17 errors. Commercial programs of equivalent complexity would have 5,000 errors.
You say that so glibly. Have you ever done the math? I did, I posted it here:
https://news.ycombinator.com/item?id=12012166
Summary: The computer would have to be better than 99.99999% perfect to save any lives.
Yes, self-driving cars will save lives if we assume that the self-driving technology is superior to humans in every way. That's almost a tautology though.
Self-driving cars don't get bored, but they don't seem to see white tractor trailers either. Technology has different shortcomings than people do. Whether tech shortcomings result in fewer lives lost than human shortcomings remains to be seen.
It's astounding to me that HN stories about self-driving cars sit right next to stories about iPhones and anti-virus software, and tone of the comments are diametrically opposed. If you read the comments on self-driving car stories, you'd think that technology can do everything better than people. If you read the comments on stories about iPhones and anti-virus, you'd think that everything has terrible bugs, vulnerabilities, and shortcomings.
Isn't Tesla using pure vision? Google also uses lidar, which fixes the problem. EDIT: Looks like I'm wrong and Tesla uses radar.
> Technology has different shortcomings than people do. Whether tech shortcomings result in fewer lives lost than human shortcomings remains to be seen.
Technology also gets better. Ten years ago, people were as bad at driving as they are today. Ten years from now, they'll be as bad as they are today.
Now compare self driving cars ten years ago to self driving cars today. And ten years from now.
> If you read the comments on self-driving car stories, you'd think that technology can do everything better than people. If you read the comments on stories about iPhones and anti-virus, you'd think that everything has terrible bugs, vulnerabilities, and shortcomings.
Those aren't contradictory.
I assume you just pulled that out of thin air but even if you hadn't, it is completely meaningless. We have nothing to compare human performance against. Humans have been driving for more than a century. What are you going to compare that to?
As to how preventable accidents are- sure, if you remove factors like the ones you say, you'll reduce accidents, but humans suffer from those cognitive impairments on top of our extremely well-tuned cognitive apparatus. Computers have nothing of the sort. They may not get distracted, but they're rubbish at doing the thinking that keeps humans safe on the road the vast majority of the time (given that the vast majority of drivers don't get killed in accidents).
To put it plainly, if you put a brick on the gas pedal and lock the wheel, you'll reduce the amount of accidents caused by cogntivie impairments like inattention and poor judgement- but you won't reduce the overall rate of accidents.
In other words, if you hand control to a system that doesn't sleep on the wheel but can also not tell a puddle from a boulder, you will increase risks rather than reducing them.
https://www.teslamotors.com/de_DE/blog/teslas-antwort-auf-da...
Personally, I would have used a more neutral title, led with sympathies for the family, and then gone into the technical detail.
This time, there's an interesting question. Did Tesla remotely access the crash data after the crash? Did they alter any data? Is that verifiable? The NTSB will probably explore that issue. The crash data record in an airbag controller becomes read-only when the airbag fires.
If their detectors don't see a white car against a bright background, that's obviously a serious problem.
Sure, if all the lawyers will promise not to take some statement out of context and sue them over that.. As long as such lawyers exist and that's the way the legal system works this is what can be expected out of statements from companies..
Assisted systems will lead to drivers paying less attention as the systems get better.
The figures quoted by Tesla seem impressive but you have to assume the majority of the drivers is still paying attention all the time. As auto-pilots get better you'll see them paying attention less and then the accident rate will go up, not down for a while at least until the bugs are ironed out.
Note that this could have happened to a non-electric car just as easily, it's a human-computer hybrid issue related to having to pay attention to some instrument for a long time without anything interesting happening. The longer the interval that you don't need to act the bigger the chance that when you do need to act you will not be in time.
[1] http://hal.pratt.duke.edu/sites/hal.pratt.duke.edu/files/u7/...
One option is the Tesla autopilot should have an indication when it approaches "low confidence" areas without disengaging, so the driver is not startled if they have to take back manual control.
I agree there can be handoffs but they mostly need to be in the vein of: I'm slowing down and turning over control in 30 seconds because there's something I don't understand coming up
If your eyes aren't at their best, you know well to go to a doctor and not be driving in the meanwhile. Will the car with autopilot refuse to start or go on autopilot if the camera/sensor/radar has an issue?
So yes you are right, its either full AI or nothing.
Asiana 214 [0] is a classic example of crashing a perfectly good airliner into a seawall on landing.
In the Boeing 777, one example of the (auto)pilot interface showing safety critical information is the stall speed indication on the cockpit display [1], warning the pilot if they are are approaching that stall speed.
Hopefully Tesla will optimize the autopilot interface to minimize driver inattention, without becoming annoying.
[0] https://en.wikipedia.org/wiki/Asiana_Airlines_Flight_214
Your point is still valid, but perhaps we approach a time when over-reliance is better than all but the best human pilots (Sully, perhaps).
There, automation lowered both the standard for situational awareness and fundamental stick and rudder skills. Then, when a quirky corner case happened, the pilots did all manner of wrong on the problem: so much so, they amplified a condition from "mostly harmless" to fatal for all.
Vanity Fair has a nice piece on this accident that's easy to dig up. Good read.
Sure, to nearly every pilot that behavior is wacky, but it shouldn't have been a surprise for more than an instant to pilots who were "operating as designed."
It seems there's no free lunch: when skills atrophy as a natural response to helpful automation it requires advancement in some other skills, should the goal of an ever improving error (accident) rate be achieved.
The pilots were under the misguided impression that the aircraft would automatically spool-up the engines if the aircraft became to slow. This was a safety feature that didn't engage for a obscure technical reason. Even with a manual visual approach the pilot can still use the autothrust for landing.
A more rigorously trained pilot (eg. Capt. Sully) would have aborted the approach and performed an immediate go-around if he got below the glidepath (or too slow) below a certain altitude (eg. 400ft Above Ground Level).
The rules requiring a go-around (or missed approach) apply for a fully automated approach and landing, just as much as manually flown approach and landing.
On the road, where relative separation is much less (and there's even been talk of how self-driving cars can reduce following distances significantly, which just scares me more), the driver might not have even a second to react when he/she needs to take over from the autopilot.
The driver might have needed to react before the auto-pilot realized it needed to react (let along could humanly respond).
There are two things that I take away from this.
* Auto-pilot should probably just keep going (or bring to a as controlled stop as possible).
* It should also collect more data to hopefully warn the driver more in advance.
http://jalopnik.com/volvo-engineer-calls-out-tesla-for-dange...
"You need to keep your hands on the steering wheel at all times during autosteering, yet not crank the wheel hard enough to generate what the car thinks is an actual steering input (thereby disconnecting autosteer). I found this to be about the same amount of effort as simply driving."
I thought that was an interesting observation.
http://blogs.harvard.edu/philg/2016/06/27/smug-rich-bastard-...
You don't design a feature that invites misuse and then use instructions to try to prevent that misuse. That's irresponsible, bad engineering.
The heirachy of hazard control [1] in fact puts administrative controls at the 2nd-to-bottom, just above personal protective equipment. Elimination, substitution and engineering controls all fall above it.
Guards on the trucks to stop cars going under are an engineering control and also perhaps a substituion - you go from decapitation to driving into a wall instead. It's better than no guards and just expecting drivers to be alert - that's administration - but it's worse than elimination which is what you need if you provide a system where the driver is encouraged to be inattentive.
User alertness is a very fucking difficult problem to solve and an extremely unreliable hazard control. Never rely on it, ever. That's what they're doing here and it was only a matter of time that this happened. It's irresponsible engineering.
edit: My source for the above: I work in rail. We battle with driver inattention constantly because like autopilot, you don't steer but you do have to be in control. I could write novels on the battles we've gone through just to keep drivers paying attention.
[1]: https://en.wikipedia.org/wiki/Hierarchy_of_hazard_control
Please do, and link them here. I'd be very interested in reading about your battles and I figure many others would too. This is where the cutting edge is today and likely will be for years to come so your experience is extremely valuable and has wide applicability.
http://www.autonews.com/article/20160321/OEM11/160329991/toy...
They're adding new features to inadequate/improperly configured hardware for what they're asking the car to do, and waiving away all liability for stupid peoples' actions with disclaimers (always be ready to take over).
Whether that's right or wrong is really subjective, especially when you take natural selection into account.
Tesla's (only!) radar sensor is located at the bottom of the bumper, if I'm not mistaken. Compare this with Google's, which is located in the arguably correct position, the roof. Also compare other manufacturers' solutions that are utilizing 2-3 radar sensors, as well as sonar.
https://forums.teslamotors.com/forum/forums/model-s-will-be-...
"Radar tunes out what looks like an overhead road sign to avoid false braking events"
https://twitter.com/elonmusk/status/748620607105839104
EDIT:
When the "overhead sign" comes down below overhead clearance of the vehicle the signal should not be masked. There should have been some braking action in this case. If there was not then the tesla autopilot is unsafe. This is the same blind spot discussed a few months ago that caused a tesla to run into a parked trailer using summon mode:
https://news.ycombinator.com/item?id=11677760
This seems like a serious flaw in autopilot functionality. Trailers are not that rare.
I would be interested if "autobrake/autofollow" functions of other car companies have similar problems.
This is why trailers require safety guards in the EU. Cars simply cannot get underneath them.
Google probably has close to this capability, but they are at the leading edge of image recognition.
In order to depend on it the AI would have to be able to distinguish 99.9+% and would have to be able to tell the difference between a green truck and a road sign. I honestly think self driving cars will require intelligence indistinguishable from artificial general intelligence.
This was state of the art in 2014 (improved in 2015, but this paper is on arxiv and gives the idea):
http://arxiv.org/pdf/1409.0575
Even top scoring Google had classification errors over 6% of the time in 2014. Object localization was much worse. Even if it is at 99% that means 1 of every 100 object will be misidentified. You probably identify that many random (non-car, non-sign, etc) objects in a week driving, maybe in a day. That kind of error is unacceptable in self-driving cars and it is state of the art. That is one of the reasons they are still testing and not selling self-driving cars at Google.
Better than human may not require quite 99.9+% object classification, but I do wonder how great the self-driving car records would be if someone wasn't always there to take the controls. It certainly wouldn't be "better than human" at this point.
Btw I dont feel Imagenet results are relevant, it only tests static 2d images. Car system can have multiple cameras providing 3d reconstruction in near realtime. Mobileye (what tesla repackaged) probably has only one front facing camera and is too stupid beyond 'hey this uniformly colored blob is in my lane'
Car system doesnt need to know the name of a dog breed, all it needs is a shape on a collision course. You can do stereo 3d reconstruction on a 10 year old laptop in real time. You can even do 3d reconstruction with one camera in real time as long as objects move slightly perpendicular to camera http://www.robots.ox.ac.uk/~gk/PTAM/ http://vision.in.tum.de/research/vslam/lsdslam.
Edit: even not needing to know details like "dog breed" it still needs at least a solid 200 object classes to cover not-so-edge cases (which is what had the 6% error rate in 2014).
Real time is not difficult, we got there 10 years ago for primitive 3d reconstruction at 30Hz on ordinary laptop, today its doable on a phone. It would definitely be better than single camera + pattern matching (current mobileye).
edit: This is from last years conference, running on a laptop in real time: https://www.youtube.com/watch?v=oJt3Ln8H03s
"our new Prius applied the brakes while driving under an overpass. I'm glad no one was following too close"
More likely someone confusing the brake warning with the auto-braking shutdown warning. Because it uses optical cameras, driving into bright sunlight or into deep shadows can cause the system to shut down momentarily, and it beeps to let you know.
Even best case scenario that tweet has nothing to do with this incident because the technology and systems are different. Worst case it never applied the brakes at all, and the driver got confused.
If the system can bias its sampling towards the ground, it would make a lot of sense to do so. Lump "above the ground" and "very above the ground" into the same category and use other sensors (for example, the camera..) to detect the edge cases where it matters.
Of course we should have undercarriage bars regardless.
What's beyond the pale IMO is that when auto-pilot was first demonstrated (at the unveil event) - "hands on the wheel" was not part of the story. Journalists and (what appeared to be) Tesla employees were using the feature without hands on the wheel. It looked like Tesla cashing-in on the positive PR without correctly framing the limitations of the tech.
Furthermore, Tesla includes sensors to map the entire surroundings of their cars, but why can't they include sensors to ensure customers have hands on the wheel? (update: comment says they do, but the check frequency is low. why can't it be high?!) It's not just the driver's life at stake, it's everyone else on the road--Tesla should disable this feature on cars [unless it ensures] drivers' hands are on the wheel. Engineers/execs at other companies taking a more responsible approach must be furious at the recklessness on display. One death is too many.
Tesla Auto-pilot fail videos: https://www.youtube.com/results?search_query=tesla+autopilot...
It's incredibly unfair to other drivers on the road to let someone else use beta software that could cause a head-on-collision.
They can and the car already does that. It just do not check very frequently compared to some of the competitors.
Because that makes it less useful. I am a Tesla owner. I am an adult capable of monitoring the car and taking control when autopilot gets confused. My hand being on the wheel at all times is neither a necessary nor a sufficient condition for verifying that I am paying attention and am ready to take over control.
> One death is too many.
I am sick and tired of absolutist statements about risk. Why do you allow cars on the road at all? Why allow cars to have cupholders? Why are drive-through restaurants legal? You make utility-risk trade-offs all the fucking time.
Then you are not using the feature the way Tesla says you should, the only way Tesla says it's safe to be used.
> I am sick and tired of absolutist statements about risk.
Recklessly rolling out tech is screwing the industry-at-large, given the regulatory hurdles that must be overcome.
>> One death is too many.
You took this out of context. When Tesla makes no genuine, up-front attempt to educate users on how to use Auto-Pilot--yes, one death is too many.
Correctly-deployed autonomous driving stands to save thousands of lives annually; what's at risk is some overeager company @#!$ing the regulatory efforts by being irresponsible at scale.
Sometimes public demand forces vendors to add disclaimers to their products despite the fact that everyone knows the disclaimer is a lie. Drug paraphernalia come to mind.
You can't have it both ways.
> a necessary nor a sufficient condition
As in, not only can you pay attention without having your hand on the wheel, but you can also have your hand on the wheel without paying attention. Having your hand on the wheel is "safety theatre"—something done because it makes you feel like you're in control, not something you do because it actually puts you in control.
Cars could do a lot of things to actually ensure the driver is in control. A start would be administering a breathalyzer-like automated attention test (dual-N-back?) before the car will start, and then every few hours of continuous operation after that. Tired drivers would be pulled over to the side of the road and told to take a nap and resume in the morning. (This would probably automatically cover drunk drivers as well, now that I think of it.)
Yes it is. Set your ego aside for a second, forget about how good or bad of a driver you are, and consider how long it would take anyone to move their hand back to the wheel and take control.
Lets be generous and assume half a second. 60 mph * 0.5 seconds = 44 feet, or roughly 3 car lengths before you've even begun to re-assume control of the vehicle, let alone take the appropriate action to handle whatever is going wrong.
Either way, your calculation does not take into account that moving the hands happens in parallel with the critical (most time consuming) step, namely mental processing of the situation and deriving a resolution. So that makes no difference.
You probably meant to say that the attention itself is missing for a split second? But that's something that can be said about car phones, eating food, etc, too. This is just within your individual responsibility, and this is what the law wants. It is, in fact, a risk-reward weighting. We can't use absolutes here ("not a single death is acceptable"), that would be ridiculous. The guy who wrote the parent comment explained that quite well.
(not a tesla driver myself, I dislike driving in general and use my car very rarely)
The truth is that autonomous driving is something every major car manufacturer could develop and demo tomorrow, but large companies like Ford/GM/Toyota are way too risk-averse (or some would say, responsible), to promote it in such a way.
Here's hoping that Tesla doesn't poison the well for autonomous driving.
It makes me a bit sad that the political zeitgeist in the tech community is leaning towards "acceptable losses" when it comes to accidents in automated cars, to the point of pre-emptively expressing disdain at ordinary people reacting negatively to such news. I sense it's going to become harder and harder for us to talk about our worries and skepticism regarding automated driving, since the louder voices claim it will all be worth it in the end. Surely — surely — you're on the side of less death? But personally, I find the utilitarian perspective distasteful. We're perfectly happy to let technology (literally) throw anonymous individuals under the bus as long as less people die overall, but what if it's you that gets hit by an auto? What if it's someone you care about, not Anonymous Driver On TV? The point is that humanity is not a herd to be taken as a whole; every life has rights, including the right not to be trampled by algorithmic decisions or software bugs for the betterment of all. (Sure, you could argue just as well that we have the right not to be run over by drunk and otherwise negligent drivers, but at least this kind of death is not methodical and has some legal recourse.) I feel this perspective needs a strong voice in the tech community too, to counter the blind push forward at the expense of human lives.
Now, this isn't necessarily what happened in this case, but I find Tesla's behavior in these kinds of situations to be creepy and self-serving, at best. Is every death going to come with a blog post describing how much safer automated features are compared to human drivers? Every auto-related casualty is, and should be, a massive event, not a minus-one-point on some ledger in Elon Musk's office.
You might scoff and say that's hyperbole, or scaremongering, and you might be right, but I took it as a warning. It is our responsibility as ethical engineers to ensure that people aren't harmed by the technology we create. It's true that self-driving cars will likely be safer than human-driven cars, but that doesn't absolve us of responsibility. In fact, it makes it all the more poignant.
I-35W was seriously bad news, plus the cell network went down.
Everyone is on the side of less death, but since there are some inherent dangers in zooming along at 120km/h, the losses can only approach zero and will never reach it. The fact that cars are used at all shows that we make something like a utilitarian decision already. They are very, very useful, with a small possibility of personal catastrophe. Same with air travel, drinking beer, or leaving the house ever, all to different degrees.
If the autocars are less likely to kill me than human drivers, I'm for the autocars.
I think this situation is a bit different than the arrival of air travel or even the automobile, because with all those technologies, humans were still making, or validating, all the life-and-death decisions (barring technical failure). I also think the kind of death we're talking about matters a whole lot. What if autos were incredibly safe, but far more prone than ordinary drivers to running over small children and pets? Would society accept that?
Many people are not comfortable with impenetrable algorithms managing their safety on a mass scale, or indeed, with making any conscious decisions at all when it comes to "trolley problem" issues. And that discomfort is well within their rights. Technological progress is not a divine edict, but something that society has to negotiate and agree upon. The debate should be respected.
Yes, this is another thing that makes me so angry about this situation.
Back in the day, I worked on a project to give autonomy to power wheelchair systems. In order to get it to market, we had to go through FDA approval, which involved testing the equipment in every conceivable scenario. We had to show how it performed in rain, snow, dust storms, sunlight, etc. and it had to be incredibly rigorous, defining all limitations, and how the user would be impacted if the systems failed.
If we came the FDA and said "Our system works in many scenarios, but we've found a pathological case where on certain sunny days it will crash into certain brightly colored objects, resulting in almost certain death for the user" we would have been summarily rejected. Non starter.
And yet we're willing to put a system with the above disclaimer (which should have been known to Tesla engineers. This is sensors 101.) on the market with absolutely no oversight? Seriously?
The human behind the wheel had been explicitly instructed to keep hands on the wheel and feet on the pedals -- and that it was his responsibility to override the machine's judgement if circumstances warranted.
And yet, before you can engage this mode the Tesla gives you a very stern warning that it's NOT perfect, and you DO need to keep an eye on what decisions it's making and you MUST keep your hands on the wheel.
We're not talking pages of obtuse legalese, we're talking about a single screenful of clear, straightforward warning text.
Also, as a licensed driver, you have obligations that you are expected to be mindful of, not the least of which is maintaining control of your vehicle at all times.
Are some people going to ignore it? Unfortunately, that's likely. But despite that reality, Tesla's autopilot is apparently doing better in terms of accidents per 100m miles driven than unassisted drivers.
In the case of any normal crossing or intersection the driver should have taken over manual control. In the case of a sudden deviation of other vehicles planned paths, I have doubts the driver could have reacted in a timely and safe manor, even without the automation. Though an even more complete sensor system may have allowed the automation to detect and react faster than a human could.
Yet, losses have to be weighed against the alternatives. Would you think it ethical to not put out seat belts, because they don't save lives in all cases? Or may even be harmful in some circumstances?
"Acceptable losses" is certainly a distasteful and dehumanizing perspective. Yet how else should we reason about seat belts? Or anything similar that is going to help save lives in the vast majority of cases, but may fail in some?
We don't always have the option of perfection. And is it really ethical to stand by as special, sacred lives end because we can't save every life, every time?
It's a choice between "human drivers, MANY MANY lives lost" and "AI drivers, fewer lives lost".
What would archagon have us do? Not have AI drivers until a perfect solution is possible? The "crash-unavoidable" scenario is an unavoidable reality as long as stopping distances/acceleration/reaction times are non-zero.
It sounds as if archagon would have us spend all our time looking at trees -- and none of it looking at the forest -- all while thousands of people die every year under the status quo.
I think perhaps the reason archagon is so aghast though is a combination of two things:
1) The illusion of [human] control and judgement. Implicit in archagon's reasoning is the idea that the outcome MIGHT have been different had a human been making a decision. Yet a human WAS in control here, and DIDN'T make a different decision. That happens. A lot. Archagon seems to be so distracted by the novelty of this situation that he/she has not noticed that.
2) The normalization of automotive deaths and safety engineering in the current industry. 90 people die per day in the U.S. due to errors made by human drivers. Every day human drivers are on the road, 90 more lives are snuffed out. If we follow archagon's hand-wringing, then how could we justify having cars AT ALL? Of course, the reality that many people would die because of the lack of cars makes even the abolition of cars a less than perfect solution with (according to archagon's apparent line of reasoning) a body count attached and all the moral implications thereof.
> I think perhaps the reason archagon is so aghast
> Archagon seems to be so distracted by the novelty of this situation
> If we follow archagon's hand-wringing
I don't know who this hand-wringing, tree-gazing, and monacle-popping archagon is, but he sure sounds like a hoot.
I really didn't offer much other than an opposing perspective and a call to be more moderate. This will come up again and again, and it would pay to have a bit of empathy here.
What should I do? If you think jfrisby and myself lack empathy, what do you think empathy looks like and how do you think it would change our moral stances?
Is this an academic question to you? Or do you actually have these questions weighing on your conscience as a result of things you've done in the course of your work?
You do not. You want to take a moral posture that condemns thousands of people to needless deaths because you're so caught up in emotion you can't see the harm your position would cause.
Wallowing in your own emotions is selfish and immoral when it costs actual lives.
Your entire premise is based on the false assumption that this system was making all the decisions. Tesla's "autopilot" does not override the driver. It does not replace the driver (yet). That's explicitly because everyone involved knows it's not ready for that responsibility yet, no matter how much they've tested it. Tesla knows they cannot possibly anticipate everything -- the whole point of this "public beta" is to gather more information and help them make a better, safer system that eventually is trustworthy enough to consistently make decisions better than people.
You posit that human decision-makers behind the wheel are an important thing we should be wary of giving up but you have ignored one simple fact: There WAS a human behind the wheel, expressly charged -- both by law and by Tesla -- with maintaining control of the vehicle.
The driver had the responsibility of hitting the brakes, and did not do so. That's an error that plays out dozens of times per day without AI drivers being involved.
But instead of recognizing that this is a case of not just a piece of software failing, but also of a human failing -- and instead of recognizing the cost of lost and shattered lives borne by society every day -- you resort to ill-conceived hand-wringing in an attempt to convince others to follow you down a path that would continue condemning untold numbers of people to needless deaths.
This is more like vaccines. They protect a great many lives from events not deliberately introduced at the cost of a small number of bad reactions. Is that number more than zero? Yes. Does it feel small if it's someone you love? No. Is humanity as a whole better off for polio vaccines? Yes, absolutely.
When we talk about weighing alternatives, we're talking about considering the status quo as an option that might be actively chosen rather than a privileged position.
Every human life is unique, special, and sacred. Yet, we cannot ethically permit ourselves to be paralyzed by this to the point where we refuse to reduce the amount of avoidable death because our options are not perfect.
What. "Acceptable losses" is, and has always been the rule when it comes to accidents in all cars, ever. We've made absolutely massive safety improvements in the last few decades, cutting the death rate to around 1/3 of its historical maximum, yet even so tens of thousands of people will die in America alone this year (probably in the neighborhood of ~30k). And next year. And the year after. According to the World Health Organization, there were 1.25 million road traffic deaths in 2013, and if anything I'd expect that to have continued to rise as more and more people worldwide gain access to vehicles.
Yet we will absolutely continue to support car use, because flexible mechanized point-to-point personal transportation is insanely valuable to us (and in fact American society at least simply could not function without it at all at this point). You can dress it up however you like, but the objective fact of the matter is that literally millions of deaths are considered "acceptable losses" here.
Full automation represents our absolutely best shot to reduce the horrendous, annual loss of human life. So personally, I find your appeal to emotion and thinly veiled "what about the children" shtick disgusting and immoral. You've invented imaginary "rights" that absolutely do not and should not exist. Your so called
>"blind push forward at the expense of human lives"
is literally the opposite of what automation aims to accomplish. And I will freely assert will accomplish, because humans are awful drivers. Really, automation is merely filling in the missing piece of personalized transport that we should have had from the start, except that our mechanical technological development was far ahead of our information gathering and processing technological development. Having a human handling that has always been a hack, and getting rid of it has such vast positives that it absolutely justifies a very strong push forward as soon as possible. Every year without driverless vehicles when we could have had them is literally hundreds of thousands more people dead.
This tech does not exist in a vacuum, and context absolutely matters.
How do you counter a utilitarian argument without humanizing the situation? If my friend or loved one were killed by an ill-considered and poorly-implemented automated driving algorithm, no amount of statistics would convince me that the tradeoff was worth it. Who bears responsibility in that case? No one? I suspect that ordinary people won't accept that for a long time to come. At least with human drivers, the courts can decide culpability. At least with a human in the mix, we're all still operating within the same social and legal order.
Your attempt at "humanizing" the situation is merely being dismissive of the values of others' lives.
Everyone who dies is special and important to someone. To wring your hands and rail against utilitarianism is to try and apply tribalism to moral reasoning.
Because this person matters more TO YOU, you're suggesting that we should treat them as if they matter MORE THAN OTHER PEOPLE.
That seems to me an intrinsically selfish -- possibly even narcissistic -- view.
To me, the key factor is blame. Who do we fault in an accident? We can't imprison an algorithm or put it in jail. But we can implicate a careless driver. A loved one should not have to die because a truck happened to be the right shade of white — not without much discussion and the possibility of legal recourse.
There are many horrible, systemic things we could do to improve the lives of the majority at the expense of a few. In the end, we usually determine that the rights of individuals outweigh the utilitarian benefit. Does the same reasoning apply to self-driving cars? I don't know. If the technology proves as safe as people make it out to be, maybe things will end up OK. But it's not a closed case by any means.
An algorithm cannot BE "careless", in the legal sense of the word. You are comparing apples and oranges. Carelessness (negligence) is a different problem than a design flaw.
An algorithm may be flawed, in which case blame lies with the companies involved in creating it and putting it in peoples' hands. Similarly, any mechanical component of a vehicle can be flawed -- and many can cause an accident if they fail. The companies responsible are legally liable already.
> In the end, we usually determine that the rights of individuals outweigh the utilitarian benefit
Cars. Airplanes. Vaccines. All of these benefit the majority at the expense of a few.
The idea that there is no liability here is absolute and utter NONSENSE. Minimally, Tesla can be sued for the flaw in their product and the harm it caused.
As far as I see, the point of legal recourse is to deter it from happening in the first place, I.e. it is purely consequentialist. If you reject such consequentialist thinking, what's the point of having someone to blame?
No, but you can sue the company that built the faulty product, and in certain extreme cases imprison the responsible individuals.
rights of individuals outweigh the utilitarian benefit
What rights are being violated here? People are choosing to use unproven technology, which actually appears to be approximately as safe as humans driving.
Have you considered that perhaps there is no argument to be countered? That your entire foundation of "humanizing" the situation is foolish and off base? Alternatively, you yourself blithely dismissed the fact that your "humanizing" easily works both ways, in fact more easily works the other way: many of us have had friends or loved ones killed by driver error, tiredness, drunkeness/distraction, or in simple accidents, deaths that would not have happened if a computer had been in control. I don't need any "what ifs" because one of my friends was killed by a drunk driver right near her school. Those millions who are going to die have friends and loved ones too you know. It is in fact a statistics game, and one of economic tradeoffs. That's human life, in fact life in general.
>no amount of statistics would convince me that the tradeoff was worth it
Which is why we do not leave societal decisions to you, but instead have a system that attempts (with varying success in some cases) to put significant distance between the moment-to-moment emotions of specific individuals and enactment of law governing all. You may not care about this imaginary tradeoff of your loved one vs thousands of other people's loved ones, but those other thousands of people certainly would. So you can and should be outvoted.
>Who bears responsibility in that case?
It would depend. If it was just an honest mistake that could not have been reasonably caught, then the answer is "no one" and it'd just be covered by insurance, or even the government, so you could say that "society itself chooses to take responsibility for the greater good". We already have a model for this in many, many other scenarios because the basic issue here is very much not a new one. A big, successful example would be vaccines, where we all have to take on a small risk for a very great general good and there is government provided compensation in the rare case that something goes wrong. However as far as individuals go, sometimes there is in fact no one "at fault" or responsible. It's human to always want something to blame but part of maturing is accepting that freak risk is a fact of existence.
>I suspect that ordinary people won't accept that for a long time to come.
We literally do accept it, and have accepted it for all of recorded history. Much of mythology comes out of the basic recognition that there are forces beyond humanity.
>At least with a human in the mix, we're all still operating within the same social and legal order.
Debatable, but also beside the point and it doesn't change the objective fact that a human in the mix is a proven recipe for failure. Your philosophy is cold comfort in the face of tens of millions of deaths. You are also ignoring the large percentage of deaths that are purely down to things such as animals crossing the road, or poor driving conditions that are not reacted to appropriately, or where a driver kills themselves (and perhaps passengers) due to human failure but involves no 3rd parties, which automation could also positively effect.
You're also ignoring positive utility, which is a critical component to the economic and societal equation just as the car itself is. Self-driving cars change the dynamics for city space usage, for people who are unable to drive due to age or medical conditions, for the poor who cannot afford the capital outlay of a full car, for those who are discriminated against by human drivers, for the environment (long charging times for electric vehicles become significantly less of an issue when the vehicles can simply go take care of it themselves while you're doing something else), and on and on. It's hard to overstate how central this is to society up and down the stack. Your viewing lens is entirely too narrow, which is precisely why individual emotional response is a terrible basis for governance.
You're also forgetting that the dream of safe and ubiquitous self-driving cars is still unproven and far in the future. What we're actually dealing with on the ground is a dangerous, nascent, and hype-driven technology that relies on a human fallback and has the power to kill.
We didn't ban vaccines because a handful of people are harmed, and we're not reconsidering their use because a bunch of mathematically illiterate loons think they're more dangerous than the alternative.
Frankly, there's more reason to ban PEOPLE from being able to drive a car than to ban AIs from driving them -- if only because AIs can be improved and people cannot.
All the ignorance of probability, moral philosophy, economics and law will not change the fact that the posture you are arguing from is one of preserving a status quo that costs dozens of people their lives every day.
We're not in your imaginary super-safe self-driving future yet, and if lives are to be carelessly sacrificed as stepping stones on the way there — because Tesla decided to beta test some dangerous software on actual, real-life roads — it is never going to happen. No matter how much of a utilitarian benefit there might be in the long run.
Your ignorant, fact-free fear-mongering is antithetical to saving lives.
1. An immense number of lives are being "carelessly sacrificed" right now, and will continue to be sacrificed every day until something comes along to fix the fundamental problem: People are terrible drivers. 2. Tesla's system did not override control of the vehicle. It was not presented as being safe or ready to operate without oversight. The law does not allow for it to be treated as such. The driver failed to hit the brakes despite being told by the car -- and by legal obligation -- that it was his responsibility to hit the brakes. 3. Tesla's system has proven to be safer than the average, unassisted driver, to date.
If the proportion of accidents involving Tesla's autopilot remained constant when scaling up to more drivers and more miles, then it would be a net win in terms of lives saved to deploy it widely right now.
Obviously though, nobody -- including Tesla -- expects that to be the case, or they'd have been touting this as something every driver should have RIGHT NOW.
Tesla is doing their public beta precisely because it's impossible to design a safety-critical system with perfect foresight. More information is needed to help them build a robust system and the public beta has proven to be an uncommonly safe and effective way to get that information.
If you think my rhetoric is inflammatory it's because you're blind to the logical consequences -- the human cost -- of what you advocate for.
If humans are such awful drivers and replacing them with automation is the moral thing to do, why is it not the moral thing to do to stop them from driving altogether, right now and for the time being, until we have automation that is better than "awful"?
If we're such rubbish drivers that we'll always kill others if we're driving, why are we allow ourselves to drive?
That's a great point but the truth is that we already treat human lives as a resource to be spent: for instance, we allow planes to fly and automobiles to be driven even though their safety is not perfect. We allow that, out of say a million flights, one plane will fatally crash killing usually everybody on board (I don't know what the real number is- I made the 1/1mil up).
So basically we accept that in order for each of us to fly to our destination, 300 ish people must die every so often. Like a kind of sacrifice.
I don't see how any of this is going to stop any time soon, especially when people are always quick to tell you how safe flying is, or how, although road accidents are common and people get killed in them, "what can you do, stop driving?". It seems to me that at some level we all accept that in order for us to have our fast transport, some people must die.
I expect Autopilot to be disabled soon.
Edit:
>So basically we accept that in order for each of us to fly to our destination, 300 ish people must die every so often. Like a kind of sacrifice.
Your position is...crazy.
No plane will take off if there is a calculated chance that it could crash.
There is no component of any machine that is 100% guaranteed to not fail under any circumstances. What the bloody hell are you talking about?
> If a component may fail
As has already been pointed out to you, all components may fail, including any that you add to compensate for the potential failure of other components.
An engineer should know this by instinct.
Adding redundancy was incorrect. Redundancy should be inherent to the design. If one must be added, the design is already a failure.
Redundancy does not mean adding a new component, but maintaining operation under component failure. For example, if the engines fail, it should still be possible to make a controlled landing.
Cf. The grounding of all Boeing 787s: https://en.m.wikipedia.org/wiki/Boeing_787_Dreamliner_batter...
One in 1.2 million flights crash. This is literally a "known chance".
Planes take off anyway.
Crashes are anaomolus events that happen for very specific reasons. Reasons that do not have any seeming connection to any other crash. I don't see how a retrospective could predict anything about the success of future flights.
So, planes have a 0% chance to crash when they take off?
>No plane will take off if there is a calculated chance that it could crash.
I don't get this comment. If there are statistics that prove one in every X million planes crash, then every plane taking off has a 1 in X million chance of crashing.
There is actually a very precise calculated chance that it could crash (but air travellers generally accept it).
What worries me is that we're dreaming up massive fear scenarios. I see the aim of the automation is to prevent the scenarios from ever happening. Sensible speeds with sensible distances.
We can never make it to having zero casualties unless we can first get to a stage where we can treat accidents as an abnormality. Something that needs to be investigated in the same manner as an aircraft crash.
I too hate terms like "acceptable losses" as well as it brings up the truly horrible "cost-benefit" analysis that Ford did [2] with it being cheaper to pay out to the dead relatives rather than fix the problem. I too worry that if we accept lower standards for software now, it will set a precendent with 'acceptable' deaths that will only get lowered.
If Tesla can apply the level of analysis that approaches airplane crash investigation as well as fixes to prevent each death - then I believe we're on the best path.
It's also worth noting that Tesla used the word 'crash' rather than 'accident' in this report, something which has been highlighted [3] as another thing we can do to improve investigations into road deaths.
[1]: http://blog.yhat.com/posts/traffic-fatalities-in-us.html
[2]: https://en.wikipedia.org/wiki/Ford_Pinto#Cost-benefit_analys...
[3]: http://www.citylab.com/commute/2015/09/why-we-say-car-accide...
In that case, it comes back to an old philosophical question (the trolley problem https://en.wikipedia.org/wiki/Trolley_problem). A framing I like is as follows:
"Suppose that a judge or magistrate is faced with rioters demanding that a culprit be found for a certain crime and threatening otherwise to take their own bloody revenge on a particular section of the community. The real culprit being unknown, the judge sees himself as able to prevent the bloodshed only by framing some innocent person and having him executed."
By killing one innocent, you can save many. Likewise for self-driving cars. By killing many innocents, you can save even more. If google said, "We're going to round up a bunch of people at random and murder them, but our cars will never crash," we wouldn't say "Innocent deaths have gone down and so it's acceptable." But if the same people died via malfunctions and chance, our response might change. The trolley problem isn't very palatable, but we'll need to collectively answer it.
If it's me that gets hit by a car, what do I care whether it was driven by squishy grey matter or solid silicon? It's not like AI-driven vehicles hurt more. I'm dead either way.
So assuming it IS going to be me that gets hit, I'd prefer we use whichever driving mechanism is less likely to be checking Facebook or texting its friends while in control of a ton of metal moving at 100km/h.
Is it less creepy if a caring mom or dad of 3 boys maims or kills me? Is it interesting to the victim that hypothetical Joe the Dad expresses remorse over the media and then business goes on for society as usual?
What matters to me is that less people get hurt or killed from preventable accidents (the simple-minded utilitarian perspective that does not weigh rich vs poor, young vs old, but just wants less people dying), because I understand that the next anonymous individual getting killed by either Joe the Driver or a Google / Tesla Car could also be me or someone I care about. So I really do care about the numbers.
The utilitarian perspective actually establishes a negotiable definition as its goal and it discusses the relationship between means and goals -- a position to talk about, disagree with, and measure.
Discussing how over 30,000 people die a year through the lens of Driver Joe, Dad of 3 Girls, is the same as discussing economic disparity through the lens of Plumber Joe, and it's a game that media and narrative elites play better than you, and can swing in any direction they want -- they can say, here's Plumber Joe, now support my economic narrative, whatever it is, because it helps Plumber Joe (does it? who cares about creepy data-backed perspectives?).
I feel that citizen decision-making improves with basic descriptive statistics over Plumber Joe, but they really prefer discussing things in terms of Plumber Joe, or in this case, Driver Jane, Mom of 3 Boys.
And this is all before even questioning the base premise — that self-driving cars will inherently be better than human drivers. Despite their many miles on the road, their combined experience pales compared to human-driven miles, especially in edge cases and harsh weather conditions. There are huge risks in introducing this technology prematurely.
Given that Autopilot is generally activated under safer, more routine driving scenarios -- decent weather, regular suburban traffic and so on; for which we can naturally expect significantly lower fatality rates -- it doesn't sound like it's off to a particularly good "batting average" so far. Especially since we've been promised until now that self-driving cars will ultimately be not just incrementally safer, but categorically safer than human-piloted vehicles.
I would like to see how many accidents would have happened under Autopilot, but were only prevented because of the human.
I'm intrigued that the color is relevant in the car's case - wouldn't it be using some sort of radar to detect and map objects rather than vision? I appreciate I am probably missing something.
The 360 degree sensors on the Tesla are for parking only (< 20 feet range).
Personally when I worked on a self driving car we were going with 360 degree camera and doing path planning based on that, but Tesla has opted not to do that.
The radar is low to the ground and probably doesn't pick up a trailer that's high off the ground. The camera could, but not if contrast is too low. (And I'm not sure if the software is able to recognize the side of a trailer anyway.)
The only way to see it would be lidar (it might have issues too) or radar. Ultrasonics don't have the range.
They should have the autopilot refuse to operate when the sun is within X degrees of the road in the direction of travel.
From what I read about humans, you need +/- 30 degrees up/down vision to get a license (varies by jurisdiction). And we kill a little over a million people each year on roads. Not sure what this should imply about a car's radar though.
It's going to get better, but it's crazy to assume it's currently much more than a highly-assistive cruise control.
Now that's blaming the user for a design flaw. There is no excuse for failing to detect the side of a semitrailer. Tesla has a radar. Did it fail, was its data misprocessed, or is the field of view badly chosen? The single radar sensor is mounted low on the front of the vehicle, and if it lacks sufficient vertical range, might be looking under a semitrailer. That's no excuse; semitrailers are not exactly an uncommon sight on roads.
I used an Eaton VORAD radar in the 2005 Grand Challenge, and it would have seen this. That's a radar from the 1990s.
I want to see the NTSB report. The NTSB hasn't posted anything yet.
Tesla is under a regulatory requirement to report all relevant mishaps to the NHTSA, even if they occur outside the US.
[0] http://www.ntsb.gov/investigations/AccidentReports/Pages/hig...
It seems to be on their radar now,
http://www.detroitnews.com/story/business/autos/2016/07/08/n...
Tragic no doubt, but I'm relieved that this was not a "Autopilot did something very very wrong" story.
Autopilot has the potential to save a large number of lives (I'm sure Tesla execs are thinking about touting "estimated lives saved by autopilot" if the numbers work out, after a few billion miles), so I hope incidents like this don't hamper public perception and therefore research.
No matter how many messages people throw at users, any autopilot that doesn't detect that the driver is awake and operating the vehicle is going to be used by sleeping or otherwise unattentive drivers. It's just human nature.
So what likely happened was the driver was screwing around and the autopilot completely failed to notice a giant white truck crossing the road and blocking their way until it was too late.
edit: reply is quite right, shouldn't speak ill of the dead that way. "Screwing around" is unfair. I just mean that "driver falls asleep/paying attention to other things" is the expected outcome of any autopilot system, and so it's unreasonable of autopilots to have any expectation of drivers for attentiveness.
For all we know the driver had half a second to react before he died. Or maybe he had five minutes. The point is, we don't know. To start from that position and say it's "likely... the driver was screwing around" is pretty crappy.
In fact, I'm judging the software, if anything. Any autopilot software that doesn't force the driver to be attentive is effectively taking the responsibility for driving onto themselves, and so cannot blame the driver for being inattentive. You can't say "I got this" and then at the last second say "oh shit, I need you driver!" even if you put warnings that you don't really got this, because you're dealing with human beings who do normal human things.
Yes, I shouldn't have said the driver was screwing around, you're right. But if the driver fell asleep, and a conscious driver would have avoided this accident? I still think it's Tesla's fault, because he'd be awake and alive in a manual car.
I'd recommend trying it before you knock it. I had similar reservations, but after using it for nearly a year, I find that I'm more attentive and more awake with it than without it. A couple of weeks ago the system refused to engage (broken sensor somewhere) and I had to drive 80 miles home manually. It was exhausting! The mental effort of staying between the lines and maintaining a safe distance from the next car is pretty significant. Getting rid of that frees up more of your attention for the bigger picture. Now, I'm sure for some people it just frees up more of your attention for texting or sleeping, I'm sure I'm not representative of everybody, but I think it can easily be a net win for many.
I always pay attention while Autopilot is engaged, but I'm able to look farther down the road and take more care when checking out traffic to the rear and sides than I can when I'm doing it all myself.
Didn't know about this. If they are truly testing their software in a car without break and steering wheel I have a whole new level of respect for them.
When you start to design a self-driving car the question of a human fallback shouldn't even exist. The car either drives itself, or humans do. Mixing responsibilities in situations of life and death doesn't end well.
A Tesla Model S P85D can brake from 60-0 in 102 ft. (see http://www.edmunds.com/tesla/model-s/2015/road-test-specs/)
Assuming the driver is perfectly attentive, and hits the brake immediately upon noticing the truck, they would still need at least 139.84 feet to come to a stop.
This also assumes the truck is no longer moving. If it is you'd have even less distance to stop.
I don't care whether you're waking up from sleep, or completely alert and instantly react, it's a huge assumption to think that any possible reaction would have prevented an accident or fatality. It's just as likely that even had autopilot engaged the brake immediately, the truck still would have plowed into the Tesla and caused a fatality.
tl;dr - two way traffic involved in a head on collision is a dangerous game, made doubly dangerous because of the combined speed of both vehicles working against inertia.
> When drivers activate Autopilot, the acknowledgment box explains, among other things, that Autopilot “is an assist feature that requires you to keep your hands on the steering wheel at all times," and that "you need to maintain control and responsibility for your vehicle” while using it. Additionally, every time that Autopilot is engaged, the car reminds the driver to “Always keep your hands on the wheel. Be prepared to take over at any time.” The system also makes frequent checks to ensure that the driver's hands remain on the wheel and provides visual and audible alerts if hands-on is not detected. It then gradually slows down the car until hands-on is detected again.
RTFA. It mentions explicitly that autopilot checks for hands on the wheel.
> That doesn't sound like what happened.
> "The driver didn't notice" well, that's why they're using autopilot.
Perhaps the name autopilot exerts to much confidence in its capabilities.
How is this not an "Autopilot did something very very wrong" story?
Tesla autopilot drove the vehicle full-speed into a tractor-trailer, which wasn't registered by its sensor systems. It's great that Tesla already has an explanation for the failure, but the situation as described is far from an improbable edge case. What they are saying is that the autopilot can't detect white objects when a bright light is present -- that's a pretty serious limitation to overlook.
Still, I agree it is strange that a white/glossy object cannot be detected with radar. Is it a stealth truck?
> CAUTION: SOLAR GLARE IN THE A.M.
It's serious enough to warrant its own road sign, but its not as much of a realistic issue on a divided highway, so maybe this road didn't have such a sign.
This type of danger is probably what the article is describing.
I guess I'd like to see an NHTSB simulation of the situation in manual driving to see how humans do unassisted.
It could be even worse. If there was a "road surface mirage" then it's possible that both the truck and the road underneath it were not visible at all. Since the forward radar doesn't have the range or coverage to see a truck at a distance, the only sensor that could have detected the problem was the camera. Because of the phenomena I've mentioned above and the parent post has mentioned, we know this can't possibly work.
Even worse, the circumstances in which the car's sensor could fail are precisely the ones in which the Mark 1 Eyeball would also fail.
I'm sure you'll say, "yes but that's a driver attention problem, not a technology problem", but to me they're the same thing. If you give drivers technology that encourages them to not pay attention then accidents caused as a result of that are partially your fault.
If you want an actually applicable analogy, I'll pull one I've used before. Say you buy an oven that has this great auto-baking feature that lets you just put some cookie dough inside it and then come back whenever you want, even hours later, to have fresh baked cookies waiting. That's how it's marketed - the AutoBake Oven. Except the manufacturer says that you're still supposed to watch the oven at all times in case something goes wrong. So you make some cookies with it and they turn out great, and nothing goes wrong. And hey, it's the AutoBake Oven, right? It says it will bake things automatically so you don't have to worry about them for hours at a time. So you logically decide to put some dough in there and then go the grocery store for a few minutes to get some items for dinner. And you come back and your house burned down because the AutoBake Oven lit itself on fire due to having subpar sensors and beta software in it. Now tell me you wouldn't file a lawsuit against the manufacturer.
Especially after hearing about incidents like this, you won't find me going through my email while letting "autopilot" do its thing. I don't think I would before this, but there's certainly a lower probability that I would now. I'm being trained.
The autopilot will avoid obstacles it can detect and the driver is expected to avoid others (since he has eyes on the wheel and eyes on the road the whole time).
I think they misleading thing about Teslas Autopilot I its current state is that they call it "Autopilot" while other manufacturers call this kind of assist features e.g "Lane Assist".
A distance-sensing cruise control can't be trusted not to crash into the back of the vehicle in front either, not least because it can't be sure it can "see" it. I suspect motorcycles and other small vehicles may fool radars just like this case seems to have been about contrast and colors in a camera sensor.
I'm sorry, how is that supposed to work, exactly? How does the driver know? Sounds like the driver has to avoid every obstacle - to avoid the case where the autopilot fucks you, like this one - and in that case, what is the autopilot doing?
Have you driven a car with a distance-keeping cruise control? It's very comfortable to let it do that and just watch it and take over when it fails (it's limited in the braking and acceleration it is allowed to do). I can't take my eyes of the road or my hands off the wheel for even a second but it's still useful because it removes all the micromanagement of throttle and brake and just lets me supervise.
The "lane assist" (what it should be and is called by other manufacturers) spots any number of obstacles that the driver failed to see - takes over - and saves the driver from an accident. Or reacts faster than the driver could react - thus saving the driver. That is what it is doing. It is not driving your vehicle. It is assisting in driving your vehicle.
It is still possible to drown with a life vest on. That doesn't mean you shouldn't wear one because "if I'm expected to stay above water - what is the life vest doing?"
Second, you're completely ignoring that autopilot will very much entirely drive the car by itself for minutes at a time (reports are anywhere from 5 to 15 depending on software version), which will obviously encourage drivers to not pay attention to the road. I don't care about disclaimers, I don't care about nag screens or chimes, and I definitely don't care about some warranty text that flashes on the screen. If you make a car that can drive itself you encourage drivers to let it do just that and you should be prepared for that eventuality. Period.
Tesla risks getting regulators to crack down on new car features across the entire industry if they're not careful.
Tesla sure has balls to pick the name "Autopilot" when it really should be called "CoPilot" and that would have saved a lot of trouble.
If only someone could have foreseen these issues...
http://www.theverge.com/2016/4/27/11518826/volvo-tesla-autop... http://www.thedrive.com/news/3753/former-google-exec-and-ai-... http://www.thedrive.com/tech/3909/volvo-rd-chief-teslas-auto...
Critics see a neither-nor limbo, with technology just
good enough to make drivers drop their guard, but not
good enough that driving off a cliff is an
impossibility. ... Shooting for but missing Level 3
autonomy would create his nightmare scenario: a car
that gets flummoxed by one of thousands of variables,
then asks a daydreaming co-pilot to wake up and save
the day.I'm sorry, but that's a cop-out. What's the point of an autopilot if you have to avoid every single obstacle and react to every single thing as if the auto-pilot didn't exist? You can't say "autopilot is safer than normal humans" (heavily implied by the blog post) and then turn around and say "but it's really the driver who's responsible for everything!" anytime there's an accident.
Unlike autopilot, cruise control has very limited functionality. No driver expects it to do anything other than keep their speed, so they are still actively engaged with the road to respond to anything that takes place.
You don't have to react - you have to supervise it though. It lets the driver monitor a system that micromanages steering, brakes and throttle, instead of doing that micromanagement himself. It requires the same attention but is less work.
> Unlike autopilot, cruise control has very limited functionality. No driver expects it to do anything other than keep their speed,
Well, these days you can often set it with a distance to the car in front using a radar. You then let the cruise control do something more than just keep constant speed, it will slow down if the car in front does etc. The driver has to pay full attention though for anything the cruise control can't handle.
You seem to imply "what's the point of an autopilot if I can't take a nap". - well the issue is that it's a little too good to be called a cruise control, but not good enough for a nap. We're simply not there quite yet. At least not legally.
If you're constantly responding to every change in conditions, what's the point of the autopilot? So you don't respond every time, and then something comes up that the autopilot doesn't recognize, and you don't know that until it's too late.
The same as a cruise control - you are supervising rather than micromanaging because micromanagement is more work than supervision. A distance-keeping cruisecontrol may be changing the throttle level dozens of times per minute. A driver supervising that system can keep an eye out for stalled cars on the side of the road, wildlife etc., withput using most of the attention on keeping a constant distance to the vehicle ahead. This complements the driver and increases safety because the drivers cognition can be used to what it's best at, and simple tasks like keeping a constant speed or constant distance can be done by a computer.
The problem with this "Level 2.5" autopilot is twofold:
1) It's called "AutoPilot", even though that shouldn't be the name of it until it's feature complete (level 4 autonomy) simply because people seem to assume that AutoPilot means something that drives by itself.
2) It appears good enough to allow the driver to not pay attention, while still requiring the driver to pay attention. The disclaimers that the driver can never be distracted for a short momet (i.e. use it like a pilot uses an autopilot) don't seem to help.
A lot of the comments here seem to indicate "if I can't text/sleep/whatever then what's the point of an AutoPilot" - clearly misunderstanding the current capability of the system!
I know what my cruise control will do (effectively nothing). I don't know how to anticipate what Tesla's sensors and "AI" can't handle.
And even though actual autopilots in planes are monitored by meatpilots, meatpilots can still take their attention away for a short time. The autopilot keeps the plane flying straight and level, and doesn't take on any navigational role (in most cases). Unless your roads are dead straight and there's no traffic, driving requires a constant navigational role.
Indeed. Tesla wanted to use that name now even though it's the proper name for a feature not yet complete. They have to use a ton of warnings and disclaimers to do it. We'll see if authorities find that sufficient. If I were tesla I'd rename it to something with "assist" for the time being.
> And even though actual autopilots in planes are monitored by meatpilots, meatpilots can still take their attention away for a short time.
So does Teslas autopilot, but the time you can safeley take your attention away on a highway may be perhaps a second or two if you are on a clear straight with wildlife barriers. The autopilot will ensure the car runs straight while you fiddle with the radio.
The same time in an airplane may be ten minutes. THAT is the difference. In an airplane theare are controllers making sure there is no traffic one in front, and there is no risk of a deer or a child stepping in front of the plane. Basically the AutoPilot will have the meatpilot respond to anything it can't handle too - which in an airplane is mostly tecnical issues.
The tractor trailer drove across the divided highway without seeing an oncoming car. I've seen this accident happen in front of me with a fatality, no auto-pilot involved.
A vehicle moves in front of you when you're sufficiently close and you've no time to react. Anything (like glare) that makes such a vehicle harder to spot reduces your available reaction time.
Making it sound like there was a static obstacle in front of the car, and it drove into it at high speed is very tendentious, and not warranted by the report, I feel. It is exactly the kind of 'omg AI kills driver' fearbait that we don't need.
If self-driving systems were ten-times safer than people, then there'd still be a lot of people being killed in them. We need to get a sane perspective on their risk.
It's unrealistic to assume that human drivers are attentive all the time. Autopilot never gets tired, never eats, never applies makeup, never sneezes, and never gets distracted by screaming children.
You have to compare autopilot to realistic humans, not your platonic ideal of what a driver should be. I'm sick and tired of people making statements about road transit predicated on the idea that real humans pay attention to the road with 100% fidelity at all times.
Yes, this is an unfortunate fact of life that human beings are no angels.
However, most modern societies agree that reckless drivers are to be held accountable for their less than perfect behavior. If a police officer (or more recently, surveillance camera) catches you while DUI, or texting, or some other of the most blatant irresponsible behaviors, they will give you a big ticket. And if the consequences of your own idiocity happen to materialize you will suffer much sterner punishment (up to criminal charges for manslaughter, if it came down to that).
That's assuming the infractor does survive. It is a known fact that a non trivial amount of people responsible for accidents end up loosing their lives (or those of loved ones) in the very tragedies they caused themselves. Death is the ultimate judge, and a merciless one at that.
> You have to compare autopilot to realistic humans, not your platonic ideal of what a driver should be.
Why?
Neither the engineers who created those subpar systems - nor the executives who pushed for an unfeasible schedule, nor the shareholders that benefited from the whole deal, - will ever face the same consequences as described above. Corporations raison-d'etre is precisely to shield people engaging in economic activities from the unintended consequences of their actions; that's why they must be held to a higher standard than the everyman down the road.
Would you imagine Tesla's Board of Directors having approved the green light for this impressive-yet-still-limited technology, if the company was required to pay for every traffic ticket that every Model S accrued while being in autopilot mode, nationwide? What about being accountable for every accident, no disclaimer licensing or such nonsense?
> I'm sick and tired of people making statements about road transit predicated on the idea that real humans pay attention to the road with 100% fidelity at all times.
People will make statements about road transit, because it is literally a life-or-death situation. Maybe you do prefer the cattle to shut up and die quietly, so your utopic techno-fantasies are never challenged?
Are you more interested in saving lives or in expressing the short sighted and sanctimonious outrage of a Luddite?
How many lives do you think autopilot is going to save if being as good as a sleep deprived teenager is good enough? And who benefits from that state of affairs?
Anyways, you are a true beliver or a paid shill, so have a good night. EOM.
Since you impose unrealistic standards on progress in an attempt to bolster your own moral position
Driving at speed, you have to make assumptions that other people will do the right thing. You pass cars all the time. If you assume that the car might side-swipe you, you wouldn't get far, and you'd be a hazard yourself.
You drive on undivided highways all the time without driving as if the oncoming car might cross in front of you. And so on. We have to assume that others are minimally competent. That's the point of road laws, not just to control you, but to let you assume things about others.
Where I'm from if you make a left turn and get side-swiped then it's your fault, period. You made an unsafe turn, plain and simple[1].
If what you describe is true, then human error is the main cause of the accident (although the AI failed to prevent it and absolutely should share the blame). I just think "AI kills driver" is a bit sensationalist don't you?
[1]Of course assuming that all other vehicles present were obeying the law (e.g. not speeding) and there were no other extenuating circumstances.
On average, humans are rather unsafe car drivers.
This peculiar wording is making it sound like the computer made the same mistake as the driver, but it seems much more likely that they made two different mistakes. The computer's mistake was in image recognition, and the person's mistake was trusting the autopilot fully and not paying attention to the road.
The optical distortions that cause road surface mirages could well entirely hide another vehicle. In this case, since the trailer was painted white, only the wheels and lower chassis would need to be hidden for the image to become unrecognizable. Not saying that's what happened in this case, but there are situations when your forward vision could be compromised. (There is a theory that the colder ocean may have caused a similar optical phenomenon and hidden the iceberg that sunk the Titanic.)
INVESTIGATION Subject : Automatic vehicle control systems
Date Investigation Opened: JUN 28, 2016
Date Investigation Closed: Open
NHTSA Action Number: PE16007
Component(s): FORWARD COLLISION AVOIDANCE
Manufacturer: Tesla Motors, Inc.
INVESTIGATION RESUME
Manufacturer: Tesla Motors, Inc.
Products: 2015 Tesla Model S
Population: 25,000 (Estimated)
[1] http://www-odi.nhtsa.dot.gov/owners/RecentInvestigations [2] http://www-odi.nhtsa.dot.gov/acms/cs/jaxrs/download/doc/UCM5...
And Tesla's statement does nothing to alleviate these reasonable doubts about putting your life and the lives of your family and friends in the hands of automotive software engineers.
With these minor bugs, Tesla seems to be doing a solid job of poisoning the well for self-driving cars. I'd like to see them explain how their competitors should not be tarred with the same brush, once the political backlash hits.
It doesn't matter how many disclaimers you give before you turn on autopilot - a driver who focused on driving the car (As opposed to letting autopilot cruise) would have probably noticed a tractor driving across the road on a bright, sunny day.
It's a dangerous system. Instead of arguing about the trolley problem, I'd first like to see a car be good at making decisions that save its passengers.
http://www.legacy.com/obituaries/ohio/obituary.aspx?pid=1799...
No, it's not important to note that. You should not be able to hide behind the word "Beta" for systems that could kill people. Either you're willing to let people risk their lives on your software or you're not, and you were.
When the autopilot acts correctly 99.9% of the time, you'd get incredibly complacent and inattentive. When you don't have an autopilot, you are paying far more attention to the road.
Imagine if the first time someone fell asleep at the wheel and crashed, you could just tell everyone "hey don't fall asleep at the wheel". And it just never happened again.
It is important to note that Tesla disables Autopilot by default and requires explicit acknowledgement that the system is new technology and still in a public beta phase before it can be enabled. When drivers activate Autopilot, the acknowledgment box explains, among other things, that Autopilot “is an assist feature that requires you to keep your hands on the steering wheel at all times," and that "you need to maintain control and responsibility for your vehicle” while using it. Additionally, every time that Autopilot is engaged, the car reminds the driver to “Always keep your hands on the wheel. Be prepared to take over at any time.” The system also makes frequent checks to ensure that the driver's hands remain on the wheel and provides visual and audible alerts if hands-on is not detected. It then gradually slows down the car until hands-on is detected again.
This is the first known fatality in just over 130 million miles where Autopilot was activated. Among all vehicles in the US, there is a fatality every 94 million miles. Worldwide, there is a fatality approximately every 60 million miles.
The wording implies that these numbers are directly comparable, but they are not. I would guess that Autopilot is much more likely to be activated in safe situations. But the other statistics include all situtations.
Not only that, you'd expect drivers in the higher-income demographic of US auto customers to have less fatal accidents.
The real comparison should between a model S and cars like the Mercedes S-Class, since this is all about car marketing anyways.
What? Why?
They also might be more sheltered from the consequences of their actions and likely to be reckless while driving.
Your assumption may even be correct, but I don't think it's obvious.
In the end, the accident rate comparison to the global population is completely wrong.
Which makes it a better comparison to Teslas, really.
http://jalopnik.com/the-tesla-model-s-just-got-upgraded-to-l...
Telsa isn't one of them...
As they say, it's not the fall that kills you. It's the sudden stop at the end.
Also, if I were the target of the PR shenanigans pulled on Tesla, I would want to use precise language, too.
Do you think that reassures Tesla owners?
That's really disingenuous. There's no way to know for certain what the driver did or didn't see, but there are a lot more salient features for them to go by other than the white trailer, like the fact that the cabin has driven across the road or that the wheels of the trailer were square in the driver's lane. A much more likely explanation is that the autopilot set expectations too high, the driver was not paying attention (like in all the youtube videos of the system in action), and the autopilot failed and crashed into the truck.
I also think that Tesla does not do anywhere near enough to educate brand-new owners, especially the non-techie later-adopter owners that are starting to buy Teslas in droves. They're turned loose with these cars and there are enough weirdnesses with the usability and user experience of driving and operating the car that it catches up on people. Add to that the cognitive distractions of 21st century life (esp mobile devices) and you have the makings of new kinds of "normal accidents."
I suspect that some time soon it will be a requirement that either DMVs issue new rules for having a license to operate a vehicle with autonomous capability, or, they'll require the manufacturer to certify that the buyer has gone through a certified course of instruction and training offered by the manufacturer or its designate.
It may well be that in this case, the victim, who was clearly a Tesla enthusiast and knew a lot about autopilot, may have passed any such testing by DMV or manufacturer with flying colors. Not the point. In general, as Tesla and other manufacturers make autonomy more mainstream, we're going to see these edge case situations more frequently.
Me, I view autopilot as a dangerous feature that I would never, ever trust. As a set of incredible SENSORS that help me with second-by-second situational awareness while driving, fine. As a replacement for my driving, hell no.
UPDATE: Just read this news on NYTimes:
"The traffic safety agency is nearing the release of a new set of guidelines and regulations regarding the testing of self-driving vehicles on public roads. They were expected to be released in July."
Good and much needed step from NHTSA here.
The current cost seems to be about 3/4 as many casualties as one would expect from the # of miles they've driven.
That statistic is misleading, though. Tesla's comparing autopilot miles to all driven miles in other vehicles, but they're not directly comparable. Autopilot can only be engaged in certain (favorable) conditions - no residential streets, no undivided highways, and you're probably not turning it on during a thunderstorm or in snow/ice.
With stuff that could easily kill you? If it were a drug, there'd be an ethical review and progressively larger trials in carefully selected groups of people receiving informed consent. Something like that might be more appropriate than "your car just updated! enjoy!" beta software.
In the case of autopilot, I'd say similar to a drug study, as it's something that can cause serious harm or death to the human subjects involved. Tesla should have an IRB for ethical monitoring, there should be a small trial group initially (and very gradual rollouts to larger groups), participants should receive informed consent at a Tesla store with emphasis on the risks and what's necessary to use the feature safely, and someone should have the authority to pull the plug on the study if it proves too dangerous.
Given the state autopilot is reportedly in, I'd say right now no one should be using it without an in-person training at a Tesla store to emphasize the proper usage of it.
Deaths aren't the only incidents you'd report in a drug study. Other adverse events are important, too. Tesla's one-in-130M-miles number isn't dealing with the fact that there are likely to be many, many more near misses, non-fatal accidents, times the car can't handle stuff and hands over control, etc.
With a proper trial, Tesla would have to document and report the number of times the radar missed semi trailers, the number of times the driver had to disengage and brake sharply, etc.
I'd be interested in specific weight loss drug examples. There are a number of them available on the market right now.
From the deceased's viral video (https://www.youtube.com/watch?v=9I5rraWJq6E):
>I actually wasn't watching that direction and Tessy (the name of my car) was on duty with autopilot engaged. I became aware of the danger when Tessy alerted me with the "immediately take over" warning chime and the car swerving to the right to avoid the side collision.
Combine this with the strange claim that Tesla put in their press release:
>Neither Autopilot nor the driver noticed the white side of the tractor trailer against a brightly lit sky, so the brake was not applied.
It's hard to see how one can miss a tractor trailer "against a brightly lit sky" when one is paying attention. The human visual system isn't that bad. A likelier explanation is that he just wasn't looking. :(
But as a human driver, I would have taken human psychology into account and guessed the turning car was going to take advantage of that gap, thus causing me to approach much more cautiously.
I don't know how fast that tesla was going, but I don't think I'd be doing more than 5-10 mph in that kind of traffic. I would also expect cars in the right lane to unexpectedly switch into the left. It seems to me the Tesla was overconfident and going about 25 mph.
edit: I feel like my tone here is too harsh (mainly because my personal reaction is that this autopilot feature is professionally irresponsible). Let me pull back my emotions a little and try again:
The only reason the autopilot feature here "saved the day" is because it was driving in an irresponsible manner in the first place. As the comment below me points out, it was driving at 45 MPH, compared to the 5-15 MPH you should be driving. For a defensive human driver, there shouldn't have been a close call in the first place.
The Tesla is driving irresponsibly because it cannot take into account human behavior. It stopped in time in this scenario, but there's obviously a point at which the stopped car could turn and the Tesla could not physically stop in time. This can happen at any speed, but the 45 MPH collision is much much worse than a 10 MPH collision.
I'd also like to point out that the autopilot was likely breaking the law. The speed limit on that road is probably 55 - 65 MPH, but the speed limit indicats the maximum speed in perfect conditions. When it's dark and raining and heavy traffic, going the speed limit is in fact speeding. If the Tesla had hit the turning car, the driver of the Tesla would probably have been cited for speeding, even at 45 MPH.
Further, it would have been common courtesy to stop and allow the car to turn and take advantage of the temporary gap in traffic. People allow me to turn in situations like this all the time, and I return the road karma when I can.
Overall, it seems like quite a failing despite the avoidance of the accident.
All those factors indicate the car was going too fast given the conditions -- autopilot or not. It appears that in general, the Tesla AP does not take road and environmental conditions into account (or weigh them heavily at least). In this case, there was no collision in spite of the conditions. But this should have been a giant red flag for Tesla engineers. Unfortunately, the story at hand tells us that they're not doing their job.
> Watching the video the first time, I didn't see the car before it was already in front of the camera.
I didn't either, not at first. But given just the first frame and knowing a collision was about to be avoided, my first reaction was the autopilot was going to prevent a rear-end due to a short stop, or it was going to stop as someone merged in from the right lane.
But you and I are not the driver of this car. The driver should have known the highway was not divided, should have known of the possibility for cross traffic, and he should have adjusted his behavior accordingly. I guarantee you there was also a sign (before the start of the video) indicating there is an opportunity for cross traffic ahead. Given that, those white headlights (they look reddish in the video, but in real life they would be bright white) should caution the driver that a car is probably looking to turn.
The autopilot did not take any of this context into account, and that's exactly why someone is dead today. It's operating with superhuman reaction speed, but with the common sense and wisdom of a 10 year old.
inattentive faux-professional driver, chatting up a client (gotta get that 5-star rating) instead of watching the toad, driving too fast in dark, wet, complicated conditions (line of stopped cars to the right).
Why is that date important? It is the end of the third quarter in the US which runs from the 1st of April until today. It is also a holiday weekend.
So while, unsurprisingly, everyone here wants to defend Tesla's tone it is hard not to look at the date of this release with a lot of cynicism. They've waited for a great day to bury bad news, and to hide any financial costs.
No doubt all the replies below will claim this is coincidence that Tesla announced this at the end of a quarter on a holiday weekend late in the day.
I've also observed human pilots on busy roads nearly colliding with obstacles when driving toward the setting sun, the visor pulled down and still trying to shade their eyes.
But robots aren't humans, and they don't have to rely only on vision, there are so many ways to "look" it seems like we should have several different ways of identifying obstacles. Different spectra at least.
Even in the confines of indoor environments with simple problems like transparent surfaces, specific problematic fabrics, and acoustic panels it is necessary to fuse several types of sensor data to provide reasonably reliable detection of the environment.
I would hope that as sensors and output analysis techniques continue to be developed and costs decrease, creators of autonomous systems would incorporate a wider variety of sensors into their products.
The robot led him to believe it was more competent than it really was.
It should really only take over in situations the driver failed to handle - asleep at wheel causing car to shift lanes when it is not safe - loud buzzer and lane correction to stay in the lane, not seeing object in the front - auto brake with buzz etc. Then it would be whole different story if it failed - the mistake was driver's to begin with even if driver assist could not correct it. And it will still save a lot of lives no doubt.
The whole idea that complex technology like this can guarantee 100% safe and reliable outcome 100% of the times is I think one hell of a over promise that will continue to cause people to become even more unaware and distracted than your usual car driver already is. And as this shows it will not always end well.
Edit: A Volvo Engineer had similar opinion - From The Verge - "Some autonomous driving experts have criticized Tesla for introducing the Autopilot feature, with a Volvo engineer saying the system "gives you the impression that it's doing more than it is.". That's exactly right.
"Neither Autopilot nor the driver noticed the white side of the tractor trailer against a brightly lit sky, so the brake was not applied. The high ride height of the trailer combined with its positioning across the road and the extremely rare circumstances of the impact caused the Model S to pass under the trailer, with the bottom of the trailer impacting the windshield of the Model S"
1) The Sensors did not "see" the trailer due to Bright sunlight & <XYZ> 2) The Collision was at the WindShield Level and above but not to the Front
Neither of these sound like "Extremely Rare" occurrences. That part is just pure PR Spin.
Request for TESLA - You are doing great things to move the whole industry forward. Do not F* it up by being too aggressive here.
1) Do not rely on purely Visual cues use (multiple) Radars/LiDARs/Some thing that works irrespective of Visible Light conditions - so an additional Radar unit on Top that is checking for obstacles at the height of the car would make sense.
2) Improved Crash sensors
Till then DISABLE the system except in very limited circumstances even if the owner tries to activate it. This is NOT a "Move Fast and Break things" situation. Some poor soul lost his life because of Tesla's inability to foresee this situation lulled him into a false sense of complacency (and partly his own inattention).
I was hoping that the discussion here would consider the outcome of autonomous vs non-autonomous cars in totality instead of having an emotionally charged outburst against a big corporation.
Yes, maybe this is not the right time for Tesla to go all defensive but that's expected & that's how almost any company would respond. Why waste all the energy discussing that only.
Yes, people will get careless with autonomous cars, but don't they do it with regular cars too ? Cellphone, coffee, makeup, etc. So I disagree that autonomous cars have to be perfect.
An automobile is like a weapon in the driver's hand. If they aren't careful, they can hurt someone including themselves. Autonomous systems are there to help reduce the chances of mundane accidents due to the driver being distracted or less than capable. They are not their to replace a human yet.
My point of view is that if a human can't avoid a collision, don't expect the autonomous system to fill the gap. It's their to catch "some" of the errors that a human might make, it's not a foolproof safety net.
I do believe that Tesla's cars are likely an aid to, and in many cases better than, a human driver.
I think, however, that a more full sensor package will be desired: one that can detect changes at multiple elevations and in many directions. One that is more than humanly ability offers.
Yes they do, but trying to do something else while driving is additional cognitive load --- they still have to be paying some attention to driving, whereas "it just drives itself" automation allows a complete diversion of attention. In other words, the automation encourages carelessness.
Is that a hypothesis or are there statistics on accidents & fatalities to back it up. Curious, not denying what you say. I don't have statistics either.
My personal thinking is that if these imperfect autonomous systems can create a net reduction in road injuries and fatalities they are worth consideration. However, I would also want to take into consideration any incidents where an autonomous system protected the safety of its own occupants, yet caused injury to those in other vehicles.
The type of analysis necessary for this would be complex as it would be dependent upon wider scale deployment of these vehicles and large amounts of unbiased data (which is rarely available in most traditional road incidents). I would imagine that currently only manufacturers of these vehicles have access to the majority of the necessary data and it seems unlikely they would release it for third parties to evaluate.
tl;dr: stupid bugs can now kill you dead.
You know how AI systems, when they fail, they really, really fail? Like, when you ask Google Image Search for "unprofessional hair" and it returns images of black womens' hair because it doesn't have a clue? Or when you ask Google Translate to translate "swallows" to Greek and it comes back with the Greek work for "swallowing" (true story) because it has no idea what either word means, just that they're somehow similar, ish?
That sort of thing. When AI fails, it fails in ways that nothing with a brain will ever deign to fail. People don't crash into the sides of barns because they mistook them for the open sky. People don't run over old ladies because they thought they were flowerbeds.
Human drivers fail in situations where they don't have time to react- but even given all the time in the world there are situations where a program will simply derp, and derp and derp again until the end of time. And if this happens in the middle of a life or death situation, people die. The same happens with autopilots on planes.
So this is what, well, bugs me. I don't guess I get to choose how I die (unless I do it myself) but I'm pretty sure I don't want to die because of a dumb bug in a stupid machine. Therefore, I don't want no cars being "smart" around me. I don't want to be the one driving them and I don't want others driving them around me.
The current tech simply can't avoid completely brain-dead mistakes and no road vehicle should be given autonomy until that changes.
And I know what you're thinking - it's not because I think some poor engineer is going to get the axe (ok, maybe that's not what you're thinking). It's because I would not want to be responsible for the code that needs to deliver millions of people billions of miles in their car. It's a daunting responsibility.
It appears to me like this is a case that shows the limitations of stereo vision as the main sensor system guiding the Tesla autopilot. I would even venture to say that it could have possibly been avoided using LiDAR technology. According to various interviews however Elon Musk is not a big fan of LiDAR [1]:
“I don’t think you need LIDAR. I think you can do this all with passive optical and then with maybe one forward RADAR,” Musk said during at a press conference in October. “I think that completely solves it without the use of LIDAR. I’m not a big fan of LIDAR, I don’t think it makes sense in this context.”
[1] http://www.techinsider.io/difference-between-google-and-tesl...
Now whether what Musk is trying to do is a good idea is a different question...
Its possible to reduce the field of view affected by sunlight down to 0.5 degrees of the sky (the size of the sun), but for any practical system it will much larger than that.
At least with the autonomous cars everybody still seems to care about eliminating more or less rare errors all together, because we seem less forgiving to technological than human error. And adding a fitting sensors and detection algorithms to eliminate that blind spot is relatively easy and cheap if the car has already a sensor network infrastructure.
I doubt that similar accidents with human drivers would even get their own statistical category.
1. Autopilot didn't cause the accident, but they're sort of making it sound like it did by bringing it up
2. If people are arguing that autopilot contributed by making the driver feel like they didn't need to pay as much attention, the only way to defend against that is to blame the driver - why would you do that?
3. It's sort of invading the privacy of someone who recently died by detailing the crash, making it about Tesla. I don't believe other car companies would issue a PR release if one of their cars got in a crash, even with accident avoidance features on?
It somehow manages to feel self serving and make Tesla look bad at the same time.
http://www.nytimes.com/2016/07/01/business/self-driving-tesl...
It's a little bit odd that the press release's sole specific reference to time -- "We learned yesterday" -- is in its 3rd word...which, for me, made me think the accident happened yesterday, or shortly before...Not two months before.
>A common phrase in aircraft cockpits nowadays is "What the heck is it doing now?" as pilots have migrated from actually flying the plane to simply being glorified systems managers. While planes have become so, so, so much safer because of all this automation, pilots uncertainty regarding autopilot functioning is a major concern nowadays, and the reason for several accidents. There are very interesting HCI challenges around properly communicating to the pilot/driver "what the heck it is doing" and clearly communicating just how much control the human has or doesn't have at any given point. This "announcement" certainly doesn't inspire any confidence that they have really thought this through deeply enough (I think they probably have, but it should be communicated like it). As a huge Tesla fan, I can't help but feel like I need to hold my breath now and make sure something terrible doesn't happen because of this, and it ends up leading to even more regs setting us back on the path to full car automation.
I really hate having been right. But from the ensuing discussion here about the rightness or wrongness about labeling features "beta" in the car, to the disconnect of how the technology was "meant" to be used versus how it was portrayed by the company itself, it is pretty clear they really fucked up.
The huge discrepancy between the cautionary tone of this press release and everything that came before it is a great reminder that while some of us may be really worried about preventing AI from having unintended consequences in our society, there are at the same time many more pressing issues that we must address that are very much caused by very human motivations and actions.
But what disappoints me the most is how the "tech" crowd will defend Tesla every time.Someone lost their life because Tesla can't put out a decent semi-autonomous driving systems and we are supposed to brush it aside?
BTW iPhone 6/6+ will probably get a recall in couple of years (if sheep complain loud enough) because again there is a design defect (like in every other Apple product, nvidia gpus, you hold it wrong, no cooling in laptops, moisture sensitive parts positioned right next to ports etc), Touch IC is not supported by metal rf shield like in all previous phones and cracks joints due to normal usage flex of whole phone body = grey bar on top of lcd and dead touch.
We have seen from other incidents that Tesla's autobraking will not engage when autopilot is disabled (potentially by tapping the brakes), which is contrary to widespread user expectation -- disclaimers notwithstanding.
However, in this particular case, it was the obstruction detection that failed. Based on their blog post disclaimer, it appears to be a visual sensing system; eye-height radar-based system used by most other manufacturers would have correctly detected the obstruction.
Is there an industry standard (yet) for the efficacy and behavior that these systems must meet?
Mercedes-Benz uses two different radar sensors: "the DISTRONIC radar is configured to monitor three lanes of a motorway to a range of up to 150 metres with a spread of nine degrees, the new [DISTRONIC PLUS] 24-Gigahertz radar registers the situation immediately ahead of the vehicle with aspread of 80 degrees and a range of 30 metres." [1]
Volvo has a combination of grill-mounted 15° FOV radar and a 48° FOV greyscale camera behind the top of the windshield [2].
Volkswagen uses radar [3], but not sure where it's mounted. Honda's CMBS is a radar in the grill [4]. Subaru uses two cameras mounted behind the top of the windshield [5], and no radar, albeit radar is used with some of their other safety systems.
[1] http://www-nrd.nhtsa.dot.gov/pdf/esv/esv20/07-0103-O.pdf [2] http://www-nrd.nhtsa.dot.gov/pdf/esv/esv20/07-0450-o.pdf [3] http://en.volkswagen.com/en/innovation-and-technology/techni... [4] https://techinfo.honda.com/rjanisis/pubs/om/JA0606/JA0606O00... [5] http://www.roadandtrack.com/new-cars/news/a6852/subaru-camer...
So this got me wondering: are there areas where retinae still do better than cameras at certain kinds of discrimination problems? Nature still has human technology beat on hardware in a few categories (particularly joints), but I didn't expect eyes to be one of them. When I look at or near a bright light, I can see objects close to the light much better than I can in a photograph, but I always assumed this was only because I had a cheap camera.
The corresponding question is: how expensive is the camera that prevents this accident?
If our ancestors had adopted the attitude that we could do nothing not perfectly safe, we'd have never left the damn savanna. We'd have certainly never allowed aircraft to fly.
In order for that to happen, it's likely that one of the vehicles had to violate a stop sign or red light.
Note that the Autopilot isn't aware of these things (it won't stop unless a car in front stops). If the Tesla was the vehicle at fault, then most likely it was unawareness of this limitation on the part of the driver (or complacency thinking there wouldn't be any traffic).
Even though I'm eager to see SDV benefits, Autopilot seems like way too much hassle for such possible consequences. An overengineered auto cruise.
Interesting, says model s was rated the safest of any car tested.
The driver of the truck wasn't asked to sign that.
I do hope they are able to learn more from this. This is just a bad coincidence of environment meets chance and someone died because of it.
I do think computer vision still has a but further to go. A white trailer and a white sky shouldn't be a problem. But everyday I'm impressed with how human sight can find and lock into the most obscure details. Like I've been sunblinded before and yet I knew something was in front of me because I noticed a grey shadow on grey asphalt and immediately hit the brakes.
We're almost there but not quite.
Nope, this is completely expected when unproven technology is packaged as a push-button feature and shipped on consumer technology. I don't work for Tesla, but I am a robotics engineer who has worked on exactly this technology, and I saw this coming from a mile away. Tesla should have as well.
Is non-standard ride-height a safety failure? Should trailers have crash panels to avoid drive-unders?
What other companies use this kind of driving assist system? The ones i know are only hinting to the driver , and disengage if the driver does not respond.
Got to be careful about these things. You don't put beta stuff in cars. It's like your doctor making drug trials on you. They should figure out a way that autopilot engages the driver and keeps his eyes on the road.
They've also, FWIW, gained quite a bit of positive press. https://www.youtube.com/results?search_query=tesla+autopilot...
This is literally the only way to develop such systems. And regardless of incidents like this, these systems will be developed sometime in the fairly near future and whichever company is on the leading edge of them will command a massive new market that will probably start growing incredibly rapidly right after it takes off.
Just because something is in development longer doesn't mean the quality is better. Furthermore, Tesla might be gaining some critical information with their early release. And whose to say Tesla didn't test this stuff to the best of their abilities?
'Significantly lower accident rates' compared to what, exactly? The highly accurate accident rates collected from the many decades of experience with Tesla accidents that never happened?
Now compare that to Tesla - beta software and sub-par sensors that can't detect stationary objects in the road (there was a recent case where a Model S on Autopilot hit a stalled car on the freeway despite plenty of time to stop), can't detect obstacles at windshield level (there was a recent case where a Model S hit a parked trailer when summoned), and apparently have trouble distinguishing between sky and lightly colored objects in full daylight (the most recent accident). This is a company that is prioritizing technology releases ahead of human lives.
Significantly lower accident rates compared to Model S Autopilot, if that wasn't obvious. In the future, you will take accidents per miles driven in autonomous mode and compare between Tesla, Google, Apple, Uber, etc. and then see which one has the lower rate. I am saying that Tesla will be the worst.
It's also a reminder to engineers to have an extremely high bar of excellence.
EDIT: thanks for telling me I'm wrong. This was a self-serving blog post on their part to damage control.
Imagine how many press releases there'd be per day if another auto company felt compelled to do one for each accident their cars were involved in.
This isn't just another horrible car accident on US roads, the car was in charge this time and a technology they've been heralding for it's safety is going to be blamed. That's why there is a federal investigation into it. That's why there is a press release to protect the stock price.
Other auto accidents are non-interesting to the public, but this one involved automated driving.
The Tesla suspension issue with non-disclosure agreements also made front pages, just a week or two ago.
http://www.cnbc.com/2016/06/11/tesla-revises-nondisclosure-c...
Tesla's blog is their primary marketing / damage control arm. Anything they publish there becomes instant news.
This is as bad as it sounds.
On the other hand, Tesla doesn't do this for every accident. They are treating this one special because Autopilot very carefully, and in a fully controlled manner, drove the car directly into a semi-trailer, which is more than a little embarrassing for the company. And terminal for the driver.