Tesla’s Autopilot found partly to blame for 2018 crash on the 405

I'm a partner in an auto insurance company, and most of our no-accident records will stay that way, statistically speaking.

If this driver had a one in a billion miles chance of crashing and Autopilot (with an inattentve driver, as they all tend to be) had a one in a million miles chance, then Autopilot is decreasing safety.

At a certain level of safety, Autopilot would make everyone safer, but it's definitely not there yet.

Some people will immediately see lights flashing and pull over straight away without checking for a safe location, so by delaying their lights - it removes that out of the equation.

Also, I would often go less than 20 down a pass during the winter because I didn’t have snow tires or chains, and was passed by plenty of locals doing 60+ because they were equipped for it.

I wouldn't assume that the driver going slowly was much under the speed limit, if at all.

Australian driving law/principles state that "drivers should drive at the _maximum_ possible speed that is: within the speed limit, and safe for the conditions". Notably absent in this is "within your ability" as if you're not capable of safely driving on all roads, the balance of probabilities says you probably shouldn't be driving on _any_ roads.

[1]: https://en.wikipedia.org/wiki/Impact_attenuator

That humans get distracted and often make mistakes is no surprise, it should also be obvious that the amount of risky behavior is correlated to the amount of _perceived_ risk in any given situation, which is probably why so many are "a simple crash."

It is, however, somewhat scary that a machine that has nothing else to do other than to drive the car as safely as possible still makes these "dumb" mistakes.

> Except humans crash into stationary vehicles in lane all the damn time.

And Tesla's argument is that Autopilot is far safer than humans.

You see the brake lights up, and as long as you're alert and keep some distance you're going to be alright. If you're not alert and/or driving too close you hit it.

Humans get inattentive and distracted, and humans follow too close for current conditions and speed. There's no excuse for either of those factors to apply to an autopilot system. If they do, it's badly designed.

False positives are dangerous. Trying to detect ‘Anything’ may result in more deaths than hitting the tiny number of exceptions.

Alternatively, the system would be overly flakey and hard break all the time.

As software engineers, we can understand the hidden costs in complexity and bugs when implementing features that don't need to be implemented. I could imagine that false positives for bolder avoidance causing more harm than false negatives.

Obviously it should be smart enough to detect any and every obstruction, but I'd rather the 99% use case get ironed out first.

Similar to how cruise control (Level 1) doesn't know how to drive a car. Yet people are willing to put up with the concept that it causes some accidents.

And worse (heavy cynism ahead): they would make the car look stupid even when not causing an accident, whereas boldly plowing into whatever might or might not be an obstacle makes the car look strong and confident, right until it looks all bent and crumbled. Car sales are incredibly marketing driven and a self-driving implementation that routinely chickens out of ambiguities that look easily solvable to humans would be a worse marketing problem than twice the number of accident investigations Tesla is facing.

The failure modes of these systems need to be much more gentle: when I'm confronted with an uncertainty while driving, I slow down and pay more attention. I don't speed up until I'm confident that a higher speed is reasonable. This is far far more sophisticated than simply "can I see something in front of me" -- I'm evaluating the probability that I need to stop based on the events I can see happening. If I see a car in front of me slow down and swerve out of the way of something, that means something much more than "obstacle detected" -> "obstacle removed".

I agree. But do they when alert and focused and when they clearly see the object? I highly doubt it.

This Tesla was all of those things, and still crashed.

We don't really know how well the Tesla automation does vs human drivers, because the statistics are complex, and Tesla only gives us deceptive summaries.

The capabilities that Tesla cars have aren't that much different than other luxury cars, but the other manufacturers don't have NTSB investigations, because their marketing isn't convincing people the car can drive itself.

The bar is much higher than that. People accept people killed by other humans suddenly becoming ill or briefly not paying attention. We accept human error. We do NOT accept machine error.

So the criterion is that autopilots should avoid a lot of accidents that humans risk getting into (e.g. lack of attention) but it must not be involved in ANY accidents that a human paying attention would get into.

In the end, no one really cares about the overrall safety or the statistics of it all, when it comes to acceptance of new systems. What matters for public perception (so in the end legislation) will be an almost impossible bar for autonomous vehicles, because they don't have the benefit of acceptable human error that we give flesh drivers.

By that measure, no tech company has contributed more to self-driving than Facebook Inc, masters of distraction. Relative safety isn't a good metric.

It takes a long time to do an investigation. This is one of a number of incidents. You should try actually reading facts instead of Elon's Twitter.

"With the new Tesla Vision cameras, sensors and computing power, your Tesla will navigate tighter, more complex roads."

There is a lot implied in the short phrasing of "navigate tighter, more complex roads". It also begs the question that Tesla's capabilities were sufficient in the first place.

"All new Tesla cars have the hardware needed in the future for full self-driving in almost all circumstances. The system is designed to be able to conduct short and long distance trips with no action required by the person in the driver’s seat."

Why make potentially misleading and untested future-looking statements when consumer safety should be the focus of a system like this? https://www.tesla.com/autopilot

“Yes, this autonomous vehicle killed your son who just happened to be in the wrong place at the wrong time, but someone else’s son would have died in an unrelated accident somewhere else were it not for autonomy.”

Eh, Tesla likes to point to this statistic every time the safety of Autopilot comes into question, but it is an apples-to-oranges comparison because AP is only used on highways where traffic incidents are already far less common per mile driven.

Detailed data is hard to come by, but more nuanced studies have been far from conclusive in favor of the "Autopilot is safer than human drivers" thesis.

Also, my understanding is that aircraft autopilot systems are designed to allow for pilot response to take up to three seconds. Three seconds on the highway is a lifetime.

Takeoff, initial climb, descent, and approach to landing are high workload times, easily holding the attention of the crew.

Cruise is much more relaxed and much less need for hair-trigger responses. My aircraft has audio and visual cues if an aircraft breeches a 2nm (or thereabouts) and is within 1000' vertically. Other than that, it's systems monitoring, weather monitoring, keeping up on the nav progress vs fuel remaining, and making sure the automation isn't trying to kill you, either subtly or acutely.

tl;dr: aircraft pilots generally must be attentive. Tesla drivers on "auto pilot" - not so much.

For example, in a car, several seconds of following distance is normal and following another car that closely is common. In a plane, which can move in 3 dimensions, horizontal separation is measured in minutes and it's not that common to be following immediately behind another plane at the same speed: https://en.wikipedia.org/wiki/Separation_(aeronautics)#Horiz...

Adaptive cruise control and lane assist, available in every modern car under the sun will make microadjustments to steering, speed, and following distance all day long.

What exactly does autopilot bring to the table?

It’s whether you actually are - and that’s very different from what you perceive. For example i would be curious to see how eye movement of regular autopilot users differs when they are using autopilot vs when they aren’t.

Based on all prior studies of attentiveness I suspect you’d find dramatic reduction in active scanning, etc.

Part of the reason for the many curves in I280 in the sf peninsula was apparently to ensure drivers had to actually do something while driving. I’m tempted to investigate whether that’s a myth or fact, but the nature of the internet means I’ll probably just find that i280 gave me cancer :)

Yes, so people ending up focusing on things like cellphones and email.

The problem isn't "you don't have pay attention to X" it is "you don't have to pay attention to anything during normal operation". It is human nature, if not outright physiology that it is incredibly difficult to keep someone engaged, and paying attention to a task that they are not actually doing.

The problem with halfway solutions like autopilot isn't that they control the car some of the time, it's that they can, essentially without warning Stop controlling it.

Self driving cars will happen. But the current model is essentially "do enough of the driving to maximize the likelihood that the human 'driver' is not going to pay attention, but don't do enough to make their lack of attention safe".

My statement isn't "cars allow you not to pay attention", it is "systems like autopilot actively encourage a lack of attention", that's backed up by the need for alert systems when they have obvious signs that the driver isn't paying attention, like "no longer holding the wheel". If the system didn't in general usage result in lack of attention then such a system would not be necessary.