Hacker News Reader

Comment by free652 | Hacker News Reader

The problem is that the autopilot almost works, so drivers dont need to pay attention most of the time.

This is actually a phenomenon that is well studied (as it happens, by NTSB). Automation that is 99% reliable - kills! Less is unrelible enough that you notice and watch it carefully, more is reliable enough to actually rely on. It is like the uncanny valley of automation.

>Automation that is 99% reliable kills!

It also saves more lives than it kills people. So?

Depending on the numbers, the “saves more than it kills” standard may be 99.9% or 99.9999%. You need data to answer the question and any claims should be backed up by numbers, followed by a bunch of analysis. It’s definitely not something we can prove about currently sold technology with a one-line quip.

> It also saves more lives than it kills people. So?

Do you have data that the rest of us don't - that is data which identifies incursions (that is near-crash interactions which do not end in a crash) where vehicular automation acts correctly and human drivers do not?

If not, than you really can't make that statement.

You can say that "Tesla's luxury cars running under autopilot show fewer crashes per mile than average for all cars", and be accurate, but that's not "saving lives".

it's not quite there yet, unless you believe at the stats published by Tesla that compares apples and oranges.

Sadly, humans that are 99% reliable kill more.

Is there a cite for that study? Everyone wants to scream about these things being dangerous, but we're still stuck in the noise floor. There are still only four Tesla autopilot fatalities over the 5.5 years it's been available, the last just shy of a year ago. And the number of cars sold (and presumably autopilot miles driven) has been rising rapidly during that period.

I mean, seriously: if this technology "kills!" we really should have some statistics on that by now, right?

Google "handoff latency". It is a measure of how long it takes a human suddenly dumped into a situation they were not previously paying much attention to, to become fully aware of it and act accordingly. In aviation this is measured in Tens of seconds. This is ok since in aviation you're far from everything when automation is on and thus have tense of seconds to act.

With cars you may need a response in seconds or less in case of a disconnect. Humans suck at that.

Some sources: https://www.google.com/search?q=autopilot+disconnect+airplan...

It seems like there should already be an industry term for this, it's like the autonomous systems version of the "uncanny valley" [0].

https://en.wikipedia.org/wiki/Uncanny_valley

There is in aviation. It's called handoff latency. It's the time between autonomy giving up and asking for intervention, and the time when something bad might happen. In aviation this is measured in minutes, worst case tens of seconds. In cars, it can be sub-second and humanly impossible. This latency is why an autonomous car that works 95% of the time is more dangerous than driving manually. I haven't seen any indication that this problem is possible to solve until we get cars to 99.999%.

Very interesting, I did not know that! Although, having flown planes myself, it makes sense. You can afford a few second of inattentiveness. Same with boats. But not in cars.

If you haven't listened, it's worth checking out Children of the Magenta on the 99 Percent Invisible podcast.

There's also a lot of interesting research from Missy Cummings' Humans and Autonomy Lab at Duke. https://hal.pratt.duke.edu/

She's been very critical of a lot of self-driving systems that mostly work but still require drivers to pay attention. (She's also a former US Navy fighter pilot.)

Well, I think that one problem is that the automation is not giving up; there is not handoff occurring. It's reporting "all's well" until fractions of a second before the crash, if that. This means that it's up to the human to remain constantly vigilant and identify abnormal situations.

Of course, when you consider that an overhead road sign and a stopped semi trailer "look" the same to the systems involved, it's not surprising that they think that everything's OK.

Aviation also has a term "Children of the Magenta" [1] to describe pilots who only ever use autopilot -- the sense is, when the task is automated, you don't get practice using it manually, so in an emergency you have no experience of actually being in control.

I expect autopilot cars could follow similar regulations as aircraft: "if you want to keep your license, you can prove you've driven in manual mode for x hours every month", where x is enough to know how to control the car in an autopilot disengage.

[0] https://99percentinvisible.org/episode/children-of-the-magen...

Reply on news.ycombinator.com