Comment by Animats | Hacker News Reader

Comment by Animats | Hacker News Reader

Parent Full thread

It's interesting how vague this is. There's an NTSB investigation still pending into a specific Tesla crash.[1] The goals are different. NHTSA asks "do we need to do a recall?" NTSB asks "exactly what, in detail, happened here?" NTSB mostly does air crashes, but occasionally they do an auto crash with unusual properties. Here's the NTSB report for the Midland, TX crash between a train and a parade float.[2] That has detailed measurements of everything. They even brought in a train and a truck to reconstruct the accident positions.

It took a combination of problems to cause that crash. The police lieutenant who had informed the railroad of the parade in previous years had retired, and his replacement didn't do it. The police marshalling the parade let it go through red lights. They were unaware that the traffic light near the railroad crossing was tied in to the crossing gates and signals. That's done to clear traffic from the tracks when a train is approaching before the gates go down. So ignoring the traffic signal took away 10 seconds of warning time. The driver thought the police had taken care of safety issues and was looking backwards at the trailer he was pulling, not sideways along the track. People at the parade were using air horns which sounded like a train horn, so the driver didn't notice the real train horn. That's what an NTSB investigation digs up. Those are worth reading to see how to analyze a failure.

[1] https://www.ntsb.gov/investigations/AccidentReports/Pages/HW...

[2] https://www.ntsb.gov/investigations/AccidentReports/Pages/HA...

Here's a reconstruction video they made to accompany the parade collision report: https://youtu.be/tMsjJWJFBbA

The only thing that matters is "NHTSA notes that crash rates involving Tesla cars have dropped by almost 40 percent since the wide introduction of Autopilot"

Even if autopilot was faulty in some way, if more people were living than dying - does it matter?

Yes. Otherwise, you'd hear about coal pollution all the time, but nuclear meltdowns wouldn't make the news (unless they became much more frequent).

Similarly, 9/11 would be a footnote compared to sketchy pain killer drug trials. The black lives matter movement would focus more on sentencing disparities and for-profit prison conditions than police shootings, etc, etc.

The reason it matters here is that it's presumably easy for Tesla to fix any bugs that are killing people.

You are making comparisons between unrelated domains to make your point, which isn't really fair. Nonetheless, I think what you are really wanting to say is there is always room for improvement, and we should strive for that even if we've already made massive gains.

Simply put, saving lives should not stop one from saving even more lives, if possible.

This assumes that the crashes avoided by Autopilot are of similar magnitude/deadliness as the crashes facilitated (for lack of a better term) by Autopilot.

That is, Autopilot makes certain kinds of accidents more likely — those in which the human driver is not monitoring the vehicle carefully enough, Autopilot misses something critical, and an accident results. It is possible, though not necessarily the case, that these accidents tend to be more serious than accidents that Autopilot prevents, such as lane-changing bumps or other less-fatal accidents.

To be clear, I'm not saying that evidence has been presented that the type of accidents prevented are less important than the type facilitated. I'm just pointing out that not all accidents are equal, and that there may be correlation/causation that makes the overall calculus more complicated.

Well, yeah. Imagine there's heart surgery which has only 20% success rate. So every time 100 people go in for that surgery, only 20 survive.

Now, someone makes a robot that has 100% success rate, but every 100 operations, it has a glitch in the sofware, where it stabs and kills the patient instead. It has been calculated that on average, using the robot only kills 1-2 out of 100 people going in for surgery, compared to the old method of 80 people out of 100. The robot is clearly 80 times better than manual procedure, so why not keep using it?

The answer seems to be - because robot killing people is not an unavoidable outcome. It can be fixed. It's a software problem, a mistake that costs someone their lives. Similarily, an auto-driving Tesla may be killing less people on average than manual drivers, but that doesn't mean that any software problems that lead to it don't matter.

Step 1: don't recall the robot surgeon. While it does have a fatal bug, recalling eat would kill 79% more patients. Oops.

Step 2: Correct the bug, so the robot stops killing patients.

The Tesla autopilot may be in a similar situation. Even if there is a number of fatality inducing bugs, disabling it would be even worse. That said, 40% crash reduction is not enough. I want to see 75%, 90%, and more.

You are omitting the PR angle: "I DON'T CARE THAT FAR MORE PEOPLE WOULD DIE OTHERWISE - BAN KILLER ROBOTS NOW!!!!"

That one is easily dealt with: "Our robot surgeon reduces the fatality rate by 98.5%. Our study of the remaining fatalities suggest we can reduce them further. We hope to reach near-zero fatality rate within the year.

Don't even speak of the bug. Just state what matters: this stuff is better than what we had before, and it can (and will) be even better. This may prevent talks about killer robots.

This analogy doesn't work because you don't do manual heart surgery on yourself. Whether or not a robot does the surgery, the hospital's butt is on the line.

The biggest difference between an autopilot accident and a traditional accident is fault. Most accidents are driver error. Even if it drives accidents, overall, down (which it will), changing the responsibility for safe travel from mostly the individual to mostly the manufacturer has huge legal repercussions. Not to mention many people's discomfort with no longer being in control of their own safety, even if study after study shows that they shouldn't be.

https://en.wikipedia.org/wiki/Therac-25

The therac arguably saved way more people than it killed. However... it's one of the most infamous software bugs in history.

It's the same story you came up with above... except it's real, and it got pulled.

They did however improve since then.

Yes it does, if it's faulty in some way it should be fixed in future versions and the results of the investigation should be public so that everyone in the industry can learn from that, just like in aviation.

Apart from that, more people living than dying could also imply that certain groups of people, maybe kids or bikers, could be at a higher risk of dying while the majority has a much lower risk, resulting in a net positive. So the acceptance of self-driving systems would also depend on the distribution of the risk.

This is a very simplistic way to look at it. We're in uncharted territory here. Did crash rates go down because driver error went down? Probably. So accidents for which the driver is liable go down a lot. And accidents for which the car-marker is liable go up a little. That last bit is still a very big deal.

Probably? Why do you think it likely that driver error would go down more than 40%?

NHTSA reports that 94% of all auto accidents involve driver error. Autopilot is a feature that reduces the number of decisions drivers make. It would be improbable for the two not to be extremely related.

>>Even if autopilot was faulty in some way, if more people were living than dying - does it matter?

If you let your friend drive you somewhere in your car because you think they're a better driver, is it their fault if they crash?

The reason could be anything - you're drunk and they're not, you're in really congested unfamiliar area near where they live, whatever. If you think they're statistically going to be a better driver it does not absolve them of liability. Someone is always to blame.

It depends who you ask. I'd bet that the widow of a crash victim would disagree...

Isn't this the same argument that could be used in the context of hospital borne infections?

Autopilot will reduced the number of fatalities in the same way that going to the hospital when you are sick will reduce the number of fatalities. However some people that use Autopilot will die when they otherwise would not have if they hadn't used Autopilot, just like how some people going to the hospital will die from infection when they otherwise would not have if they hadn't gone to the hospital.

I agree! It's a net positive, and I'm in favor it.

But it doesn't mean that people dying from faults "don't matter". Every HAI-caused death is a tragedy, and they certainly matter.

The issue is much less black-and-white than the GP's simplistic view.

Yes, it does seem vague. Isn't the NHTSA investigation meant to be a non-biased root-cause-analysis about last year's trailer crash? Instead, this just seems to be a grading report on Tesla's autopilot compliance.

Section 5.3 pretty much sums up the root cause of this accident: the driver assist requires the driver to be alert, and the driver had 7 seconds to react and didn't.

There's no requirement that driver assist software magically become self-driving software in that situation. Put another way: it's nice if the software can avoid a problem, but not required.

The main remaining concern is that drivers not be confused about what they are required to do; there's discussion about that in this document.

Not really. The NHTSA report says that auto-braking is initiated only when the radar and MobileEye vision systems both report an obstacle. That reflects poor sensor fusion. Tesla took an off the shelf radar product from Bosch and an off the shelf vision product from Mobileye, ANDed their processed outputs together, and called it a day. They don't seem to have confidence data from each sensor and some means to fuse it. If the radar is reporting "target dead ahead, 3 secs to crash" (radars have range rate data), and the vision system is reporting "unrecognized situation ahead", that ought to at least trigger a slowdown and alert.

This is probably why Musk pulled radar and vision processing in-house. Combining the data from both sensors is more effective, but more difficult, than combining the results from processing both sensors separately.

And that's the problem of "aaaalmoooost self-driving cars, just a few corner cases:" 1. the corner cases are right where the risks are, and 2. the hype of "autopiloted" cars creates an unrealistic expectation of L5 autonomy.

NHTSA is unfortunately highly captured by the industry whereas NTSB has a separate chain of command that ensures more independence.

Very interesting, thanks for posting

Reply on news.ycombinator.com