Comment by tannerc | Hacker News Reader

Comment by tannerc | Hacker News Reader

In most complex systems the reality is people will encounter harm. The commercial airline industry is a good example of progress through crashing.

In 2017 there were zero reported accidentally deaths [1]. Compared to 1972, one of the most dangerous years to fly, where more than 2,300 accidental deaths occurred on commercial flights.

How did we go from thousands of deaths to zero? Primarily through crashing. Then working to evaluate what went wrong and improve the design, processes, and programs around commercial aviation.

As one pilot explains [2]:

"One of the ways we’ve become so safe was to realize that our efforts were never going to be good enough."

So yes, I think technologies like autopilot will save lives. But it won't be able to do it magically, unfortunately some harm will occur because we—in all our human wisdom—just can never accurately predict all possibilities. It's the same problem most self-driving companies are facing today, and the reason promises of self-driving being here today have fallen short. The world is far, far more complex than we realize... especially when we put a computer out into it and tell it to "go forth."

https://www.reuters.com/article/us-aviation-safety/2017-safe...

https://www.usatoday.com/story/travel/columnist/cox/2018/01/...

If Tesla's AutoPilot was regulated like the airline industry with post-crash reviews, mandatory improvements, and open sharing of data we likely would see that. At the current time Tesla's system is proprietary/secret and their oversight is lax.

If we look at the history of airline safety, going back further, we can see that it took many years and numerous crashes to get to the point where it became well regulated and crashes were independently investigated. Before government stepped in, private companies weren't doing a great job and their biases (inc. bad headlines/liability/cost) got in the way of improving passenger safety.

So let's absolutely keep with the airline analogy and regulate Tesla's AutoPilot like the airline industry.

Hopefully it takes less than ~50 years which the airline industry took to get to that point. Seems that this would need to become a popular opinion before becoming reality.

Regulation saves lives. We don't even have to look at the airline industry. Seat belts saved lives. Tailpipe emission controls saved lives. Tesla's Enhanced Autopilot and other Level 2 (partial autonomy) solutions will save lives.

Not always - it has to be appropriate - there are cases where regulation costs lives by locking in suboptimal solutions because the costs to re-certify can be too high.

We need to have the appropriate balance.

The NTSB does actually investigate their stuff. And then it does issue recommends.

I see a cost issue, easy to justify a full investigation every time a commercial plane crashes, how do you do a full investigation every time somebody crashes a car or is crashed into. How do you get the improvement loop that the airline industry had?

I thought the topic was AutoPilot crashes, not all car crashes. Is AutoPilot crashing enough to be cost prohibitive?

Not currently. If we get to mixed 50/50 traffic...possibly. (Saying "but won't happen, because magic future technology" would be literal handwaving)

If we required full investigations now, while the numbers are small enough for it to be feasible, then perhaps by the time we get to 50/50 traffic, the technology will be that much better.

At least plane crashes are usually mechanical and not software, or it's straightforward software (hello MCAS), investigating a machine learning's system behavior ("what input lead it produce this output?") sounds like a more expensive problem.

Until you look you don't know if i was hardware, software, road conditions or user, unless you look its just a a pile of twisted metal.

We already investigate nearly every car crash, it's just that we usually find one of the drivers at fault instead of one of the car manufacturers. The resources are there, currently provided mostly through police and insurance agencies.

> How did we go from thousands of deaths to zero? Primarily through crashing.

No, the reduction in deaths was through not crashing.

The airline industry would not have got safer faster by deliberately doing more crashing. "Okay, we need to increase safety, so let's crash as much as we can!" is something they didn't say, because that wouldn't have made any sense.

What you've done here, and the reason you've been upvoted to the top spot, is you've applied the Silicon Valley mantra of "failure is good" to airlines, and by extension, to autonomous cars. That mantra isn't really even true for software - it's not that failures are good, it's that learning is good - but it just about works in a world where people turn can turn free VC dollars into Medium posts and acqui-hires by failing, and hence where you might as well learn that way, rather than proceeding slowly with painstaking care; this approach doesn't really work in worlds where failures are underlined in blood.

I’d really like to see the details of the “proceeding slowly with painstaking care” looks like.

Airplanes never had airbags, crumple zones, etc. The odds of surviving a car crash are much higher than odds of surviving a plane crash. It seems therefore that testing in cars can be done with lower risk of loss of life. Yes, there will be fender benders and some deaths.

NASA manned space flight.

> NASA manned space flight.

NASA is horrible at keeping astronauts alive. Worse than the Soviets/Russians, as far as I know.

Apollo 1 and the two Shuttle disasters do not paint a good picture for NASA's safety in practice.

NASA is great at keeping astronauts alive. It has no launch vehicles.

Except the airline industry got there because of regulation and federal agencies investigating every single death (or even incidents, near crash without any deaths!) Right now, there isn't any equivalent in the car world.

I would like to introduce you to the National Highway Traffic Safety Administration (NHTSA) at https://www.nhtsa.gov/.

Here's an example of one of their famous investigative reports on a Tesla Model S crash while operating under AutoPilot back in 2016:

https://static.nhtsa.gov/odi/inv/2016/INCLA-PE16007-7876.PDF

The airline industry also « gives up » under a lot of conditions.

Scenarios where a plane would refuse to takeoff or divest to another airport... are conditions where employers would still expect their employees to get to work, or their deliveries.

I'm not interested in having a self-driving car until I don't have to pay attention to how it's driving. If I have to be alert and paying attention the entire time, then it doesn't provide a benefit.

That said, I'm totally fine with a self-driving car that gives up when it can't handle something. As long as it can safely remove itself from traffic and come to a stop (e.g. pull over to the side of the road), then alert me that I need to take over, that's fine.

To be clear, the impossibility of perfection is no excuse for reckless behavior. For example, if your system cannot reliably detect stationary obstacles blocking its path, and you are relying on the driver to save the situation, you should be using all the available technology to ensure that the driver is, at all times, alert and paying attention to the road.

You have to appreciate that a system which increases your overall safety but that comes with so much overbearing annoyance that users refuse to use it is itself useless.

If NHTSA mandates driver attention management systems in all cars then this calculus changes. If the driver attention management system is only required while AutoPilot is on, then the requirement is actually counter-productive to overall safety, e.g. results in increased deaths.

This reply ignores the very significant fact that Tesla's autopilot, and any other with similar capabilities, both enables and invites the driver to not pay attention to the extent that Tesla itself insists is necessary for the safe operation of the vehicle. If actually conforming to what Tesla says is necessary is too much of an annoyance, it is not ready to be on the road.

Very interesting to not have chosen 2019, a.k.a. The Year Of 737 Max. #cherrypicking

I guess this is how people justify killing for peace. Tesla stock holders must be celebrating that their cars get to run a few people over in the name of "progress". (Well ok, they get to celebrate because their failed pet project gets to have another go - potato/potato)

If you find something that's pure good, let me know. There's always a price to pay with any activity. The question is just if the positive outweighs the negative. If you can't see both sides the you're probably polarized and biased to one approach.

I think everyone wants better transportation and fewer deaths. But 0 deaths with an emerging driving technology sounds really hard. There are just so many edge cases and physics is pretty unforgiving on our bodies past 40mph.

> I guess this is how people justify killing for peace.

Extreme but (hopefully) relevant counter-example: was the killing of tens of thousands of people (soldiers and civilians) by American citizens in Europe during World War II justified? This is a very complex question, and any reasonable answer will be even more complex and nuanced, so I'm not really asking anyone to directly answer it in this venue.

To be clear, I'm not talking about Tesla here, only the first sentence in your response, in isolation.

I think there are some circumstances where some deaths as payment for far fewer deaths that are morally and ethically sound and defensible.

If we can agree to that, then we can move on and analyze what Tesla is trying to do.

Are we in a war with Machines now? Did Tesla state that it needs more Lebensraum, which, regrettably, requires purging the globe of everyone else?

I'm not trying to invoke Godwin here: just that war happens be somewhat hard to compare - "people die, ergo same thing" seems a rather forced comparison.

Perhaps the only fitting parallel seems "history will be written by the victors."

Also note how well all the other USian Wars On A Concept turned out...

Of course there are, and you're absolutely correct in your example - fighting against war is a noble cause.

But this is not a war in any way - there's already options in place to save lives - better policing, stricter speed limits, safer vehicles/roads/crossings. Allowing new technology on the road without the proper safeguards, (and the in my opinion abhorent justification that a few unlucky people might just get run over for the greater good) is not one of these options.

The "dangerous self driving cars on the road" option is only helping the pockets of Tesla's investors, not the public.

Reply on news.ycombinator.com