Why We Should Adopt Driverless Cars That Kill People | Hacker News Reader

Why We Should Adopt Driverless Cars That Kill People

medium.com

24 pointssonabinu45 comments

>> Regulators test new products until they feel certain that people will not be put at risk. Inventions from elevators to aspirin were subject to fierce scrutiny before they were broadly adopted. We can’t apply the same level of scrutiny to driverless cars.

This article is clearly written by someone who has never worked on a safety critical system. There is a method for establishing risk and making risk-benefit tradeoffs, not just feeling certain. If we did that many more people would be killed.

If the author is arguing about public scrutiny, then I generally agree with the argument, but existing safety engineering practices will work for driverless cars.

Source: I work on hardware and software for surgical devices (among other things).

I think he's arguing for virtual rather than real world testing to verify extremely low probability behavior. At $0.55 a mile, it can be hard to test out to five standard deviations, but if the question is, does the software correctly handle these events, it is really just a test script.

That's fine and that is already an acceptable method so long as you can make the case for why it's representative and do some real world testing to validate it.

5 standard deviations? Who cares about handling lots of easy situations.

Just create a big old test suite, procedurally generated. It would make for a nice game and driver training tool too. Verify against average and expert drivers to set grading. Additionally get statistics about prevalence of bad driving, which will allow you to check if the driverless system is safe enough using plain old cost-benefit. Moreover, include assisted driving systems (emergency takeover) in the evaluation.

No need to require perfection, failing gracefully is also acceptable. For example, in a weird situation, the car could slow down and engage notifications at drivers.

I worked for the FAA on these sorts of Risk Analysis Panels when I was younger. It's basically a trade-off of the likely damage or impact (a rating something like stubbed toe to everyone on the plane dies) vs the chance it will happen (once a day, year, 100 years, shouldn't ever happen but can maybe possibly).

An example of this is babies in laps on airplane. What is the benefit? People with kids will fly rather than drive. But are kids safe on a parents lap? No not really? Are they SAFER on a parents lap in an airplane than in a cross country trip in a car-seat? It turns out they are. So you have babies in laps on airplanes. It's an analysis of safety not a mitigation of situations where people might, with extremely low probability, die.

I think we can charitably take it to have an implicit "by the appropriate standard for evaluating risk". That is to say, "Regulators ... feel certain people will not be put at risk" -> "According to the industry standard(s) for evaluating risk, the new product is no more dangerous than the prevailing situation".

> I encourage our greatest minds to build the tools that will computationally create almost all possible scenarios a driverless vehicle can encounter

That's not as simple as the writer seems to think it is. Autonomous vehicles can only interface with the world via sensors, and sensor data that looks real, but is actually computationally generated, is one of the best ways to make a model overfit. It can learn the exact function and simplifications a developer has made in order to generate the fake data.

Then you can say "alright, record the data and modify it". But you can't actually modify, for instance, point-cloud data that much to account for arbitrary viewpoints. What you can do, and what most companies do afaik, is record the data as a "unit test". The car then responds to the test and, given the result actuation, it is evaluated whether the test passed (e.g.: you want to drive into the sidewalk? test failed). So if the car wants to do a correct maneuver in the test (i.e., one that passes the evaluation), but it is different than the maneuver the test has recorded, then you suddenly don't have the data to assess how well the car did. I know there are ways companies use to solve this but I assume it's not generating data, given the problem presented above.

tldr: generated data isnt as straight forward and can lead to bigger problems later on.

> So if the car wants to do a correct maneuver in the test (i.e., one that passes the evaluation), but it is different than the maneuver the test has recorded, then you suddenly don't have the data to assess how well the car did.

Wouldn't engineers notice that when examining why the car failed the test, and modify the test to allow the new correct maneuver?

Do we really care how well the car did, rather than if what it did is safe?

Instead of generated data, try hallucinated data using AI to generate weird situations. Albeit random training is also pretty good, as is parameterization.

> These learnings are immediately disseminated among the entire existing and unborn driverless car fleet

You can stop reading there. Unfortunately we are not living in an Open Source world. Walled gardens will keep improvements for themselves, knowledge will die with dying companies, version incompatibilities will create a complex situation.

I am very willing to believe in the thesis of the article, but the way the Software industry has been evolving recently does not give me much hope.

The author has seriously underestimated (or, more likely, overlooked) the difficulty of a key step in his proposal: enumerating all possible circumstances. How many different "truck crossing ahead" scenarios are there? How many different ways might something be misidentified as the sky? or something else? How many combinations of basic errors are there, in which the basic errors themselves may be harmless, but the combination is not? Even deciding whether two cases are the same or different is often a matter of opinion, and it is further complicated by the fact that what is the same or different to a given system depends on details of the implementation of that system.

The author is not shy about making other assumptions, either: "Sebastian Thrun postulates that, when subject to real-world conditions, AI can double its performance every 18 months" "let’s assume that a driverless car is perfect at navigating scenarios it has previously handled safely"... The foundations of this article's argument are not nearly as sound as the author seems to think.

What a crock of shit. Would we accept the same reasoning for some barely-tested miracle drug? "If we do the testing, and the drug turns out to actually be miraculous, then lives were needlessly lost during the testing period". Indeed ... but what if the drug is not miraculous, or has dangerous side effects?

Hands up who wants to trust their health to the untested assertions of a pharmaceutical company

"Hands up who wants to trust their health to the untested assertions of a pharmaceutical company"

Well, that depends. Do I have a mild case of acne, or am I dying of cancer? My willingness to try an untested drug is likely to be very different in those two scenarios.

Drugs are rated against state of the art treatments. In driverless cars, this would be against human drivers.

Except you also have the liability issues. Car companies will be unwilling to accept the liability that drivers currently assume.

When cancer kills someone, you can't sue cancer.

"Willing" won't really enter into it. It depends on what the jury decides, and the car companies don't really have any say in that.

That's my point exactly. It's not simply a question of whether driverless cars are better drivers than humans or not...

If a driverless car kills someone, the car maker is the only one I can think of who would be responsible. A passenger who wasn't driving can't be responsible. So legally the insurance needs to cover this. If they can't be insured, they can't be on the road.

I'm not sure if I expressed that point well, but what I was meaning was that the car makers will have to accept the blame if they're going to be selling cars that do the driving (or more precisely, take the burden of the blame for bad driving).

As an aside, to muddy the waters even more, I'd say even if the car gets hacked and made to kill someone, the car maker might still be legally responsible. It's a minefield for sure.

But what if you taking the drug for your cancer kills me? (I mean the self driving car you bought for your inability to drive kills me).

Well, you hire a lawyer and convince a jury that's the case.

If the car's software is defective, the company should clearly be held accountable.

This is a bad analogy. I don't think there's really any question as to whether driverless cars in the long term are going to be good. The only question is when they'll be good enough to start using en mass. And the point the article is making is that, unlike wonder drugs, driverless cars will be constantly improving themselves while being used. So if you start using driverless cars at the point where they're expected to be marginally safer than the average driver, then in a very short amount of time they should become significantly better than the average driver (and should still keep improving).

You can't just go updating critical safety software on a driverless car remotely all the time.

Each update will need to be tested and approved for safety for every vehicle it needs to run on.

Also companies will understandably be unwilling to share their data with competitors, and the data is probably not translatable between companies anyway.

This is actually a good analogy. Driverless cars will be good, but what one company claims as driverless car technology might not ever be good. For example you need complex machine learning algorithms (Well I guess even a naive bayes will do) for facial recognition. But facial recognition cannot be solved by some hardcoded logic like checking the relative sizes of two circles and a cylinder you assumed as eyes and nose. Like this example, it's possible self driving car is far difficult than just adaptive cruise control, or Tesla's radar looking two cars ahead etc. These are minor hardcoded logic and is no guarantee of solving the problem.

Plus, a bad pharmaceutical drug only harms the user, a dangerous car on the road is dangerous to people who don't want them.

This is part of the wider issue of who is responsible for driverless cars. I buy insurance for a car that I drive, and the company charges me based on risk factors (age, where I live, how often I drive, etc) and the driver alone is ultimately responsible for the actions of the car. If I hit someone I am responsible and liable, if I'd been drinking I am prosecuted and the family of whoever I hit has a focal point to blame. But with a driverless car there is none of that, where is the agency, I may have asked the car to take me from A to B, but would it be valid for me to be held liable for actions the car takes on the route that I have no control over? This almost leads to fatalism, there will be people injured and killed, but there is literally nothing we can do to predict when or where these incidents will occur. I agree that there may be a strict statistical case for them, but I can see why it will be very hard to get past the human aversion to them.

In Germany those decision violate the first article of its Constitution: Human dignity is inviolable

I think those decisions are necessary are driverless cars but will start a huge discussion about the ethic behind calculation the minimum victim. This includes making decision if saving a child is more importing than saving a elderly.

This is why I think vehicle autonomy will never happen the way we want it to.

Collectively, the death toll of driving will be significantly reduced, but it will only take a few people being killed by driverless vehicles for people to make a fit about it, because there is a clear victim.

In an accident, it's exactly that: an accident. It wasn't an intended decision, it just sort of happened. With AI and autonomous vehicles, everything is a decision, and someone gets the worse outcome, and every little bad thing that happens is going to cause an uproar.

I'm so sure that something similar has happened before, I'm struggling to come up with it though.

I do not know if this is real but in Hot Fuzz (there goes the credibility) the main protagonist insists that traffic accidents are actually collisions. The name accident implies that there is no culprit but I believe in vast majority of fatal crashes there is a faulty party.

Some older statistics I read claimed that (in UK I think) 30% of crashes were due to alcohol and another 30% to fatigue. In these cases for example there is a culprit and the crash was no accident. For me, accidents are cases when a mechanical part breaks or an oil spill makes the vehicle incontrollable. But people crashing into each other because of alcohol, fatigue, boredom, distraction or speed should be responsible for their acts and should not claim being in an accident.

>In an accident, it's exactly that: an accident. It wasn't an intended decision, it just sort of happened.

I think that one of the issues is that it's often not just an accident, not something that "just sort of happened". Safe driving practices do work. E.g. checking your blind spot, not running stop signs, always looking both ways before going when the light turns green, etc. Of course, you can only do so much to protect yourself against other people's bad driving. But there is a moral component to car accidents; they happen more often to bad drivers.

You're absolutely right. I 100% agree, and there's a single person at fault whom victims pursue. Who faces the blame whenever a driverless vehicle is forced to make a decision to strike a pedestrian or hit a wall rather than a head-on collision? Driverless cars do.

We may not like it but that's how the public will see it.

You can show that the car did its best, statistically, or at least no worse than a panel of drivers (jury). Black boxes have a long tradition and self driving and assisted driving cars should have one.

The answer really sometimes is "you are out of luck".

For a bug or a really bad known issue, authors and designers (car company) should be liable.

I'm pretty optimistic that the clear and vast befits of self-driving vehicles that benefit everyone from commuters to shipping companies will push over that "new and scary" hump.

Just look all the civil liberties and privacy that we have given up over the last 15 years in exchange for "safety" and being connected through social media and the internet in general.

If you would have told someone 20 years that we were going to start making teachers wear body cameras to record every word and movement of everyone they interact with, they would thought you were crazy, yet here we are.

I want those two hours of my day back, to use upon my laptop- if that means i have to gamble death with a better loaded dice, im in

Most accidents are also decisions. They just often involve unconscious ones, or ones made in a split-second without thinking (e.g. "do I swerve this way or that way?"). Humans aren't that good at doing detailed cost-benefit analyses during split-second decisions, and so we call the results "accidents", but all that really means is the human wasn't skilled enough/fast enough to prevent the problem, whereas a computer in many cases will be. If a driverless car is forced into a situation where it has to have a potentially-fatal collision with someone, a human in the same situation would have the same problem.

So really, the only question comes down to, what's the ethical approach to deciding how an AI should prioritize human life? If a driverless car is forced into a situation where it must kill one of several people, who dies? (in practice that probably comes down to figuring out which action is the least likely to result in a fatality, even if all actions are extremely likely to result in one they're not going to have identical probabilities) And perhaps more interestingly, if the car has to decide between probably killing its driver, and probably killing someone else, which way does it decide? (though again, this probably comes down to which person is more likely to survive the accident, and that's probably going to be the driver because they're protected by multiple safety systems whereas someone walking along the road is not)

Brakes. It should use the brakes.

This isn't about the trolley problem (it's like Godwin's Law for autonomous vehicles, I guess), it's about the fact that we already routinely license drivers that end up injuring and killing people and are making no particular effort to improve that process, yet act like autonomous vehicles aren't acceptable until they're orders of magnitude better.

Every person about to be involved in a traffic accident needs to make that decision, and the decision each individual makes can vary wildly. Maybe to hurt the least amount of feelings the car should just do a coin flip when its only choices are hitting 2 people.

There is sometimes a way to indeed reduce an impact of a crash. Coin flip is not really the only thing acceptable as human drivers usually act selfish. Therefore such a strategy should be acceptable. As would a strategy based on statistics.

The only important part is that the driver is informed how the car may react in terms of the strategy and that the decisions of the driving system are recorded.

I find it so hilarious that these threads always seem to degenerate into moral discussions about some fantasy world where driverless cars have to understand the meaning of everything in the world in order to make judgement calls about the worth of different human lives.

Literally all a driverless car has to do is hit the brakes if it's going to hit an obstacle. There's no moral decisions, no discussions about who deserves to live and who should die. Just use the brakes.

I just wrote the code. And it's open source so any car company is free to use it. Here it is:

if (obstacle) then { brakes = true }.

You're welcome.

There is at least one start-up I know working on this very problem using simulation to prove that automotive software can handle these extremely rare events without having to do trillions of miles of real world testing.

which one?

I assume they're reading this thread.

"To a first approximation, let’s assume that a driverless car is perfect at navigating scenarios it has previously handled safely."

As a back of the envelope calculation, I was with you until that sentence. That is an unwarranted and very aggressive assumption.

Speculative summary: because cars with drivers kill more people, and while drivers can be assisted with technology, there's more opportunity to drive the accident rate further down with driverless cars.

This is assuming that assisted driving is worse than autonomous. Until a serious study is ran and a good test suite designed, we are left with fear and suppositions.

Does this ever once make the argument suggested in the title?

ph'nglui mglw'nafh Carthulhu R'lyeh wgah'nagl fhtagn

Reply on news.ycombinator.com