Hacker News Reader

Comment by brighton36 | Hacker News Reader

Doesn't this 5x requirement hurt more people than (say) 1.00001x? What am I missing here...

This is not the government saying "we the government require...". It's the results of a study of what people believe. people's risk tolerance is almost never rooted in a rational calculation. Risk tolerance is based on emotion, and self-driving cars currently trigger an emotional response.

As soon as self-driving cars become a regular part of people's lives and not an exciting new thing, the calculation will shift to a much more rational one

This calculation is actually very rational. What you seem to ignore is that, with conventional cars, there is a relatively small amount of "known unknown" risks. There are of course significant risks, but almost all of them are known not only in kind, but also in quantity. Drunken drivers, dumb people, broken brakes, whatever. We have several decades of data regarding these risks. The amount of "unknown unknowns" can also be assumed to be relatively low, given that the concept of humans driving cars has quite a history now and largely stayed the same for a good number of decades.

With autonomous cars, even once you have a few years of safety data from a large enough number of cars to be able to make the call of them being 5x less dangerous than human-driven alternatives in that data, you will still end up having much more "unknown unknowns" (of which I can't tell you any, because they are by design unknown) in addition to also having much more "known unknowns" like the possibility for large-scale software bugs potentially causing thousands of casualties at the same time. These risks will only go down slowly with enough time, there's practically no way of fast tracking getting these down, hence you have to incorporate a large enough risk buffer in your assumptions for rationalizing to even start using that fancy new tech, and the only place where this risk buffer can come from is having a much bigger difference in the "known knowns" department of risks.

Those unknowns already are being elucidated by experimental fleets. Self-driving cars won't be deployed en masse before the vendors can already demonstrate solid statistics worth hundreds of millions of passenger-miles, which will be sufficient to get the fatality rate.

How much does that tell me about potential software failure modes that don't kick in until a significant scale (speaking of double-digit percentages of all traffic, these test fleets are not even close to that) has been reached? Or about weird, but potentially fatal side effects of incorporating rules put up by regulators into the software that cannot be tested with today's alpha testing fleets because these rules might not even exist yet? Or about how good all these different AI vehicles of different vendors in very different software and hardware revision states interact with each other (think of situations like HFT trading algorithms that run each other into a doomsday spiral, just with vehicles at an intersection twitching around quickly in weird ways, trying to interpret each others actions)? Or about the hackability of future robotic cars (think for example of those slightly modified fake traffic signs)?

Nothing. That's why regardless of how impressively big these test fleets are, there will be a lot more of these unknowns.

Some of them seem like tail risks to me that are unlikely to dominate fatality statistics even if they were to occur and will be quickly patched or recalled if needed. Many of these hypothetical concerns could also affect existing driver assistance systems and aren't unique to autonomous vehicles. Hacking can also happen with human-operated vehicles. Interaction between multiple self-driving ones can also be tested with experimental fleets by concentrated local deployments.

I think this high level of certainty is basically just the governments way of acknowledging they are terrible at gathering/defining useful metrics and so with a wide margin there's very little room for error on the politicians part. I'm unsure if this is overly cynical, but I don't expect politicians today became career politicians by worrying about safety more than protecting their political status. Further, I suspect media would look for any definition possible to blame politicians for deaths so politicians feel it necessary to be blameless before allowing interesting, progressive ideas to materialize.

I would say that we tend to reduce human flourishing to exclusively being alive. I think the 5x multiplier maybe covers things like loss of liability in an accident, a sense of ownership of the vehicle, loss of privacy or obscurity, regulatory or operational infrastructure costs associated with a switch to self-driving, freedom of choice, etc. All of these have some ultimate impact on human flourishing beyond just a binary dead or alive definition. My opinion is, if these aren’t included in the 5x, they should be.

1.0001x is definitely not acceptable because dangerous drivers bring the total-human-safety metrics down. The "average" (median, or even maybe 25%ile) driver is probably much less likely to be in an accident (or fatal accident)than the drivers who drive most dangerously, e.g. frequently texting while driving or driving while intoxicated. So for most drivers, 1.0001x the average human rate would actually be worse than driving themselves, although they may find the risk acceptable.

You are correct.

However, how many people get hurt is not the only thing that needs to be considered. It's very likely that when a self-driving car would be equally safe as a human driver, the people who die in accidents caused by the car would be different than those who would die in accidents caused by human drivers, and so you'd end up with situations where individual next-of-kin could make entirely legitimate claims after accidents that their loved ones would be alive if not for the hellspawn car.

Trying to convince juries that it's alright because for every person who die in the cars, two other people who would have otherwise died got to live would probably be tough. Especially as the accidents that the self-driving cars are most apt to prevent are ones that could at least partially be considered caused directly by bad choices made by the driver (DUI, distracted driving, falling asleep at the wheel).

Once the data gets good enough that you don't need to do statistics on it[0], it becomes a lot easier to sell the idea to the public.

[0]: Relevant xkcd: https://xkcd.com/2400/

This is a psychological experiment. People aren't rational.

Reply on news.ycombinator.com