Hacker News Reader

Comment by machinelearning | Hacker News Reader

Question for people here: Is it justifiable to accept these flaws in the short-term if it results in a car that has a lower-error rate than human drivers in the medium/long-term?

View on HN

There is no need for us to accept the flaws in the short term. People can keep developing this tech without us mass adopting it first.

I considered this before posting, but I think that human disengagements are an important supervisory signal for the model. You can’t hire enough safety drivers to scale this process and releasing it is the fastest way to get there. Simulations are not going to get you all the way because you can’t simulate the real world perfectly. Do you have any thoughts on how to develop this without releasing it?

> You can’t hire enough safety drivers to scale this process and releasing it is the fastest way to get there.

I am sorry but I sincerely hope you are never made responsible for the release of anything remotely safety-critical.

Please re-read what you wrote. You are saying that because a business does not have enough money/resources to scale a process in real-world conditions that the solution is to release something and verifying in the real-world, on a public road, risking real human lifes?

If you don't have enough money to test it then you don't have an actual product/business.

> I am sorry but I sincerely hope you are never made responsible for the release of anything remotely safety-critical.

Wild to infer that from someone asking a hypothetical ethical question.

Why are so many on this site so aggro?

Had to google aggro, no aggression intended at all, but I frankly found the severity of GP question too scary to be considered.

> I considered this before posting ...

To me this is not someone asking a hypothetical but someone who is aware that what they are about to write is controversial yet are still considering it to be the best way forward. I happened to strongly disagree with that.

If there is clear evidence that these cars would have lower fatality rates than human drivers (I don't think this evidence currently exists), then holding off release to continue development is potentially a moral bad, for the same reason that we end clinical trials early when they demonstrate clear life-saving potential that would save lives in the control group.

I don't think it is morally reprehensible to ask what other people's intuition around this problem is.

> Is it worth killing n random people, for a uncalculated and unknowable chance of saving n+m lives in the future. The pool of people randomly selected to die have not volunteered or consented to be part of this project. There are ways to achieve the same lifesaving endgoal without the upfront sacrifice of lives.

The part which really strikes me as morally reprehensible is where the companies are saving money on test drivers and controlled test environments and externalizing those costs onto every other driver sharing the road with their training data collectors

This reasoning renders any governmental policy change of any sort impermissible.

It isn't coherent to apply these forms of deontological ethics to state action - a random set of people will die with both state action and omission of action, I see no reason why not to pick the option with the smaller expected number of deaths.

But this is all besides my original point: this is a legitimate moral debate to have and the rhetoric used by the above commentator was entirely uncalled for.

Hey man, I understand this is a sensitive topic that people have strong opinions on so proposing something that sounds controversial can trigger a lot of assumed conclusions. Can we take a step back for a sec?

First, before we assume that it is inherently risky to release this tech in the public, let's consider where the risk comes from.

There is some risk caused by the inadequacy of the technology to handle certain edge cases. This can result in the vehicle making dangerous maneuvers. However this risk is mitigated by allowing the driver to control the car as soon as the car does something wrong. I'd imagine that the vast majority of errors like this can be handled safely by a human driver who's vigilant and has control of the car.

Some unknown proportion of these might be unavoidable accidents caused by the self-driving software that a human would have probably made too (e.g. a deer running onto the road).

Another possibility is that the self-driving car causes an error that a human might otherwise not have caused and could not have been avoided by a human taking control. If you group the latter two cases together, and the accident rate is lower than that of the driver without self-driving enabled, you could argue that it has a positive impact.

The other source of error is human error. Some argue that self-driving makes people complacent and they might not be as vigilant as if they were operating the vehicle themselves. I think companies are trying to address this too by implementing driver monitoring systems, however this is completely avoidable by the passenger and its a stretch to say that self-driving cars are risking human lives because of this.

Hopefully I have conveyed the reasons why I don't think public testing is necessarily inherently risking human lives (of course this is dependent on the state of the tech being released). I'm sure you understand that every company has a limited runway and a window of opportunity to scale their technology. I am 100% with you on making sure that the product doesn't risk people's lives recklessly. However, I think the optics of this make it seem like far more lives are at risk than reality. I'm open to changing my mind as more information about the safety of the tech comes out but I am not de facto against it for the reasons I mentioned above.

Waymo appears to disagree with you, they ran tests with safety drivers and in controlled conditions for a very long time, and they eventually got to the point where there are customers riding in unmanned taxis. Tesla is just applying a "move fast and break things" approach to pedestrians' lives in order to satisfy shareholder interests and executive egos. They're not the only ones, as we saw from Uber's absolutely shameful performance that eventually lead to a death (safety driver involved but not solely responsible, of course)

I was under the impression that while Waymo did some tests unsupervised, most of the rides that non-employees can hail still have a supervisor. Is that not true?

I've seen youtube videos of unsupervised waymo cab rides, and when it got stuck in one they sent techs out to rescue it. It's probably still uncommon.

It depends on what kind of error it produces.

If the error rate is only 0.5% but the death rate of those errors is 100% I am not sure it will be justifiable.

Even if the death rate is low but the _perceived_ impact is high.

Ex: Imagine a week where self-driving cars have a bug that only mis-identifies grandmas. Only a few grandmas die but the perceptual impact is massive.

No, absolutely not. I'd say that Tesla flirts with the opposite outcome, that we saw with nuclear power. Early catastrophic failures resulting from premature deployment can produce a very reasonable revulsion away from the technology, such that the technology will have a PR problem well past the "break even" point. That folks continue to be bullish on self-driving tech despite cars repeatedly hitting stopped emergency vehicles at speed tells me that tech enthusiasts haven't learned from history.

The problem nuclear power has had is nuclear reactors are a great tool to build materials for a nuclear bomb. If everyone accepted them then access to nuclear weapons would have been a lot easier.

Are you implying that autonomous navigation isn't a stepping stone to powerful weapons?

Useful but not on the level of access to nukes.

That remains to be seen. Mass-produced autonomous drones could quite conceivably get similar kill counts as nukes.

It's about risk trade off. If it's 2x better than humans is it worth it? 10x? 100x?

36,096 deaths in 2019 in U.S. ~1.3 million worldwide (I couldn't find injury statistics this morning)

If the flaw is found before someone dies from it I'm not concerned. If 1 person dies instead of 10 I'm all for it. (I'd take 2x better than humans any day)

We can’t ignore core usability and basic safety issues by saying “on average, this is better.” End users can’t be expected to know that they’ll probably be fine, but an edge case they don’t understand will kill them one evening when they drive past a stopped ambulance.

Why not? If it saves 30,000 lives per year is that not a meaningful improvement? I suspect my understanding/acceptance of "unknown risk while driving" is flawed in some way, or at least very different from the general populace.

The problem is, it's not a level playing field. 100 incidents of a person ploughing into a bus queue and killing a child, each is news for a day, everyone accepts the tragedy and moves on. A self-driving car does it once though, and the mob will be at the factory gates with torches and pitchforks.

That's an interesting problem. I wonder what social discussions need to take to reach acceptance for "AI assisted driving". Mandatory AI breaking to prevent driving into a bus queue?

I'm also curious how to find the people who do object vs theory-crafting all possible concerns people could have.

The same argument would justifies human medical experiments, and we know they actually work, and result in faster access to improved drugs and vaccines.

Unlike autopilot which could still be crappy after 10 years

Yet we have made them illegal after some rather nasty precedents. Seems like hsitory repeats itself

I think the difference is that human medical experiments immediately cause suffering at a higher rate than whatever disease they're trying to fix.

I believe that there have been studies showing that today's "self driving" cars are already statistically safer than regular cars.

Hm. I find this very analogous to the MRNA vaccine debate: how high an error rate of a new technology do we accept, and to what degree does that choice have to be made at a community rather than an individual level?

I'd feel best if that decision was made at the smallest community level possible, so ideally county by county rather than federally. That lightens the burden of politicians making the wrong choice or being a citizen who disagrees with the right choice.

You get far more localized really shitty situations if you devolve to that level. Sometimes central decision making is good.

Having to switch the mode of operation of your car depending on what side of various county lines you are on seems like an obvious regulatory failure.

The error rate would certainly go up at first due to the additional complexity, but the cost of each error goes down. Besides, I have to switch the speed at which I'm driving far more granularly than at county lines, and no one's worried about a car's ability to handle that.

I don't disagree that sometimes central decision making is good, but in a complicated situation where any decision will have some negative consequences depending on the specific context seems like a textbook case of not being one of them.

This comparison gets me the most. People bringing up human error rate vs machine error rate. They’re not comparable at all. And it will be evident when you have majority autonomous cars on the road. Human errors are more or less RANDOM. Machine errors are NOT.

also, human failure modes are better understood, and can be better anticipated by e.g. you can still have a somewhat predictive mental model of how a swerving drunk drive might behave.

We don't have a good frame of reference for how machines might behave with their failures, which means that accidents could be worse than they would be otherwise.

The severity of the accident is an interesting point. Though it intuitively feels like there are ways for software to mitigate the severity of an accident when it realizes it is about to crash than a human who might be asleep, intoxicated or otherwise have a slower reaction time.

The machine's ability to recognize that it's about to crash may actually be one of the issues here, since often the self-driving/driving-assist car crashes are cases where the AI just completely misinterpreted the environment and made bad choices.

A human driver is somewhat likely to eventually realize what situation they've gotten themselves into (oh no, i can't stop in time) because of the multiple different feedback loops and information sources they're working with combined with their experience as a driver. For example, a drunk or very tired driver is operating with impaired decision making and response time, but they may eventually notice and respond - while an AI misclassifying a fire truck as a stop sign may very well continue misclassifying it until impact.

One way to mitigate this would be via sensor fusion - even if your vision or radar sensing fail, you can rely on data from other sensors to do things like apply emergency braking.

Unfortunately at least one vendor has decided to ditch radar, lidar, etc and just go with vision!

1. I’d challenge the premise that human errors are random. There are a ton of patterns that cause accidents including intoxication, low visibility conditions and tiredness. I haven’t done a statistical analysis but I’d hazard a guess that only a minority of human accidents are truly random. 2. Why does the randomness matter if the error rate is lower? Certainly if the errors are predictable, they can be discovered and fixed or avoided?

What do you mean with that, and why does the error type/"randomness" matter?

If I can reduce the error rate by 90%, but the remaining 10% are "random" (whatever that means), is that worse than not reducing the error rate?

The only part that is random is if it gets you this time. People follow too close almost constantly, if that would cause an crash every time only in the most rural areas would people be able to drive even one mile without a crash. Note that the cause is pervasive: following too close.

Others are bring up tired, drunk, texting... All real problems, but following too close is universal to nearly all drivers.

Reply on news.ycombinator.com