Tesla driver using Autopilot kills motorcyclist, prompting another investigation | Hacker News Reader

Tesla driver using Autopilot kills motorcyclist, prompting another investigation

theverge.com

20 pointsAlexandrB42 comments

Is the only way we will accept something like autopilot is if it has a perfect driving record? Is it not enough to be better than humans on average? Thousands of people die in automotive accidents, but we don't think twice when a human is behind the wheel.

Many people are charged with crimes related to automotive accidents (DUI, for instance, or laws related to cell phone usage). We require drivers to carry insurance. We are constantly regulating things to reduce deaths and injuries in automotive accidents (seat belts, crumple zones, airbags, forced recalls, etc.) - we constantly think about automotive accidents when a human is behind the wheel. Why should we just shrug when some Tesla mows down a pedestrian while the driver is watching youtube videos?

Always prosecute the driver. There's no such thing as "the car did it".

And if someone wanted to intentionally murder someone, seems like a perfect excuse "the car did it".

Unless we are going to victim blame.

But the point of the autopilot system is that there doesn't need to be a driver right?

So here you are saying, "Hey, we have this great system that is such a better diver than the human. Here are the statistics that show it." and then turning around and saying "Hey human, why were you not a better driver. You need to be paying attention."

A bit of wanting to have your cake and eat it too, isn't it?

"autopilot" is just a marketing term for the next decade, it's not fully automatic, it's actually "computer aided driving" (where driving is the human) but that description isn't sexy enough to sell

The GGP talks about "something like autopilot" acceptance and the eventual goal of the system is to cut the human driver out.

GP is talking about insurance and how we regulate drivers, stating that we should be looking at the same for autopilot systems.

Parent says "Always prosecute the driver. There's no such thing as "the car did it"."

At what point does the "autopilot" have any skin in the game so to speak? Is it a binary thing, one day we wake up to an update and the cars can drive themselves with no input?

Or will will be forever stuck in a quasi beta test, where the driver is liable for the actions of the system but has no meaningful way for controlling the system(autopilot) or even knowing how the system operates.

You can't have it both ways. Either the system is better than people and is going to take responsibility or it isn't. No one will accept a system they have no control over and do not understand but have to accept liability for.

There may come a day where someone is truly the passenger in the driver's seat in a car, like a person riding a bus.

Today is not that day. Nor is it next year. Or this decade.

A whole lot of people are going to die this decade, both drivers and victims hit, because marketing wants people to believe you don't have to pay attention once you turn it on rather it's an assist, not a replacement, for the driver.

It's not autopilot right now. It's driver assist. As long as the driver exists, the driver is responsible.

In an "old school" car, if an accelerator gets stuck and you mow someone down, then they find spaghetti code in the accelerator or it mechanically got stuck from design failure, you still have liability for killing someone but your prosecution won't be the same as someone who purposely drove over someone they were angry with.

The parent comment didn't mention wanting "autopilot" at all. Actually, as I type this, I'm wondering if your comment was unintentionally posted as a reply to the parent?

GGP talked about autopilot acceptance

GP is talking about holding Tesla responsible the same way we do with drivers ala insurance and regulation

Parent says to only hold driver responsible

So while the parent did not mention autopilot per se, the comment can only be interpreted in the context of talking about autopilot/ like systems. Otherwise, "Always prosecute the driver." does not make sense. That is the only option when you are talking about standard accident investigations(baring manufacturer defects e.g. faulty ignition switch, rollover/tires, sticky/abnormal acceleration pedals), nothing else in a standard car has ahem autonomy.

You are technically correct for tesla vehicles (though there's some grey area around how Tesla markets things - you can't just handwave away legal liability for a death with "but the fine print!"), but not for self-driving vehicles in general. Waymo taxis, for example, explicitly does not have a driver.

And Tesla has explicitly advertised that this Waymo scenario is where they're headed, so it's a question that's definitely worth thinking about now. "Blame the driver" isn't looking like a tenable position long term.

not sure how carrying insurance is related to accident prevention, Its mandatory in both cases.

The amount of cases we charge people for accidents is extremely rare because in most cases its an actual accident. People swerving into oncoming traffic, not checking both sides before making a turn...etc.

> Is it not enough to be better than humans on average?

No, not at all. The average driver is 1/3 distracted and 1/4 drunk. Autopilot and similar systems must be at least as good as a skilled, attentive, unimpaired driver.

> Is the only way we will accept something like autopilot is if it has a perfect driving record?

Putting it plainly, this is bullshit. No one -- no one at all -- is asking for autopilot to be perfect. People are asking for it not to drive full-speed into obvious obstacles.

> People are asking for it not to drive full-speed into obvious obstacles.

But isn't this a weird criterion? An obvious obstacle for a human might not be obvious to the computer -- but importantly, it could be the other way around too: the computer detects obstacles that are obvious to it, but a human would miss.

Shouldn't the goal be that the computer considers more obstacles obvious than the human?

I get the emotional argument that it's hard to convince a person off the street of this, but isn't HN a place for more nuanced debate?

It seems like you're debating the meaning of "obvious" -- "What is obvious? And to whom?" My meaning was, "obvious to a skilled, attentive, unimpaired human."

> Shouldn't the goal be that the computer considers more obstacles obvious than the human?

That would be excellent, as long as it's correct :)

In my view, the question should be whether it's more or less safe than humans driving. If use of automated driving results in less death than humans driving, considered at the broadest possible scale, then that strikes me as an excellent outcome. Instead, each and every death under autopilot results in all the hens clucking as though it's proof the technology has failed. That's ridiculous.

>The average driver is 1/3 distracted and 1/4 drunk.

Kinda curious where you are getting those numbers from.

> The average driver is 1/3 distracted and 1/4 drunk.

It seems to me the point is that an automated driver is never distracted and never drunk. The automated systems should get credit for that. That is, they should not necessarily be compared to a skilled, attentive, unimpaired driver. That is, as you admit, not the current state of affairs. Relative to the status quo, automated driving is becoming much safer.

>... must be at least as good as a skilled, attentive, unimpaired driver...

... dealing with motorcycles.

I don't care if it's 25% better than the public for non-motorcycles if it's an AI-powered cager.

It seems a bare minimum is releasing enough data that the public can assess whether it is better on average. Right now, Tesla releases data that show that when autopilot is engaged, it is safer on average. However, this data is not adjusted for the fact that the miles where users choose to activate it are on average safer. So really some sort of base-risk adjusted comparison would be the bare minimum I would like to see them publish. They really don't share much data.

Humans understandably are extremely averse to the idea of dying for a stupid reason, especially if control over the death is shifted away from the person, in which case they are more "furious" than "averse". It's why we hate drunk drivers so much. If I introduce a car that reduces deaths to near zero, but the remaining deaths are caused by a robot arm that is programmed to pop out and stab the driver, people are not going to accept that. If my car reduces car deaths dramatically, but the remaining deaths are caused by the car killing you by accident in an extremely dramatic way that you are confident you would have avoided if you were driving, e.g. swerving off the road and driving at full speed into a median, or into a truck that you can see but the car can't, people are not going to accept that either. You can not shift a person's ability to protect themselves away from themselves and occasionally kill them with your solution simply because your system is better than the average across all humans, even if it's much better.

It's frankly a little baffling that this part of human psychology even has to be explained. It's like the trolley problem: There seem to be some people who don't understand why you wouldn't simply pursue the result that holds fewer deaths - who don't understand why the trolley problem exists at all, what the point of it is. I suspect there is a relationship between these people and the people who don't understand how there could be any part of decision-making about autonomous vehicles apart from the raw numeric value of deaths vs non-autonomous vehicles.

Fair enough, but approaching the problem rationally, is that really how we should measure success? Are you satisfied with a solution that results in objectively more death just because you claim to have some part in creating it? That strikes me as ridiculous, and the outcome is that we're holding autonomous driving to an impossible standard.

Thought experiments!

Cancer is cured but 99% of its death toll is replaced by wholly random spontaneous human combustion. Is that better?

What if it wasn't purely random but instead the chance of your spontaneous combustion slightly depended on if your socks are worn? Is that better?

I posit both suck.

The difference here, I think, is that you don’t have to use autonomous drivers. To the contrary, notwithstanding laws against it, you do have to share the road with distracted and drunk drivers.

This is the reply among all discussion I've read about autopilot that captures my thoughts best, well said

I really doubt Tesla is even close from being better than human for now, seems like it needs to be in really specific and simple situation to act properly. There's countless videos showing how clueless it is as soon as it is in an environment that is a bit more complex, including attempting to run into pedestrians. Most humans don't do that.

This is just an investigation which is important in cases like this. They need to determine if Teslas can't see motorcycle riders at 1am in Draper, Utah. There's all kinds of oddities that can happen so these independent investigations are necessary.

Side note, imagine if you got bulldozed by a Tesla and died only to have someone write this comment. I know we live in a huge world and we can't think of everyone this way, but there should be a modicum of compassion for the guy who just died. His life has ended for what amounts to a potential programming error. It doesn't matter if another human could do it to. This man is actually dead. It's tragic.

Even if we accept some statistical failure rate, we should still investigate lethal failures. Like any good RCA it may uncover some learnings that could help the implementation and/or regulations around self driving.

I have long said that if someone invited the car today it would never be approved for use

Society is FAR FAR FAR FAR FAR more risk adverse which IMO is holding us back, we can not advance because perfection is impossible

The problem with an autopilot and the reason some of are so worried is because if one fails, the others will fail in the same way.

If one driver hits a bicyclist at a stop sign, we can assign the blame to the driver and figure out the cause i.e. driver inattentive, lack of reflective gear on bicycle. We can then take steps to mitigate this, driver education, requiring lights on bicycles.

If an automated system hits a bicycle at a stop sign, we can try and do the same analysis but my understanding is, we have to interpret the data and might not actually discover the root cause, in which case we have no way of mitigating the failure. Add in the fact that the explanation is difficult or does not make sense to the general public(i.e. the body of the trailer was white and contrasted with the sky as the car went up the hill(public: anyone can see the difference between a trailer and the sky!))

We understand how and why people fail, we can empathize with them and assign blame to them. We don't understand how the computer is failing and can neither empathize with them nor are we actually assigning blame to them(This system is in beta; You must have you hands on the wheel at all time and be ready to take over; X is not liable for any damage caused by use of this system, the user assumes full liability etc.)

If you have proof that autopilot is safer I'd love to see it. Cherry picked and non-standard data selectively released from the company selling autopilot for thousands of dollars doesn't count :).

For one, "better than average" is hard to measure / subject to lots of confounding factors, so people (rightfully) find those claims hard to believe. For another Autopilot often fails in different ways than human drivers, so it's easy to find examples where a human driver would have been fine, which exacerbates the perception that these are not in fact better.

Yes we do. We have a ton of rules before, during and after humans are behind the wheel.

You are correct that we should probably be thinking even more about it, and other countries are doing that, slowing down cars, providing separation from other modes of transport and pedestrians, and reducing car dependence altogether.

Note that the "looking into 16 crashes" part of this article pertains to:

> PE21-020 is upgraded to an Engineering Analysis (EA) to extend the existing crash analysis, evaluate additional data sets, perform vehicle evaluations, and to explore the degree to which Autopilot and associated Tesla systems may exacerbate human factors or behavioral safety risks by undermining the effectiveness of the driver’s supervision. In doing so, NHTSA plans to continue its assessment of vehicle control authority, driver engagement technologies, and related human factors considerations.

NHTSA is evaluating whether or not the fact that Autopilot allows you to 'lock it in' makes you less attentive when driving. Compared to Ford's Blue Cruise[0], it seems like it's better for regulatory safety to make a system that can turn off at any time and doesn't make loud noises to alert the driver when it turns off, so that the driver won't expect the system to work without them looking at the road.

0: https://youtu.be/GCRNYP5Qg34?t=325

It's weird to compare the fact that autopilot could add 9's of safety to situations that are already 99.99% safe, which at the scale of "almost all drivers" means a lot of avoided deaths, while at the same time potentially adding risk to situations that with humans in the loop would be much safer.

For example, if it makes deaths on highways go down by 1 in 10k, but now you need to accept the risk of the car turning you onto oncoming traffic at an intersection because it was bad weather.

It's a tough change to accept when the failure hits you personally, and not one I know we should blindly agree.

That road was completely empty and that guy must have just caved in and started looking at his phone. You didn’t see a motorcycle with functioning head and tail lights at 1AM? Sure. The article says these crashes are because of Tesla cutting corners but everyone who has access to FSD is made aware that this is beta software and that human intervention might be needed at any time. It’s definitely the drivers fault, not Tesla. I don’t want an entire division of the self-driving effort to be killed because of some frumpy journalists.

Humans will human. I'd you give someone a system that doesn't require their attention or put significant effort into holding it for long enough, they WILL quit focusing on it and become distracted. Especially in something like a car where they can't get up and move around to keep alert. If your system doesn't account for the human, your system is faulty.

Right and this has been known across a broad range of activities for a long time and is exactly why a reasonable company like Waymo decided against providing better driver assistance and instead focused only on actual self driving.

If system marketed as self-driving can't notice other road users in those conditions it should be banned from use until it can. I could reasonably accept if there was very heavy fog or blizzard limiting human visibility to a few meters. Anything else, shut it all down and let them prove that it can not ever happen again.

Doesn’t make sense when it can never exist unless it grows on the road, regardless of which company you’re talking about, and also when it’s already statistically safer than humans. If your standard is “never ever again” then you either have to admit that you want all cars banned or that you’re wrong.

For context: there are ~6 deaths per billion miles driven in the US[1].

According to this article, Tesla autopilot has killed 19 people since June 2016. That means it would need to have been used for 3 billion miles of travel in order to just be as safe as the average driver.

1. https://injuryfacts.nsc.org/home-and-community/safety-topics...

I ride a motorcycle, and this is the second most serious accident after turning in front of a motorcycle (typically left hand turn in the US at an intersection). Both cases are because the driver didn’t “see” the motorcyclist or misjudged their distance.

Maybe Tesla should do a recall to install a flashing light on top when drivers are on auto-pilot.

Teslas are high on my list of vehicles i scrupulously avoid being around on two wheels.

For precisely this reason. Visual spectrum object detection at night when i'm on a vehicle that's underrepresented in training data and also a difficult visual target means that i want to be nowhere near you.

Reply on news.ycombinator.com