Self-Driving Mercedes Will Prioritize Occupant Safety Over Pedestrians | Hacker News Reader

Self-Driving Mercedes Will Prioritize Occupant Safety Over Pedestrians

blog.caranddriver.com

29 pointsTuring_Machine72 comments

Open articleView on HN

Relevant comic:

http://www.smbc-comics.com/comic/self-driving-car-ethics

And an actual extended trolley dilemma, wrapped in a joke:

http://www.smbc-comics.com/?id=3556

Ethically, are all people's lives equal? If not, is it wise to encode it into our robots?

From the article, the fact people won't use a car that favors the children over the occupants is interesting, and perhaps justifiable. After all, humans favor their own families and friends -- maybe it's equally reasonable for robots to prioritize the safety of their owners first.

Ethically, empowering machines to make ethical decisions is not worth a discussion. The reason that ethics is a philosophical pursuit is because it's unsolved. Humans can debate the merits of a given ethical philosophy, but machines cannot stop and contemplate the consequences.

We have never stopped engaging this discussion. Encoding it into circuits and setting it loose is not participation, it's profit.

For whatever is wrong with modern politics, and most of it is wrong, it isn't a closed book. We deserve better than that.

If we pursue continue to pursue autonomous technologies, machines will inevitably be put in situations where they make life and death decisions for us, because there isn't a human around to make those split-second decisions in real time.

So encoding our ethics is going to happen whether we like it or not. It's just a question of whether we as a society talk it out and make these ethical choices explicitly, or wait for corporations to make those choices for us.

That's my point. It's unsolved. We are allowing corporations to make choices for profit that humans have not resolved for ethics. We should categorically reject these sorts of automations because they present a danger we can't determine, according to the ways we evaluate danger.

"they present a danger we can't determine"

Why not? Counting the number of deaths per passenger-kilometer seems pretty straightforward.

What if self-driving cars wind up killing fewer pedestrians than manual cars, even if occupant safety is prioritized?

I am cynical enough to believe that ethics will have very little to do with it. They make good PR, but don't hold your breath expecting them to play out in real life. Ethics are complex, and complex in computing implies brittle.

Because life has a tendency to imitate art (satire in particular), I'll just note that SwiftOnSecurity captured the essence of the dilemma: https://twitter.com/swiftonsecurity/status/77302259638836019...

I wonder what this actually amounts to at the code level. Is it even possible to predict accident outcomes reliably? If not, the trolley problem isn't even relevant and there is no need to decide.

Given that they are planning to reduce accidents by 99%, it seems like political suicide to even take a position about the few remaining cases. I know this is counterintuitive, but I wonder if it would appear more fair for the car to decide randomly in those cases.

I agree with you but, do we know if the people who would be affected [despite the new technology] would be a subset of the people who would have been affected without the new technology? If the people affected are not a subset then I think it becomes more problematic.

> it seems like political suicide to even take a position about the few remaining cases

But these cases eventually will happen in real life, at which point whatever the car chose effectively becomes their 'position'.

> Is it even possible to predict accident outcomes reliably? If not, the trolley problem isn't even relevant and there is no need to decide.

That's just kicking the can down the road. As we collect more data and technology improves, we will know the statistical distribution of outcomes under various scenarios.

> if it would appear more fair for the car to decide randomly in those cases

Random choice is still a decision. What if there are 4 occupants? Should you change the weighting? If not, then you are implicitly saying 1 life is equal to 4, which doesn't make sense.

> If not, then you are implicitly saying 1 life is equal to 4, which doesn't make sense.

In countless ways it does. This is why the trolley problem is a philosophy teaching tool.

Utilitarianism is absolutely intuitive, but it isn't the only way to assess the morality of an action.

I agree, but whatever algorithm the car makes in that situation will be implicitly endorsing one or more ethical philosophies, meaning people who subscribe to a different one will have cause to object.

Which is why we should not enable machines to make ethical decisions.

So we shouldn't make self-driving cars?

Yes, we shouldn't make self-driving cars.

Instead let countless drivers kill pedestrians if such cars turn out to be safer to everyone around?

The ethics of this escapes me. It is not a nuclear warhead or a biological weapon. We can decide utility of self driving cars either pre- or post-factum, but not given no data.

Cars have very limited utility that can't be better solved with mass transit. Individual vehicles are grossly inefficient for basically every scenario except rural transport. Replacing those uses would nearly eliminate auto accidents and have a ton of other benefits as well.

My point is that the car (really the programmers) should avoid choosing at all. The difference between "the car is programmed to kill pedestrians" and "pedestrians are statistically more likely to be killed in accidents" is intent, which has a lot of ethical and legal weight.

Also, if your code has a decide_who_to_kill() function, Dieselgate starts to look like a faux pas in comparison.

Yes, yet by 'not making a choice', you still chose a definite probability distribution of outcomes. That distribution will need to be justified.

If some people die and others don't, people will ask 'Why did this person die and not that other one?'

I highly doubt that 'We flipped a coin' is going to be an acceptable answer.

What would you propose the software do in a scenario where it does have to decide?

Throw a fatal exception, appropriately, and stop operating?

I do agree with you, but I'm not certain you can avoid making a decision when your software will inevitably be provided a scenario where it has to do something, but no option is good.

All of that said I would be surprised if we expect our cars to swerve and dart around the road to avoid obstacles, I figured it was going to be down to slaming on the brakes in a fraction of a second in most cases. As human drivers, swerving makes things significantly more dangerous in most cases.

I would never want someone to explain to my family that the reason my self-driving car plunged over a cliff is because STATISTICALLY I had a better chance of surviving the fall than a child had a chance to survive being run over.

In other words, are you or anyone else reading this ready to die RIGHT NOW if it somehow saved the life of other humans? Probably not.

I applaud MB for what they are doing. If i an to buy a self-driving car then I would only take one that protects my life and the lives of my occupants (my children, family, friends) above the lives of all other people.

Google is rumoured to be developing a system that attributes a score to every road user based on their age, net worth, criminal history and the number of Facebook friends they have to help determine who to kill in the event of a trolley-problem like dilemma.

It doesn't need to reliably predict outcomes for the problem to arise.

Imagine there's an obstacle moving in front of the vehicle, which could be a bag blown by the wind or a cat or a child chasing a ball into the street. Are you going to write code that treats these cases the same? I hope not. I don't want to be sitting in a car that swerves all over the road for a plastic bag, or one that doesn't for a child.

As soon as you have code to identify obstacles and choose among multiple ways to avoid them, then these issues immediately come up.

" A self-driving car identifies a group of children running into the road. There is no time to stop. To swerve around them would drive the car into a speeding truck on one side or over a cliff on the other, bringing certain death to anybody inside."

Call me cynical, but such behavior seems logical. I would not drive myself manually to a certain death to save pedestrians that violated traffic rules, so why should autopilot do that to me? That raises an interesting possibility of future cars marketed by their "ethics".

> pedestrians that violated traffic rules

You might be able to make this call as a human, but can we assume the car is capable of deciding on that?

I assume autopilot would always drive according to rules (I doubt they would get licensed otherwise), so if pedestrians pop up in front of a car, I it's likely their fault.

I do find it hard to imagine a situation where the auto pilot has to choose one life over another -- I've never had to choose that and I've been driving 20+ years...

Someone runs out into the street, likely the car can brake or swerve. I've stopped for my share of kids chasing soccer balls without so much as a spilled latte.

Deer, moose, cattle on the road maybe it's different... The car will have to determine where to place the impact. But hopefully cars with more censors avoid these things better than people do. We are pretty bad at spotting deer...

Are we going to see driving tests for autonomous cars ?

Hopefully we will. It would be unethical to allow untested drivers on the road, automated or manual.

Pedestrians specified to be children. So, as a child, you never ran out into the street? If it was your child would that change anything?

Ah, a good, old "let's think of the children" argument. As I said, autonomous cars should always drive according to rules, and places, where children are expected to run into a street (suburbs, school areas etc.), usually have speed limit of 30 km/h (at least in Europe), so it's hard to imagine a deadly scenario at such speeds. However, if children managed to run into a highway, well, then blame their parents for neglect as this such not happen. To fantasize even more, what if there are my children in a car too? Should autopilot kill them as well to save these children on a road?

I'm not making a "think of the children" argument here, but rather pointing out the argument that you didn't really address when you changed the "group of children" to "pedestrians that violated traffic rules". If you actually saw a child appear in the street in front of your car, I don't think the violation of traffic laws would be part of your calculations in the moment.

Yes, the scenario given is absurdly contrived, but the issue is real. In any collision avoidance scenario, once things have already gone wrong, you have to make tradeoffs, sometimes involving potential dangers to both people inside and outside of the vehicle.

If children run into the highway, blaming the parents doesn't answer the question of what the car should do. If nobody else is on the road, you can swerve. What if there is a vehicle in the lane next to you, will you swerve towards it and risk a minor accident to avoid one that is more likely to be fatal? How much risk would be acceptable to avoid certainly killing a pedestrian? In highway driving sometimes bad things suddenly happen directly in front of you and the only choice is between different accidents. Humans make these decisions and autonomous vehicles are going to have to make them too.

Yes, these are (and with autopilot hopefully will be even more) very rare corner cases, but in such cases people instinctively act to survive, so autopilot doing the same is a logical step imho. Of course, if there's a choice between minor accident and safety of pedestrians (children or not, that doesn't really make any difference), autopilot should risk it. But the original case here was "certain death vs children on the street" which I still believe that autopilot should never do.

I think people instinctively balance their survival against others, and drivers differ. I know drivers who will take dangerous evasive maneuvers (to themselves and passengers) to avoid hitting a cat.

The difference with self-driving cars is that someone has to write the code, so these tradeoffs becoe explicit and potentially subject to legislation. I think there are easy solutions that sound good but aren't going to hold up after a few accidents happen.

"So, as a child, you never ran out into the street?"

Nope, I didn't. I was taught to never, never, do that at a very early age, and was supervised before that.

Yep, sometimes kids do escape supervision, and that's horrible. There is no 100% safety in this world.

This is the opposite of what it should do, but who will buy a self-driving car that may prioritise a person you don't know over yourself.

Ultimately, the software is good enough when the chance of death (to the driver and car occupants) is significantly lower than compared to a human driver.

Externally, self-driving cars should always be safer to 3rd parties.

As someone who cycles, and who has over 300 cycling websites including a 50k user London one... I will happily promote any campaign against self-driving vehicles that externalises it's danger. To effectively ban them from major cities until they are able to prove they are safer for all road users. I'm sure many other groups will feel the same.

So just because you don't drive means they should kill themselves to save you? Ridiculous. Clearly the morally right thing to do is to prioritize the driver, which the car exists to serve (within reason)

I sure as hell wouldn't drive a car that makes the exact opposite moral decision from what I would make, killing me in the process

> So just because you don't drive

I do drive too. Most cyclists are also drivers.

Feel free to purchase such a car, just don't have the expectation that crowded urban areas will permit vehicles that externalise their danger. There are more pedestrians and cyclists in London, with a much stronger lobby, than there are those who drive here, and even the pro-car press will delight at the chance to lay into self-driving cars.

> "This is the opposite of what it should do" ... "As someone who cycles, and who has over 300 cycling websites"

Something about self-serving? I think you only feel that way because you don't drive. Other self-serving people will naturally want something to protect them. Duh. Given what ScottBurson said, though, it's unlikely cars will be stuck in such a tough spot.

In my opinion, it should be optional on the passengers' part. Maybe some benevolent individual doesn't mind prioritizing other people's lives over their own. But those who choose to save themselves will end up having to explain, possibly in court, over their decision, rather than it being the car manufacturer who will never have a "correct" answer no matter which option they choose.

> you only feel that way because you don't drive

I do drive too. Most cyclists are also drivers.

Thank you for deleting my "I think"./sarcasm It was an important qualifier on my statement since I obviously don't know much about you.

It will be interesting to see the first time a car manufacturer gets sued for negligent homicide. The occupants of the car _can_ agree to a TOS that says the car will do its best to avoid killing pedestrians -- the pedestrians of course cannot (and why would they?) agree to said TOS. So from a liability perspective, I can't see this policy lasting long.

Yep - I don't want anything on the road that will prefer to kill me over its user who put it there.

Perhaps a different way to spin this to the selfish: would you prefer to have only one vehicle on the road that would prefer killing you, or would you prefer every other vehicle on the road to prefer killing you.

> I don't want anything on the road that will prefer to kill me over its user

Replace "user" with "self" and you have an expression against every other human with self-preservation instincts.

Believe it or not, people generally don't like killing other people, game theory be damned.

Witness, for example, the US army having to develop particular training to reduce the 20%-odd soldiers who ended up just not firing their weapon at all when in action (no data on how many fired but intentionally mis-aimed).

Of course! The occupant will usually be richer and much more well off in society than some peasant who still uses its feet to move from a place to another.

It's not classism, it's probability!

As a human being I would choose my safety over that of someone I don't know and I would choose my family's over mine in all circumstances. If I'm ceding control of my car to an AI wouldn't I want it to do the same?

Especially when the probability of the issue being the pedestrian's fault continues to rise as the driving systems get better- I wouldn't want my car to put me in danger because someone else screwed up.

That being said, if it got to the point where it could deduce the likelihood of injuring the pedestrian vs. the probability of risk to the occupants while maneuvering to not injure, that would be best.

The latter is why all cars should have black boxes. Including ones driven by humans. We can evaluate fault that way.

I think situations where one has to choose between someone outside the vehicle and its occupants are not very common.

They do happen. In fact, I know someone who almost had to make such a choice. It was in France (he's French). He was doing somewhere around 140km/h (call it 90mph) on a rural limited-access road, and came around a curve to find a car stopped in the left lane of a two-lane roadway -- apparently on account of a flat tire. One of the occupants was out of the car and walking toward the right lane, as if about to cross to the right shoulder. The timing was such that my friend was able to pass by in the right lane before the pedestrian got there, but had he been two seconds later, as my friend said to me, "I would have had to hit him".

All that said -- it was 20 years ago that I heard this, and I don't recall hearing any similar tales since. This one obviously stuck in my head, so I think I would recall. I've certainly never been in such a situation myself.

So the argument that self-driving cars will reduce fatalities so substantially that we don't need to worry too much about these edge cases makes a lot of sense to me.

>So the argument that self-driving cars will reduce fatalities so substantially that we don't need to worry too much about these edge cases makes a lot of sense to me.

Even more so considering this case likely wouldn't form part of the "edge case" pool, since a self-driving car presumably wouldn't be driving at 140 km/hr down a non divided rural road. The speed limit on such roads in good weather is 90 km/hr.

Perhaps the limits were higher in the past, but I can't imagine they were ever that high.

Sorry, I wasn't clear. It was a divided highway. Yes, there really was someone stopped in the left lane of a divided highway changing a flat.

They really should make it the option of the passenger. It removers manufacturer liability. You can always force the passenger to decide before driving, just like forcing Windows upgrades.

Say whatever you want about this, but they know their audience.

This. The reality is that no one will buy a self driving car that leans towards killing them over pedestrians.

Especially because people, when driving, blame someone else for all of their problems; this plays into the whole "Why do I have to suffer because that asshole ran out in front of me in traffic?" concept most drivers have and want reinforced by their vehicle AI.

Never mind the fact that you were turning right on the no turn on red, and they had the crosswalk...

The hope being AI driven cars won't turn right on the no turn on red, so the remainder of cases will really come down to pedestrian "negligence"

Yes, but my point is that the consumers vilify the pedestrian. Let's not even get into the cyclist.

Don't remind me; yesterday's discussion[0] about dooring was among the biggest shitshows I've seen on Hacker News. There's something Godwin-esque about online discussions concerning the relationship between cars and bicycles.

[0] https://news.ycombinator.com/item?id=12674533

What about the "swerve off the road into crowd to avoid oncoming truck" case?

The AI should be much better than a human at avoiding it, and the truck should be AI controlled too. In cases where something like a physical failure that is so sudden the truck can't warn nearby AI driven cars I doubt the car will be able to react, AI or not

This is just defining away the problem, though. You're not going to be able to orchestrate a flag day where every vehicle on the road changes from Level <= 3 to Level > 3, so Level 4 and 5 vehicles are going to be sharing the road with simpler vehicles.

While those cars share the road I expect the cars to protect their occupants. If a car was on a one way street with people on both sides and a truck barreling towards a head on collision I'd expect the car to try and swerve and possibly hit people on the way towards swerving, but stop as early as possible. I'd expect humans do that, not out of selfishness but out of reflex, and I'd imagine AI could perserve more life than a human doing the same (and not have to rely on reflex so that even in situations where the human might panic and hit someone on the sidewalk it would manage to stop if needed)

> The reality is that no one will buy a self driving car that leans towards killing them over pedestrians.

Ah, I was assuming OP was only calling Mercedes-Benz buyers heartless bastards, rather than all of humanity.

Also, I would, as would many other people. I think it's mostly a moot point, though: do you know of any trolly problem that's happened in real life? If something bad is happening, the car should probably just stop.

Ah but then the trolly problem is just behind you.

Laws and courts will of course vary throughout the world, but my understanding is that you're generally not responsible for getting hit from behind, unless you're reversing or something, so I'm not sure why this is any more of a problem for self-driving cars.

Unless of course the car behind you is also self-driving. But then we've really just recursed.

Based on the customer base alone, one would have expected this more of BMW?

This is arguably better than Microsoft/Apple/Google's strategy: send all the diagnostic data and then crash the system.

I feel self driving cars will have have a larger field of view so probably will not get into the same situations as a human plus in the future when all vehicles are interconnected the truck will already be aware of what the car is going to do and will avoid it as well

Yup...and you really don't want to be in a situation where other people can trick a self driving car into killing the occupants(for, example, stepping out in front of it from behind an obstacle). By always prioritizing occupant safety, those kinds of "tricks" become impossible.

It's odd that Mercedes would speculate on this when mentioned level 4 and 5 systems don't even exist yet.

The first time a self-driving Mercedes kills a pedestrian these remarks are going to come back to bite them.

In what situation does a self driving car run into a pedestrian other than them jumping/running onto the road within the cars legal stopping distance?

Swerving to avoid another car, software bug, sensor malfunction, a plastic bag getting blown into its field of view, and being interpreted as a dangerous obstacle...

This is good because I'd hate to be a passenger when the AI notices an opportunity to save a drunk homeless person passed out on the railroad tracks by parking the car i'm in on the railroad tracks to block a train. We all have to admit that even if AI functioned as well as the human mind, a lot of this stuff will be a tossup and like the "trolley problem" mentioned by WMF on this thread.

Reply on news.ycombinator.com