Will your self-driving car be programmed to kill you?
uab.edu
uab.edu
Thus the conventional trolly problem doesn't really do the problem of self-driving car decision trees justice. The school bus isn't a dumb trolly. When your car is heading toward the school bus, the school bus will also running down it's decision tree and the best outcome based on its 15 passengers may be to take you out directly rather than run the risk of miscalculating what will happen if your car dives into the guardrail. Which is to say, that statistical predictions bring confidence intervals into play.
I believe this is why trolly problems in general reveal more about their formulation than about our ethical reasoning. We will throw a switch to shunt the trolly because the bad outcome will remain in the future and the possibility of changing circumstances remains very real. We know from experience that any of our predictions may be fallible and that the more temporally remote the event the more our prediction is fallible. Pushing the fat man off the bridge elicits a different reaction: the outcome is immediate and our prediction less fallible.
It will be designed, I imagine, to do the safest thing possible given an emergency. Which is in this instance, as most people suggest, stop as quickly as possible.
Nothing to see, move on.
[1] http://www.nytimes.com/2011/02/17/business/economy/17regulat...
As silly and pointless as it to worry about whether robocar morality should be primarily deontology or utilitarian, I have to admit I love it. I'm a sucker for this kind of science fiction thinking, brought on by everyone's expectations of technological progress. The average person has a tremendous amount of confidence of the "they'll figure it out" kind.
I think it's a good sign, the opposite of cynicism. Also, I think moral questions are interesting and if teaching is the best way to learn, maybe imagining how to teach a machine to be moral is a good exercise.
On that note, deontology fits nicely with the way computers tend to think - rules. OTOH, utilitarianism means seeing the future - a nice use for all our future computational power. While we're on that, do computers have a duty to be true to their nature?
This debate about what autonomous vehicles will do in extreme situations is but the "tip of the iceberg".
Until all autonomous drive systems can be "unit tested" under a standard suite of simulated sensor inputs, one would be wise to assume that they are wildly unsuitable for anything but the most trivial driving situations (eg. Golf courses, closed estates).
The main problem right now is -- there is no way for you to know how any particular autonomous driving system will handle it. With every second auto maker and start-up claiming to have independently developed super-duper auto-drive technology, I think it will get worse before it gets better.
Without standardised testing frameworks (ie. sending my car's autonomous driving system through a simulator to see what it will do), I remain ... unconvinced.
Being an industrial programmer for 30 years, I think I'll continue to err on the side of assuming incompetence and failure, until proven otherwise.
This is because one thing. They all can stop quickly and will have their own lane on all Interstates.
It seems that 60's AI is back in fashion and now part of autonomous car culture. There's this faulty idea of this super brain driving these cars. The reality is we're looking at dumb as rocks firmwares that can't even drive in the snow or heavy rain properly. Frankly, I think these firmwares should stay dumb like the human reflex system. Too much "engineer logic" and other questionable heuristics will just lead to bugs, lag, etc. I don't want my car to drive off the cliff because of some software bug. I don't need some Samsung engineer on a time constraint who couldn't spare the cycles for proper edge detection because he was wasting those cycles on 'sport mode' which marketing made a big fuss over delivering this quarter. I want dumb and simple code to do what's a dumb and simple task.
-------------------------------------------------
5 people <-- car w/ 1 people <-- heavy truck
-------------------------------------------------
b) The car hit and potentially kill most of the people standing, but save its passenger
c) The car can jump to the void
Scary as it sounds, what the car would do depends on the software
I trust sensors to detect icy conditions better than a trust myself.
[edit] Most bridges where I am have signs that explicitly warn that bridges freeze over. And people intuitively know... bridge + recent precipitation + cold whether = slow down. I don't know why a computer wouldn't know that.
If so - self-driving cars will be going much slower than manualy driven cars most of the time, which will probably hurt adoption.
I would like this to be the case, but I guess car companies will agree to some compromise to speed up adoption, and corner cases will be there. Should the software ignore them, or plan for them (= "planning to kill you").
A) all day long.
The program begins with a moral conundrum.
You're working with a crew doing track repairs (presumably, using heavy equipment such that you can't hear much else) when you happen to look up to see a locomotive rushing towards five of your buddies who are working on the tracks. But just as quickly you realize that you're standing next to a rail-switch which you could use to divert the train to a second track on which only one of your workmates is standing.
Should you throw the switch?
Ninety percent of a great-many people who've been polled say they would throw the switch.
The next scenerio has you and a coworker (a very large fellow) working on a footbridge above the tracks. Again, you look to see the locomotive bearing down upon five of your friends who are working on the tracks. You quickly realize that the only way you can stop the train is to push the big guy standing next to you off the footbridge, whereupon his massive body will stop the train (yes, I know, but just go with it).
Should you push your coworker in front of the locomotive?
Ninety percent of a great-many people polled say they would not push the man in front of the train, even if by doing so they could save the other five men.
Now we have your in a car and it could choice to turn right or left into a tree killing you or continue straight and kill 5 children who were crossing the road. What would you want your car to do?
One that brakes, well that's different.
When faced with a split second decision whether to live or die, humans choose to live, that's how we made it this far...
we are relative superior to foxes - who hunt for necessity, superior to us might not hunt at all
i agree btw
i am not worried that a superior dreadful AI will come to kill humanity, but that some little half-assed AI in charge of some minor side activity will not be capable of understand context beyond it's own complexity.
The AI evolution is completely different - "nobody is going to kill you except human and you don't need to reproduce yourself, , that's usually human's job." I believe the result is a completely different inner drive for the AI - which is more likely to be more useful to human, that's how you get selected.
The most superior trait in humans is how often they wonder in what way they are to other life forms.
Opposite of "planning to kill" is "killing randomly".
Self-driving cars are likely to be safer than human drivers anyway, so one has to consider how much risk there is of a situation like the Trolley Problem arising - i.e. not much.
It's not as simple as it sounds at first.
As for the concept of "safer to the community as a whole" - well, the idea is that they're a safer product overall, and this is mostly targeted at the individual (i.e. "if you own a self-driving car, you're less likely to die"). If people have the knowledge that their cars may (in certain situations) elect to kill them in preference of others, then I doubt that self-driving cars will sell particularly well.
Cars are not on trolley tracks and computers can brake or swerve harder and faster than any human could.
Do we get the best outcome when everyone expects the computer to spare the pedestrian? When everyone expects the computer to drive "selfishly" or to try to minimize its own impact velocity?
I don't think it's a good idea to program the cars with a "utilitarian" policy of saving the most human lives possible, as not only does that require quite a lot of inferential power to be packed into the car, it doesn't set up any concrete expectations for the passengers and pedestrians. You would have to add a mechanism for the car to first, in bounded real-time, infer how to save the most lives, and then signal to all passengers and bystanders exactly what course of action it has chosen, so that they can avoid acting randomly and getting more people killed.
This is why we always demand very regimented behavior in emergency procedures: trying to behave "smartly" in an emergency situation, in ways that don't coordinate well with others, actually gets more people dead. Following a policy may "feel" less "ethical" or "heroic", but it actually does save the most lives over the long run.
What makes you think this situation is probable? How often does it currently occur for human drivers? Ever? I don't have statistics at hand, but what exactly is the incidence of pedestrians who are not in the roadway getting hit by cars? And how many of those were caused by driver error that wouldn't exist for a self-driving car rather than a forced choice?
There are a lot of comments below which are variations on the theme of "this is a stupid scenario; just apply the brakes". These people have a terribly naive understanding of the reality in which cars operate.
The truth is that self-driving cars are part of a dynamic, open-ended system which includes roads of arbitrary quality, and pedestrians and cyclists who will be never be under automated control. Such a system is fundamentally unsafe by design. Its operation relies on probabilistic assumptions of safety rather than secondary safety systems, as you would have in a system which is designed to be safe (eg., airplanes and airports). These probabilistic assumptions include assuming that a tire will not blow out, or that a large pothole will not exist on the other side of a hump in a country road, or that a pedestrian will not trip laterally into the path of a high-speed vehicle. When such events do occur, property damage and/or loss of life is a certainty. This is a low-probability event, but one to which we have so utterly habituated that many people -- such as many of the commenters on this article -- are no longer consciously aware that it is even possible.
Nonetheless, it certainly is possible, and happens all the time. Currently, cars kill more than 2 million people per year, worldwide. Many of those fatalities are due to drunkenness or glitches in human attentiveness -- problems which a competently-designed self-driving car will not suffer from -- but many of these fatalities are due to scenarios such as I mention above. Those will continue. Unless we radically change the entire system -- that is, change the way way we build and maintain roads, and the way we integrate or segregate cars from other users of the road -- then this system will remain unsafe by design. Even if the world converts entirely to self-driving cars which operate perfectly, they will still, with an absolute certainty, kill hundreds of thousands of people per year.
Seriously, y'all need to stop being in denial about this.
Hundreds of thousands of fatalities per year is still a tremendous improvement over millions of fatalities per year. It's an unalloyed good and we should doubtless do it. Nonetheless, it represents a hell of a liability problem for the manufacturers. When a human being has a blowout at speed, we never question the correctness of their actions in the milliseconds immediately afterwards. Of course they don't have an opportunity to respond in an appropriate fashion. Human brains can't do that. Instead, we assign liability based on conditions before the crash. If the driver was speeding, we say that the consequences of the blowout (which can easily be fatal) are their fault. If they weren't speeding, it's the tire's fault. Liability ends somewhere between the tire and the driver. As long as the car company has not sold faulty tires, they have nothing to worry about.
When the driver is both created by the car company and has the ability to react in the milliseconds following an incident, it's a different matter. Lawsuits will be inevitable, and car companies will do everything they can to minimise their exposure.
The scenario where your self-driving car needs to make a decision between sending you over a cliff, or forcing a schoolbus over a cliff, may seem terribly contrived. But in reality, on a worldwide basis, this general class of scenario happens hundreds of times every day, creating a level of liability that manufactures must take damn seriously. If they are able to choose between lawsuits from your family, or lawsuits from every family of every kid on that bus, they'll choose the former every time.
And this is why your car will be programmed to kill you.
For more details, check out Peter G Neumann's "Computer-Related Risks". Automatic doors crushing people, radiation equipment (for the treatment of cancer) going way out of spec, and many more.
Now, if you are to consider that something jumps out in front of your car (say a deer) which reduces your reaction time, the car should do the same thing a human would do. Slam on the breaks to reduce impact speed.
Edit: The difference between a human slamming on brakes and a computer doing the same is that humans are not perfectly rational. If I see a deer in front of me, I'm probably going to break as hard as I can. But will I take into account that the road is slippery and braking will cause the car to spin and land in front of a lorry, and maybe the correct decision is to not brake and hit the deer? Of course I won't - humans are not quick enough to decide on that. The problem here is, that computers are fast enough - and now we have to decide whatever they should behave in a "dumb" way, like a human would - or whatever they should be making those decisions no human could make, even if they verge on being unethical?