Hacker News Reader

I always question whether humans are really any better at this calculation. In the gnome example, many people will probably stop to avoid damaging their property, not see the car behind them careening forward, and end up getting rear ended anyway. In an ideal world, ethical issues like the one you presented would be major hurdles but humans aren't going to be perfectly rational in the moment of many accidents, especially the most dangerous high-speed ones when there is physically not enough time to do ethical math.

> we all still get in cars believing we'll be better than other drivers. Who's going to get in a car when any given car has an equivalent chance of making a bad decision all on its own?

Given the fact that plenty of people take taxis, rely on Uber and Lyft drivers they've never met, and take public transportation like aircraft and trains with little knowledge of the operator, this seems like a non-issue to me.

The other thing to bear in mind: I would guess that the vast majority of accidents are caused by people not paying attention -- distracted by a phone, person, great view, or even just because they're looking the wrong direction. Cars don't do any of those things. They will always be paying attention, looking in all directions at once.

So if they eliminate all of the accidents that are caused that way, but increase the number of garden-gnome related accidents...wouldn't that be worthwhile?

Another, less flippant way to say it: yes, we can imagine scenarios in which the number of accidents will increase. But what is the relative frequency of those types of accidents compared to run-of-the-mill rear-end, red-light run, lane change due to inattentiveness?

And even if the number of accidents increases: Does that increase cause greater monetary loss than the considerable cost savings of not paying drivers.

If given a choice between changing from what you are used to which has drawbacks, to something new that has drawbacks, even if those drawbacks are markedly milder, almost everyone will choose to stick with where they already are. Consciously choosing a position which has drawbacks that you know about makes you feel personally responsible for those when they occur. That is infinitely worse than suffering the drawbacks of the "default" you stay with which you can give yourself a moral pass on because it wasn't a situation you chose to begin with.

There's also the issue of what a self-driving car could do... and what they WILL do. Much technology COULD be amazing... if companies did not function in the ways that they actually do. Because of that, technology often fails to meet any of the promises made. Development is supposed to be cheap and fast, and safety and security get left at the door. No company is held liable for such things, and it would be expensive and bothersome to fix, so they're not about to bother.

There's no ethical question here. You either see that it's a gnome, or you see that it's a child.

Humans may or may not be less situationally aware overall, but we have the gnome question down cold. Cars don't. They see "obstacle in front of bumper radar."

The OPs point is well-taken because it illustrates a technical risk that few are discussing: the "last 1%" of self-driving software may be equivalent to solving AGI.

>the "last 1%" of self-driving software may be equivalent to solving AGI.

Exactly right. You need a semantic world model, reasoning and prediction. AGI bottlenecks self-driving, assistants, text understanding…

I think we are overestimating amount of intelligence we humans need while driving...

The edge cases are unclear: construction zone, heavy equipment, school kids, another car poised to enter the road.

They are probabilistically modeled already. You keep tracking other objects and compute various possibilities for their movement and depending on the past and present behavior pick the most probable one. If car encounters lower-than-threshold probability, it can always slow down or stop, like what non-reckless humans would do. Even end-to-end deep learning can recognize construction zones and learn to drive around traffic cones.

The edge cases are endless (wheelchair woman chasing a chicken), and without a semantic world model the car will have no idea what to do. Low-level object avoidance architecture isn't enough.

Actually that case would be detected as two objects on a collision course and car would stop/slow down. Mind that sensors scan surroundings with high frequency and some of them can see "beyond the walls" like radar. Ultrasonic sensors can also detect the state of road surface (dry, wet, icy etc.).

Yes, but when can the car proceed, and in which direction? Helps to know what a wheelchair, a woman and a chicken are and how they behave. Helps to be able to speak with them.

Superhuman sensors will be great, but you really can't dodge AGI for true full self-driving.

It's probably super easy for the car to stop and open the window for a person to talk ;-)

True, you could have the passenger making "shoo" noises :D

I do think the "last 1%" may be a very hard problem to solve, but the majority of crashes are simply people not seeing things or making bad decisions. "Recognition Error" and "Decision Error" are the two most frequent categories of accidents which basically boil down to not seeing obstacles while performing a maneuver (changing lanes and not seeing a vehicle in that lane, turning and failing to see an oncoming vehicle) and driving too fast respectively. ( see https://crashstats.nhtsa.dot.gov/Api/Public/ViewPublication/...)

I don't see why the "last 1%" problem needs to be solved when we have hundreds of thousands of accidents every year caused by people not paying attention, not reacting in a timely fashion, and driving unsafely; all problems that computers are likely to excel at.

Because a half-baked product is dangerous (re: handover to human) and unsatisfying. You really want the version with no steering wheel. I’ll take the intermediate one, but let’s work toward the good one.

Are humans any better at flying? Doesn't seem to impact how they feel about getting into an airplane.

None of these contrived scenarios compares to the rate at which humans simply fail to stop when they should have stopped. I don't know anyone who got into or was rescued from an accident by pulling one of these maneuvers. I know plenty of people who got into accidents because someone (them or someone else) failed to stop when they should have stopped.

Considering humans can barely stop when they should, I don't give them a high likelihood to also be able to have the situational awareness and make the multiple split second decisions required to both decide to, and take action upon, directing their car towards the least-valuable obstacle in an emergency. By the time they decide what to do they will probably be rear ended.

Edit: your examples also ignores that with self driving cars, people won't be put in these situations in the first place. A driver not paying attention to the road, looking at their phone would be... not a safety hazard.

As long as other cars (or humans, or animals...) are on the road, robot cars will face these scenarios.

The OP's scenario is contrived to make a clear example, but it happens regularly when you're driving: you're forced to make a choice between acting and possibly hitting something, or doing nothing and being hit.

We don't need self driving cars to solve the 80% case of preventing human drivers from hitting things. We need radar and auto-braking, which already exists. Full autonomy requires solving the harder problems.

There are 35 thousand deaths per year as a result of car crashes, and over 6 million total crashes[0]. If autonomous cars reduce those deaths to hundreds and overall crashes to thousands, I think you're underestimating the number of people that will sign up for that.

[0]https://crashstats.nhtsa.dot.gov/Api/Public/ViewPublication/...

You're thinking about the tail end of the distribution. If self driving cars can solve 90% of the mass of the distribution, i.e. driving on highways and urban streets, it would be a significant improvement in terms of efficiency. Of course there will always be morale questions like the one you proposed. But quite frankly, the return on investment of solving the tail end scenarios like that is really small.

You can't ship without restricting where the car goes, punting edge cases to remote drivers, or actually solving the tail end with AGI. Actual full self-driving is far more satisfying than "you can go from here to the airport" or "you can drive in downtown SF."

But actually shipping a product that can operate autonomously in a subset of areas and conditions is more satisfying than not shipping a product at all.

True, I’m a Model 3 fan :)

The danger in this contrived scenario is still the human driver, who is distracted and careening down the road.

If all the cars were autonomous, it would more likely to be a matter of frustration than safety: the gnome causes a traffic jam because the cars won't endanger the passenger by driving into the opposing lane. Then what? They find an alternate route? They automatically call in a service crew to clear the road? Not the end of the world.

MIT (and I assume others) have collected data on these types of scenarios. For example, swerve and kill a bunch of people, or stay the course and kill a mom and baby?

http://moralmachine.mit.edu/

Hopefully in a not too far future you and all the others that are convinced that are better drivers will be forcefully signed up so your scenario can't possibly occur given that the other car will be self driving and always paying attention.

What if the jackass on the phone was in a self driving car that just stopped?

There is nothing hard about valuing gnome vs child, if you can identify them. Computers can evaluate huge problem spaces in milliseconds. The hard part is the perception: identifying the child and the gnome. Deep learning has allowed us to make gigantic leaps in the quality of perception but there is a long way to go in uncontrolled environments.

I also think people incorrectly think that the moral choice dilemmas that self-driving cars will face are somehow the difficult part. What will happen is that we will provide valuation functions that are somewhat close to our intuitive collective understanding of value of life in different forms. Computers use them to evaluate these situations and we as a society just adapt to the new ethics of the AI age.

I will get excited about a self-driving car on the day that an auto manufacturer fully opens up details about their development process - size of team, experience levels of team, breakdown of salaries, tools they used, methodologies used, etc. I also want a PERSONAL statement from every single engineer who worked on the project saying that they take personal responsibility for the safety of the vehicle and that every single one of their own concerns about safety or foreshortened schedules or testing were addressed to their full satisfaction.

I also want to see the CEO, CTO, and other C-level execs of the auto company to release a legally binding agreement holding themselves personally liable for criminal negligence if the vehicle is later found to have obvious flaws - obvious determined by an independent group of several thousand unaffiliated volunteer experienced software engineers.

Right now, if a car company slapped a 'self driving' car together that was designed to seek and destroy pre-schoolers, no one would say boo. Certainly not the courts. Toyota killed people by having ludicrously egregiously bad development practices (their developers did not even have access to a bug tracker... they followed 6 of 94 industry standard recommended practices... and worse) - and the courts acquitted them of negligence. That is disgusting, and makes it impossible for any reasonable person to trust any auto company doing development of something as significant and difficult as making a self-driving car.

They know that they will not be held liable no matter how bad of a job they do - would you drive across a bridge built by bridge engineers hired, managed, and made to work the way software engineers are hired, managed, and made to work in modern companies?