Hacker News Reader

Comment by cowmoo728 | Hacker News Reader

Maybe I'm too optimistic about self driving but it seems that a fail-safe default of "upon encountering an active emergency situation, stop at a reasonable distance away" is good enough and better than what a significant minority of human drivers do.

View on HN

If the dangerous situation is a flash flood or landslide stopping may be entirely the wrong response. The entire problem with these rigid failsafes and the whole idea of ai-driven cars is this idea that a programmer can predict every situation the car will encounter by extrapolating from the climate and road conditions in places like san francisco.

One project I worked on was a safety critical system that almost got sunk by an overly-simplistic fault response. The device would reboot if it detected any sensor readings out of range. One day someone put a bag over it while it was delivering concentrated oxygen from the air and it rebooted because the ambient o2 sensor was reading well below what anyone could reasonably expect were the machine not enclosed. The patient had to be bagged while the machine came back on.

You simply can’t predict every emergency situation and you most certainly shouldn’t assume a simplistic single-response model is enough to get the passenger to safety. Humans are really good at inferring correct behavior socially and it would be much better for an AV to watch what other drivers are doing to figure out how to handle unexpected situations. To the degree an AI-driven system is unable to do this it is completely unsuited for use on the road.

I think many of us aren’t used to working on anything actually critical to human lives.

And I wouldn’t be surprised if quite a few of those who are have never actually fully understood what that means.

I certainly don’t.

But then, there’s a reason I stay away from life-critical systems.

Who are the significant minority?

This is the same claim always given by the A.I boosters. Who sees firefighters fighting a fire and decides to just plow through and make matters worse?

Because by "significant minority" what you are saying is that 2 or 3 out of 10 human drivers do this.

Exactly. The fact that some humans are drunks or psychopaths does not mean that AI mistakes are going to just be accepted. We send people to prison when they make those kinds of mistakes, what do we do when machine learning models kill while functioning entirely normally?

Send a low level engineer to prison, of course.

That's actually not as wild as you think, because that's what happens when a building or bridge falls. The contractor that built the bridge, just like the company that designed it, the agency responsible for inspections and maintenance, etc, is also liable for damages, including prison if it's deemed so by the courts.

Of course, software engineers get treated with kid's gloves, otherwise, everyone just freaks out and cries and cries, because their robotic utopia dreams are meaningless nonsense.

Apparently, smart enough to code, but not smart enough to take responsibility for their actions.

> We send people to prison when they make those kinds of mistakes, what do we do when machine learning models kill while functioning entirely normally?

Assess a massive fine. Money is lifeblood of a corpo.

How do we know we can make the machines smart enough to detect the dangerous situations it is incapable of handling on its own?

It's just a bit too perfect, no?

How do we know we can make humans do it?

What about a dangerous situation like a criminal trying to enter the vehicle where the right response is to avoid them and drive away?

Reply on news.ycombinator.com