Hacker News Reader

Comment by gambiting | Hacker News Reader

I feel like this exact same problem is going to be encountered when automatic cars are introduced. Automatic systems will be good enough to take care of 98% of situations, but in the 2% when the system is just not good enough, the humans will not fare much better, if at all. Good drivers will still exist, average drivers will remain average, while people who should never be in command of a moving vehicle will now be "driving".Only very few of them will be able to cope with truly critical situations. Of course all of that will be dismissed on the basis of automatic cars saving innumerable human lives, but I believe we will observe the exact same phenomenon.

View on HN

In some ways one can draw parallels with this and the increasingly complex multilayered abstractions of software - it decreases effort so people can throw together a bunch of libraries and quickly create some app that does something useful, but at the same time decreases motivation to understand the details of what they're building or how things actually work, making it much harder for them to troubleshoot when things go wrong.

I can see the day when teenagers can't drive a non-automated car, just like most people today have a hard time driving a vehicle with a standard transmission. It wasn't that long ago that everyone had to know how to drive a standard.

In EU if you passed your test in a vehicle with an automatic transmission you can't drive a manual, unless you re-take the test in one. So that kind of already happened.

My point is that if we let teenagers drive fully automated cars,they might find themselves in a truly critical situation they will not be able to get out of, due to their lack of experience and training. People will then say - if they were experienced drivers, and if not for all this automation, they would be able to get out of this situation. Because that's exactly what this article is saying - if the aircraft wasn't as automated, and the pilots had a lot more experience actually flying, they would have realised that the plane was, in fact, stalling, and maybe would have saved it. That does not change the fact that automation, overall, saves more lives than it takes.

The difference is that an automatic car that gets in over its head can just stop, presumably pulling off the road first if it's able. In this instance, instead of falling back to various alternate laws and finally crashing into the ocean, the car analog of AF447 could just pause and let the pilots figure things out at leisure.

Well, it's not too hard to imagine a situation when the computer loses control over the breaks, engine and the transmission - so it quickly realizes that in its own state it cannot bring the car to a halt. In that instance an automatic car would do what that airplane did - assume that it's own instruments are unreliable(after all, maybe the breaks work when you push the pedal, it's just a sensor that's broken), and give the control back to the driver. And then the driver would have very little time to react - as soon as the autopilot disengages the driver would have to avoid hitting other traffic, buildings or falling off a cliff. And then we arrive in the same situation as those pilots were in - an experienced driver, who had many hours of actual behind-the-wheel experience could possibly save the car and himself from an accident, avoiding obstacles long enough for the car to stop. An unexperienced driver, who has only ever "driven" automatic vehicles would lack the wisdom to avoid crashing, would most likely panic and let the car hit something. To me, the comparison is almost perfect.

A broken sensor wouldn't completely disable the brakes. It would make them less precise, but the computer could still apply braking force. You'd have multiply-redundant systems to ensure that they were always available. It would take way more than just a couple of sensor failures as happened here, and that kind of massive failure only appears in really dire circumstances coupled with bad design, e.g. that DC-10 that lost all hydraulic systems when the engine exploded, because they managed to route them all past the same point.

Yes, but the computer only knows as much as its sensors tell it. If according to the sensors the breaks are not working, it doesn't matter that they defacto work - the only logical conclusion that the computer can take at that point is assume it's unable to apply breaks and relinquish command to the driver. And even multiple sensors can become broken - like here, the plane had 3 sensor pipes and they all became clogged with ice. The autopilot assumed that the values are invalid, therefore it couldn't continue functioning - and gave control back to the pilots.

Also, don't forget that you can't have redundant systems for everything, unless you want to be driving a tank. Imagine driving an automatic car and then some bird poo falls on the laser-sensor and the car literally can't see anymore(I am exaggerating, but the car can surely be blinded by something, the laser sensor on top can become dirty or damaged). The best it could do is apply full breaking force,but if you are on a motorway and there is an 18-wheeler behind you it could be a fatal idea. Again - the autopilot can't continue, it has to give control back to the driver - and the driver might crash the car if they are not experienced enough.

I actually think autopilot for an airplane is much, much safer than in a car. Airliners fly at several tens of thousands of feet in altitude and have glide ratios that are rather impressive, between 10:1 and 30:1 meaning that at cruise (where airliners spend most of their time on autopilot) the flight crew has several tens of minutes to figure out what's going wrong. Even if the airplane loses power completely!

Further, these airplanes are kept fairly widely separated in altitude and the skies are rather sparse, so collisions are something to be avoided but it's not all that hard.

In a car though, the safety margins are much, much lower. Because there are vehicles everywhere the time to collision is probably measured in milliseconds rather than minutes. Because there are no redundant systems to ensure safety once the car "realizes" it can't control itself anymore you're prettymuch hosed. Are any of the regulations on self-driving cars taking this into account and at least mandating that the control system has sufficient battery back-up for it to at least TRY and gracefully de-energize the vehicle?

Once cars become self-driving you can't really rely on humans to suddenly jump in and take the wheel and save the day. They're going to be asleep or not paying attention or watching a movie or whatever. And if they are going to be required to watch the road paying 100% attention to what they car is doing, what's the advantage of having the car drive itself? You're in a constant battle of "what's the car going to do?!" which is actually more tiring than just driving it yourself.

An easy solution might require that self-driving cars only self-drive in lanes which are adjacent to a shoulder of sufficient width giving the car an out that requires only minimal controls, so basically far-right or far-left lanes on a freeway.

A self-driving car might also need to have multiple, redundant regenerative braking systems so that it can power itself enough to power the controls to safely guide the car to a stop. But then it also needs multiple control systems so that if one of those fails, the car isn't out of control.

The economics of having an autopilot on an airliner (which costs many millions of dollars) and a car (95% of which cost less than $100k) are really, really different.

I think it's clear enough that you're right just from experience. We've had autopilots of various kinds of airplanes for decades, and they can fly pretty much autonomously at this point.

But in this one specific area, of how to handle failures and returning control to the human, I think cars have the advantage. It just doesn't compensate for all the other places where a car autopilot is vastly more difficult.

So my gut reaction to "cars have the advantage" is "you're clearly wrong!" but that's not really polite or enlightening. Rather than argue about it (which I'm sure we could do endlessly) I'd like to know why or how you think that cars have the advantage. I saw earlier that you posted "because you can always have the car pull over to sort a problem out" and I would agree that provided the car can still control itself, yes it could do that.

I guess my assumption on when the car tells a human being to take control are in a few circumstances:

1. it loses the ability to control a driving input that ostensibly the driver still has the ability to control (steering servo fails)

2. it loses the ability to control a driving input that the driver also can't control (tire blowout)

3. it doesn't know what to do or can't make a decision so the driver is the tiebreaker (crash imminent, road disappears, brand new road with no map data, etc)

4. it loses a sensor input completely or starts getting data that it considers garbage and thus can't safely operate the vehicle (LIDAR or camera fails)

5. loss of power so that autopilot just straight up fails

I could probably think of some more scenarios but the point isn't to be exhaustive but illustrative.

So in some of these circumstances the car does have an opportunity to pull over and wake the driver up and say "hey you need to drive now" but in others, it's going to be split second.

If ALL the car failure modes resulted in the car pulling over and the driver taking back over at his/her leisure then I would 100% agree with you. But there clearly are failure modes where the driver is going to get put back in the control loop with little/no warning and be asked to make a decision perhaps faster than they can wake up and/or process the scene and catch up enough to decide well.

So basically what you have to do is make some kind of guarantee about "the car will never ask the driver to take over with less than 60 seconds of notice" or something like that. But the hardware cost to enable you to make that kind of guarantee is substantial; sensors, battery backups, autopilot systems, actuators, etc all in triplicate. And I don't know that you can do with just two because in the situation where you go from two to one you don't have any kind of ability to determine if one of the pieces of software is malfunctioning due to memory corruption.

Although the price of doing all this is going down drastically I suspect that it'll be a while before people are willing to pay enough to get the kind of redundancy they have in airplanes.

So what am I missing? I'm really curious.

A sudden catastrophic failure means you're probably screwed, indeed. But for cases like AF447 where it's small sensor failures, it seems to me that the car should be a much easier environment to work with, because you can always write some code that goes, "if important sensor X is wonky, pull over and stop".

For example, when the airspeed data was lost, the Airbus had to fall back to an alternate behavior which is part of what confused the pilots. In a car, you could skip that fallback and just go straight to "pull over and stop," but the airplane has to keep trying somehow. Similarly, the angle of attack went beyond the range the designers had anticipated, and the computer assumed that it was a bad reading. But it kept going, because it had to. In a car, if there's a bad reading, it can again just go straight to "pull over and stop".

Big failures will always mean big problems, but what's striking about AF447 is that it was such small failures. And when it comes to small failures, I think a car is in a much better position to deal with it because it can just cut the gordian knot.

Yeah when you put it that way, your position makes a lot of sense. Thanks for sharing.

I think the disconnect is that I see that sensor as hugely important. If you were in a car and it lost the ability to sense speed via the speedometer, how would it pull over and stop? How would it know how quickly to steer towards the shoulder? How would it know when it had stopped?

When airline autopilots were developed you didn't have a dozen ways to sense speed. With a car you have the speedometer, GPS, probably some kind of estimates from LIDAR and vision, and the ability to integrate output from the accelerometers and feed all those into a kalman filter to make a really good model for the car's speed even if the more accurate methods fail. Certainly there are good odds it could perform well enough to move the car to the shoulder.

I guess I am less worried about those kinds of more minor failures and more worried about the kinds where humans are suddenly thrust back into the loop. On airplanes the pilots nearly always have a way to control the plane and plenty of time to react since they've got 5-30 minutes worth of glide.

In a car when something goes wrong you might well have vehicles on either side, in front and behind if it's heavy traffic. And in that case if the car finds itself asking for a human driver very suddenly who has to start making very good decisions faster than a human is capable of.

In my mind the "you can always pull over and stop" only works if you're assuming that the car is driving on an immaculately paved stretch of nearly or completely empty road.

Reply on news.ycombinator.com