Hacker News Reader

Comment by bjourne | Hacker News Reader

Even if you work in a gc language with a vm and all memory errors are checked, here is the major, MAAJOR, wisdom you should take with you:

The crucial aspect in the failure scenario described by Michael is that the stack overflow did not cause an immediate system failure. In fact, an immediate system failure followed by a reset would have saved lives, because Michael explains that even at 60 Mph, a complete CPU reset would have occurred within just 11 feet of vehicle’s travel.

We have seen this scenario played out a million times. Some system designers believe it is acceptable to keep the system running after (unexpected) errors occur. "Brush it under the rug, keeping going and hope for the best." Never ever do that. Fail fast, fail early. If something unexpected happens the system must immediately stop.

View on HN

When you're talking about a control system "Fail fast, fail early" isn't necessarily an option. In this particular example, while it's arguable that temporary loss of control might be better than the actual outcome, it's still a pretty unacceptable thing to happen.

At the point where you are trading off one failure state for another, you need to think very carefully about which one is truly worse, and sometimes "Fail fast, fail early" is a much much worse choice than the alternative.

When you're talking about a control system, you need to be able to safely recover from a hard crash anyway, because one could happen at any time for purely hardware reasons. Thus, forcing a hard reset in case of inconsistency should be safe.

Experience (which always should trump theories about how things should work) proves that you are wrong. What would have happened in this case if fail fast was used and the hardware had immediately resetted upon the first byte being written outside the stack area? Likely the bug would have been caught and fixed during testing because debugging a randomly resetting cpu is comparatively easy. It's a hundred times better than people being killed because small unexpected errors are allowed to accumulate.

>Likely the bug would have been caught and fixed during testing

Sure, for testing, you should absolutely fail early and loudly, because there are no consequences for doing so. But in the real world, and especially in control systems, "failing" can cause more damage than persisting in a corrupted state.

> small unexpected errors are allowed to accumulate.

I'm not arguing that you should just ignore errors completely, just that the correct response to a particular error must be very carefully considered, and your blanket suggestion to reset immediately is often a terrible idea.

I think a far more appropriate response to a single byte being written outside of the expected area, is to shut down all unnecessary processes, and tell the driver to pull over. This would be far safer than just randomly disabling the controls, however temporarily.

Sending uncontrollable accelerations for hundreds or thousands of feet is safer than a maximum of 11 ft of coasting?

Yes, there are always trade offs to be made. But a corrupted system is utterly unpredictable - if a control port can do something, it eventually will, at the speed of electronics. On the other hand, a shutdown is entirely predictable. We design for it, because every system does eventually fail and shut down. It's a 'normal' mode of operation in this sort of system design. That tends to weight the design heavily towards very fast resets and redundancy in mission critical (people die if you mess up) systems. I can design a system to handle a shut down control board (limit the robots arm's travel, etc). I often can't do anything if the arms are waving about randomly, under power.

I'm not opining randomly; I've worked in flight control, robotics, UAVs, and factory machinery. Until my current job, I've always had to worry about killing somebody. You really can't let software that controls dangerous equipment continue to run in a damaged state unless the rest of the system has a supervisor mode that can override and limit the system's behavior. Even then, I'm having trouble thinking of a scenario where I would prefer to leave that process running vs shutting it down.

What does it even mean to shut down the control systems on a moving car? All stop? Coast? What's the physical behaviour that isn't likely to kill someone?

At most, it means you're coasting with no power assist for steering or brakes -- you'll need some extra braking distance, but that's what the shoulder is for, and it'll take a bit more than the usual effort to turn the wheel, which means that getting onto the shoulder, especially if you're in a middle lane when all this goes wrong, is going to need to be smartly done.

Taken all together, the impression I get is of a problem in driving skill that's roughly as difficult as a blown tire at highway speeds -- possibly a bit easier, actually, considering the deleterious effect a blown tire has on steering control. An alert and competent driver should be able to handle either situation without posing a deadly danger to herself or anyone else.

The same, I think, cannot reasonably be said of uncommanded acceleration. With a blown tire or a failed ECU, all you have to do is use your ordinary driving controls to get off the freeway and bring the vehicle to a safe stop. With a throttle stuck wide open, you are suddenly a race car driver, only you're neither in a race car, nor in a race. You can't bring the vehicle to a stop with the ordinary driving controls, at least two of which -- the accelerator and the brake -- are no longer responding properly or at all, which is itself frightening and disorienting to the driver. In order to bring the vehicle to a stop, the driver must shut off the engine with the key, at which point the problem reduces to our failed-ECU worst case above, just with some more speed to burn off.

But there is no circumstance, in either the normal driving regime or even any other abnormal one, where turning off a moving car is a proper or safe response to any situation, which is why almost no driver has ever given the slightest thought to doing so -- and when your car's speeding up past ninety all of a sudden, and you're not telling it to, is maybe not the best time to be thinking up new ways to interact with your car that you've never thought of before. At a guess, I'd say some people whose Toyotas ran away on them were able to come up with the idea in time, and they mostly survived. Others ran out of time before they thought of it, and those unlucky souls mostly died.

I agree with most of what you've said. But point out that you didn't go with the worst case example, loss of steering / brakes on ice.

Unwanted acceleration on ice would be worse though. I can't imagine, in rush hour conditions, being unable to control my acceleration. At best, you'd rear-end someone, and their car would stop you.

Yes, it is an option. It should fail as fast as possible, that's why you have contingencies in place. For instance, the Space Shuttle's computers checked their results against one another. If a discrepancy was detected, the computer would be immediately voted out. In case all computers were compromised, there was yet another computer with different hardware and software.

The point is that any erroneous result would cause the offending computer to be voted out immediately. And that's what you want, otherwise the system will appear to work fine, while being in an undefined state.

It is an option, but not always the best option.

I think it was one of the Mars landers where they disabled error detection for landing. If the system reset near the ground then the lander would crash before the controller had rebooted, whereas a small error might not have disrupted the landing at all.

Fail fast is good advice, but should be weighed against other strategies to choose the most appropriate one for the circumstance.

Control system doesn't mean steering. The _unsafe_ choice is to continue. You are speaking in generalities when Toyota software was not constructed to carry on in the face of failure, it just pretended it didn't happen. Huge difference.

> a complete CPU reset would have occurred within just 11 feet of vehicle’s travel.

Then it fails again. Resets again. Fails again...

It may be preferable to restart, but in many failure scenarios, if the source of the failure is unknown, it is not a given that you'll be able to restart cleanly again.

I agree that failing fast to an extent is preferable, but that too requires careful testing and consideration.

Except in a modern car the effect would be the vehicle stops accelerating and starts coasting - brakes and steering are mechanically connected, so in the event of no control system the default situation is safe.

The missing bit is generally that if it fails, it needs to set a note that it should not resume normal operations.

If it keeps failing, then the car does not function and it gets towed to a repair shop. It could be failing for completely valid reasons, not just a programming error. Failure is a expected, we need to account for it.

What makes you think it would fail again right after the reset? It hasn't necessarily been put through the same sequence of state changes.

And the second wisdom to take from this is make your control system boot up fast, just a few hundred ms. Maybe you have time to load a few necessary parameters from flash memory but after that everything should be good to go. You don't want to wait 1 minute for windows to boot up.

Reply on news.ycombinator.com