How I Almost Destroyed a £50M War Plane and the Normalisation of Deviance (2016)

[1] https://en.wikipedia.org/wiki/Creeping_normality

Like many books in sociology it comes at the problem from multiple perspectives (in different chapters) such that you can't really hold the author to any one clear conclusion.

Big picture, the space shuttle was "unsafe at any speed" from the very beginning. It was estimated that it had a 3% or so chance of blowing up on any flight which turned out to be just about right in hindsight.

One point of the book was that the Space Shuttle had many things wrong with it: they had a committee who's job it was to review all the safety risks and decide which ones they could accept. "Normalization of deviance" was not a bottom-up matter of bad practices, but a formal part of the organization.

If things had gone a little different with the O-Rings then perhaps the "Challenger Disaster" wouldn't have happened, but then it would have been something else. I remember in the early days the thermal tiles were falling off, but after they had done a few flights they convinced themselves it was OK. It was the Columbia disaster that proved that they weren't OK and couldn't ever be OK.

Normalizing deviance is more akin to deciding that things that were obviously out of the norm are now accepted as a new normal without the gradual degradation, at least in my way of thinking.

Or is it a case of market/management pressure pushing the limits that is the driver? With those pushing the limits getting a competitive edge - that draws others into competing in the same way.

It's one thing to have a script check the return code of a test suite. Often times aircraft checklists include things like physically verifying that something is present and moving in the correct manner when commanded to do so. Obviously you can add sensors, but then how do you check that the sensor is working?

At the end of the day, most of the manual checklist items are still for a human to look at or listen to a thing and make a judgement call.

Yes, but two things. Airplane technology advances happen slowly but surely. Small changes can have big consequences when flying. So rightfully any changes are thought about for a long time until they're deployed in practice. And so far airplane safety is miles and miles above any other mode of transportation, so it would appear that this cautious behavior is working.

But even if we had fully automated "turn key" planes checklists do something for the pilots. They force pilots to understand their instrument panels, where the controls are and how the plane largely operates. Things like the start up procedure can vary really radically between planes even with the same types of engine. In an emergency, say an electrical failure, this can be really crucial if certain systems need to be restarted/turned off -- it's best that pilot doesn't spend time during an emergency understanding where everything is. Even if they've flown the same plane for years, they're only human and you forget what you don't use.

The trust comes from understanding the system and associated risk that contribute to conditions where the system is out of normal operating range. Pilots learn that doing simple checks on parts of the system that are likliest to contribute to abnormal operations is life saving in the long run. E.g. may seem simple that control locks werent taken off but it happens and it's why we check the controls prior to take off. It's why we put the engines to fill throttle to make sure oil pressure is as expected.

Anytime there is an accident many pilots will read the reports and add an additional check or reemphasize a current check that they need to continue to use.

As far as automation. It could fix it. But it increases system complexity so there will be this valley where for a while the automated system isn't much better than normal checks, infact might be worse due to over reliance. Also logic in software or hardware can lie to you in ways that mechanical realities of moving control surfaces, checking oil level, checking tire pressure can't be beat (at least with a cheap solution). This is why the FAA for critical components requires multiple systems doing the same thing and concensus algorithms.

Recently I bought a lemon for a motorcycle. The ECU started giving me error codes twenty five minutes into my first ride. Engine started cutting out. I couldn't start it. On the other hand in a Cessna I could lose all electrical power (assuming magnetos are still operating) and still operate the plane just fine, although without radio comms. The Cessna is from 1972, while the motorcycle from 2017. Granted I could've checked wires or inspected more carefully but non the less, the 2017 motorcycle is a more complex system.

Conveniently the word "deviance" has multiple meanings, two of which apply here.

The article uses deviance in the sense of "deviation from the behavioral norm" (where that norm is a external process). I completely agree with your assessment that these processes can become cumbersome and a barrier to doing work (at most big companies the process becomes the point and dominates actually getting work done). Automation can go a long way to improving that, both in reducing cynicism and letting people spend their time on the stuff that actually moves the company forward.

However the manual checks in aviation also serve to find unexpected deviation in the shape or performance of the device. The reason the first officer walks around the aircraft is not for exercise but because she may see something "out of the ordinary" for which no test exists -- something that should be straight looks subtly out of kilter, indicating that, say, a bolt on the wheel has come loose and under load the wheel tilts slightly. This kind if check is harder to find in software which uses many fewer humans intuitions, which is why we have code stylebook and code reviews.

“Some tone after that first miserable try, I did what I should have done to begin with. I went to the library and pulled out a few articles on how flight checklists are made. As great as construction-world checklists seemed to be, they were employed in projects that routinely take months to complete. In surgery, minutes matter. The problem of time seemed as serious limitation. But aviation had this challenge, too, and somehow pilots’ checklists met it.”

...

“It is common to misconceive how checklists function in complex lines of work. They are not comprehensive how-to guides, whether for building a skyscraper or getting a plane out of trouble. They are quick and simple tools aimed to buttress the skills of expert professionals. And by remaining swift and usable and resolutely modest, they are saving thousands of lives.”

> can't it be all automated

No. Some of it certainly could, notably gear related like engine checks as in your car. And yes current systems are not very flexible, the strong bias towards "tried and trusted" does slow down the arrival of fancy electronics.

But checklists also include confirming you have weather information, making sure your surroundings are clear of danger, that your fly plan is defined... This is more about checking the human component and can't be automated.

> how come do pilots trust a system that has these layers of manual security checks with their lives

This is the best we came up with so far. Manual checks also give you a sense of control.

Also, general aviation is a much bigger portion of aviation than scheduled passenger airline service: number of airplanes, pilots, airports, and jobs. Any automation is going to be narrowly applied outside of general aviation, it's just not yet a general purpose solution.

Great articles there.

Running a red light could easily harm others - a driver could swerve to avoid you, crashing in the process; if a driver hits you because you ran a light, that could also cause a chain of accidents; and if a driver killed you because you ran a red light, that could seriously harm their wellbeing.

And the cyclist had a pretty bad day too.

Running a red light is dangerous and can have plenty of collateral damage.

Another example, I tend to pull over the stop line at stop lights, so that cars don't accidentally creep into me. It happens even in designated "cycle boxes", which cars are supposed to stop behind to increase visibility of cyclists.

Sometimes you can get into a situation where you're in more exposure than you originally bargained for as well, when new cars show up or the situation changes, and I'll sometimes abandon the original plan and bail off the road when it's safe. Often breaking some laws when doing so, but I'm going to take responsibility for my own safety when I need to.

What I dislike seeing is cyclists just flying through reds, darting over crosswalks when pedestrians are still crossing, or weaving in out of moving cars so they can feel like an NYC bike messenger for a minute. It's not without consequence to others, you just don't notice that people had to jump out of the way or slow down so they didn't hit you. That makes it bad sportsmanship in my eyes.

At least where I've ridden in the States, cyclists tend to be treated as something between cars and pedestrians. For example, a driver often squeezes by you on a 2-lane road with oncoming traffic. In these circumstances, it's hardly surprising that cyclists make up their own rules. I regularly break the laws for cars on a bike, because I am more maneuverable, and I'm the one who's toast if either of us makes a mistake.

When deviance is normal, you have to adapt.

I thought it was safer if cycling on roads, to ride forward on to the one coming traffic, rather than have traffic come up from behind you.

Clearly you're not a pedestrian in Manhattan.

There is however a good rule I heard was in use somewhere: Cyclists can regard stop signs as yield signs, and red lights as stop signs.