Hacker News Reader

"...they had a committee who's job it was to review all the safety risks and decide which ones they could accept."

Having a committee dedicated to assessing risks seems like a good idea. Having a group consciously deciding between risks you can live with, risks you have to live with, risks you can mitigate, and risks that stop everything is a fundamentally good idea.

The problem is taking risks without thinking about them, or taking risks without knowing exactly why they haven't killed you before. And even more unfortunately, this is all on an experimental project That Cannot Ever Be Allowed To Fail, Ever™.

Do committees who do such reviews equally ask themselves - if this does fail - can we live with it and justify our initial risk assessment and stand by it?

So often - risky decisions are made - many pass without incidence and yet, problems happen. Often such risks are reviewed in hindsight and the weight of the realisation of risk that pass into actuality. Though when they don't fail, nobody passes a second thought and over time, such risks get normalised.

That's a fair point, but the fundamental fact is that at some point, someone/something/some process needs to accept a level of risk, or perform some type of cost/benefit analysis. Sometimes the analysis is straight forward or easy to criticize. For example, when the costs are high, and benefits relatively low. It's super easy to criticize the shuttle disasters, because the cost of cancelling the launches are so obviously lower than the lives of astronauts (at least in popular discourse).

The issue is that all mechanisms to do so are susceptible to some variant of the problem you highlighted. As so highlighted in the OP's submission, individuals can make the exact same mistakes.

I feel that a lot of things comes from the fact that we are constantly encouraged to over-quantify risk. That is we take things where REALLY we don't have a good grasp of the real numbers, and then make something up. If we're good, they'll even be in the right ballpark. If we happen to choose to follow the course of action, and everything is working "just fine", we go back to our mental risk model, which we always knew was kinda made up, and then say "well, I guess I guessed wrong", and assign a lower risk.

Yes, it does get down to risk factors and the parallels with the reinsurance/insurance market with actuaries, is an aspect not lost upon me. As we know you can't put a price on a human life - but they do in the end.

When I've been involved in such things, the phrase "Dear Mister Coroner" has come up regularly.

A agree that having such a comittee is essential. But (at least in hindsight) the risk assessment for the space shuttle seems very strange. The only rocket ever built that requires human pilots is also the least safe human-carrying rocket ever built, both in track record and design (no launch escape system, no abort possible for large parts of the ascend).

In a comittee maybe you can justify the human deaths as nessesary risk, after all people die in traffic all the time. But that ignores the political dimension. The Challenger Disaster was a giant blow to the entire shuttle program: The space shuttle was only funded because of support by the military. With the entire fleet grounded for 2 years and much needed performance improvements canceled that support vanished (see for example: the last DoD shuttle mission was in '92).

It seems like Apollo had a worse track record even if you consider 13 to have been a successful mission rather than an incredibly near miss.

That's a really good summary / perspective. The more I read the Challenger Launch Decision, the more I was convinced that mistakes like the Challenger are unavoidable. We can try to reduce the risk, but given that we're always operating with limited resources, we just can't eliminate all the risks (from the design and implementation as well as the decision making and operation).

But "normalization of deviance" is also partially how the science / the engineering works. The science and engineering is pushing the boundary and turning what used to be unknown into known and tested through experiment / experience. Learning from experience is fundamentally how the science and the engineering work. So it's really difficult to tell when it's a normalization of deviance and when it's a normal scientific/engineering process.

Things could have been done differently with the space shuttle.

Note that the crew of the Challenger wasn't killed by the "explosion", they were killed when the vehicle compartment they were in hit the water.

If the shuttle had an escape mechanism the way every other manned space vehicle had, they might have survived. Had they built in an escape mechanism, there wouldn't have been any room for cargo. From a safety perspective launching cargo + passengers on the same rocket was a mistake.

Also the shuttle was designed to do a military mission that it never actually did, which would be launching, orbiting the Earth once, and then landing at an airstrip close to the launch point. That requires a large "cross-range" (ability to turn when reentering the atmosphere) which in turn requires larger wings which are more vulnerable to something like what happened to Colombia.

One of the reasons I really like Challenger Launch Decision is because it does a very good job of avoiding the hindsight bias as much as possible, by getting the readers to understand the perspective and the cultural norm of the organizations and participants, and tracing the decision making process itself and influences on those, instead of technical decisions themselves that were / might have been wrong in hindsight.

In hindsight, we can see mistakes in terms of specific design and technical decisions. But I don't think those are the biggest lessons from the disaster. We will continue making such mistakes, since technical misjudgement itself will always happen. I think Challenger Launch Decision's main point, that the organizational structure and culture have large impact on individual decision making is the lasting lesson. The book's closing sentence is excellent:

"The lingering uncertainty in the social control of risky technology is how to control the institutional forces that generate competition and scarcity and the powerful leaders who, in response, establish goals and allocate resources, using and abusing high-risk technical systems".

> "The lingering uncertainty in the social control of risky technology is how to control the institutional forces that generate competition and scarcity and the powerful leaders who, in response, establish goals and allocate resources, using and abusing high-risk technical systems".

Seems to me that the relationship between Boeing and its various 737 MAX customers could be seen in a very similar light. Abusing high-risk technical systems as a tool to extract resources to cover for artificial scarcity in budgets or market expectations.

You seem to be implying that something near complete safety is practical with enough work. I'm not sure this is the case. A big problem with all endeavors, and not just space, is that things that work (or don't work) in theory somehow often end up doing the exact opposite in practice. This is why things become more safe at scale: you get to see those abnormal conditions play out in reality, and work to prevent such failures in the future. In rocketry we're still babies. We've launched, in total, something like 8000 rockets - hundreds have failed. And we've only launched around 600 humans to space on a small fraction of those 8,000 launches.

Just consider something like a software launch. No matter how much effort you expend trying to make things go perfectly, or how many months of beta testing you engage in - something's going to go wrong. And that's in a nearly completely deterministic environment. Space travel involves countless external nondeterministic factors. Another big difference is that in the software world we get instantaneous scale - up to millions of instantaneous independent trials. Spaceflight advances one costly trial at a time. In the entire history of rocketry, we haven't even begun to approach scale and so all the pretext of safety we could ever elevate is just that - a pretext.

It's for this reason that being open about risk and danger is important. It's also why I think aiming for complete safety can be paradoxically harmful. It creates scenarios where open discussion of danger and risk becomes taboo since it can result in negative consequences. The most critical factor in all of this should be informed consent. Making risk taboo goes directly against that.

Disagree. It's possible to launch software without things going wrong - I've been involved in one. Certainly NASA was not taking the absolute safest approach possible - they had people internally advocating for safer practices (which is not to say that there weren't legitimate other pressures).

what you are describing is not normalization of deviance, but rather proper risk management. that they made an incorrect assessment is a different issue.

normalization of deviance is when a process isn’t followed. when the reality of the process is different than the understood, risk-managed, process.

People writing about normalization of deviance in healthcare and aviation describe something very different from what is in Vaughn's book.

In this interview, Vaughn said "Social normalization of deviance means that people within the organization become so much accustomed to a deviant behavior that they don't consider it as deviant, despite the fact that they far exceed their own rules for the elementary safety", which seems to me to be pretty close to how people in healthcare and aviation are using it.

http://www.consultingnewsline.com/Info/Vie%20du%20Conseil/Le...