The Cult of the Root Cause

It's also worth pointing out that you may not be able to fix the root cause. The sailing example is great here:

- Can you fix the crack in your hull whilst you're out at sea? Almost certainly not.

- Can you even tell you have a crack in the hull until you've reduced the water in the bilges by running the pumps for a time? Again, quite possibly not.

Treating the symptom is really the only sensible option, unless it's serious enough that you need to put out a Mayday. Again, not a course of action that addresses the root cause but, in some situations, absolutely the right thing to do. To take things up many, many notches, the sinking of the Titanic was an appalling tragedy from which relatively few people were saved, but I guarantee that nobody at all would have been saved if the people on the ship had opted for a series of committee meetings about how to solve the problem of the iceberg. (Not to say there weren't a very large number of hideous blunders in the management of that situation going all the way back to the ship's design and fit-out.)

Moreover, another problem with Five Whys is that, applied heedlessly, it's an extraordinarily arrogant philosophy, because it makes the assumption that you can know the answer to those five whys. Often you can't, at least not without going on a journey, and fixing a few things along the way, and without that, you can simply be wasting your time trying to answer questions that you can't answer in your present situation.

And that really, and somewhat poignantly, cuts to the root cause of why I view the kind of thinking frameworks/fads popular in business with a degree of scepticism: over-applied or misapplied, they paralyse people into inaction, and thereby provide fertile stock for breeding mediocrity.

This article is creating a straw man. You don't do a 5 Whys and fix what you think the root cause was, you fix the issues that were most serious. If there are problems too big to tackle immediately, put in short term mitigations and incorporate your new understanding of the system's reliability into your future plans.

If there is a cult of the root cause, I have yet to meet it.

Here's what doing an exercise similar to 5 Why's gets you:

- An understanding of where issues come from. Whether your plan of action starts from the top, bottom, or middle, taking the time to step back and broaden your perspective before you jump in to fix a problem helps to make sure you're going after the right things for the right reasons.

- A culture of _not_ just picking the most expedient and facile solution every time issues come up, and going with that. In companies I've been in, the pressure is almost always on to find the dumbest, hackiest, absolutely fastest path out of trouble. Spend multiple years solving every problem that way, and you are in deep trouble! It takes institutional courage to push back against that, and having a practice in place to force you to stop and think now and again gives you an opportunity to summon that courage.

- A culture of ownership. This seems a little counterintuitive to me, since if you follow root causes deep enough you're liable to stumble onto people and process problems that are way out of your control and pay grade. Looking at root causes this way, you might think it's a process of passing the buck. However, by shining a light on such things, and finding people to address those things where they have no owner, you can push towards a better collective ownership of the real issues that face your company.

No good management idea is free from abuse, of course. You must exercise taste and judgment in deciding how deep to push with root causes, and what to do with the discoveries. I would think it's rather self-evident that 5 Why's doesn't mean you always ask exactly five questions in a strictly linear pattern. But for heaven's sake, make sure you ask more than one!

I've never taken the 5 Whys literally. It's obvious to me that all root causes are not '5 Deep', therefore, this can't be a literal objective. I see it as a metaphor for being an effective problem solver, as a reminder to second guess the cause I've identified and ask myself or my team, "Is there a deeper, underlying cause?".

The most interesting cases are complex systems where a fault/even results from a combination of multiple factors. In that case, asking for a "root" cause is unproductive, and a better question to ask would be: how might the system be patched, with the least pernicious side effects. This is also why I don't always like the drive for more "accountability". As can be seen in the other HN thread on the front page about data driven medicine and the side effects on the healthcare system, you need to be very careful about which of the causes you decide to intervene on.

When you closely manage something to reduce variation, you also lose any information you might get about the system from the variation of that quantity. This point is nicely made in another post on the same blog: http://reinertsenassociates.com/the-dark-side-of-robustness/

Especially with reflexive systems (involving humans), the appropriate response might sometimes involve performing no intervention, or performing an intervention downstream, to modify its assumptions about what it receives (eg: adding error handling).

My ops postmortem template tried to elicit breadth. Once you’ve got a forest of causes, you can apply Five Whys to add depth.

There’s some overlap among the following questions. The intent is to elicit observations and ideas, not to uniquely categorize them.

* What are all the factors that could have prevented the incident?

* What are all the factors that could have detected the issue before production?

* What are all the factors that could have detected the issue sooner when it did occur?

* What are all the factors that could have accelerated mitigation? (Including, especially, changes that could have reduced the risks of mitigations considered too risky to apply.)

* What are all the factors that could have accelerated remediation?

* What could have reduced the scope or impact?

It’s common to come out of this with a laundry list that overfits the last incident and, if applied, would increase the complexity of the system and add risk. We’d typically apply one or two fixes, and stockpile the rest to see if any of them would have addressed any future incident. Usually most of the “solutions” turn out to be specific to the single incident that prompted them.

My favorite, goto, on this one: https://blog.acolyer.org/2016/02/10/how-complex-systems-fail...

The truth is that by “fixing” the root cause you will sometimes destabilize the complex system you are running.

> Second, it assumes that the best location to intervene in this chain of causality is at its source: the root cause.

It doesn't though, that's just a built-in assumption for the lazy. The point of Root Cause Analysis and the "5 Whys" isn't necessarily to get to the root and fix the root...it's to provide a framework for traversing a problem set. The point of this methodology is so that you traverse the problem, understanding each step along the way...not that you simply jump to the root and try to fix it blindly.

Most of these examples don't seem relevant to RCA at all.

> shifted from pumping, to plugging, to hull repair

Pumping and plugging are immediate response, just like a decision to temporarily shut down the website when a compromise is discovered, or pulling the plug on a smoking computer. What do these have to do with root cause analysis?

I totally appreciate OPs insights here. It can be so tempting to see things linearly. But obviously the real world is anything but. There's a curiously wicked theory out there by David Abrams that part of this seeming human predilection for linearity has occured somehow through neural conditioning involved with adopting systems of writing, compared to the crazy fractal ocean of sensory input that living in a real living landscape begets. The writing has a start. A finish. It progresses in a single visual direction, and what's more, the pieces themselves only represent reality. The letters themselves are completely different from the things they are describing. And they might be one of the first objects in human history that work like this...

Anyway, the book is called the Spell of the Sensuous, and it's dense af, but bursting with fascinating lines of inquiry.

People are notoriously bad at playing five-whys. If you haven't reached an ideological or metaphysical problem by your third 'why' then you aren't playing well enough.

My experience with Root Cause Analysis is it's often an exercise for poor managers to deflect accountability by diluting it among more junior people.

However, 5-whys is very useful as a design principle instead of using it to respond to failure.

It goes something like:

0: build a thing why 1: because customer asked why 2: because thing is what they think they need why 3: because it is one solution to a gap in their ability to achieve something. why 4: because that something is an economic need. why 5: because market opportunity to fill that gap with something, maybe this thing, maybe something better.

The biggest problem with the 5 whys is that the whole concept is taken out of context. In a manufacturing context every repeatable problem is a problem with your process. So in that context it makes a lot of sense to search for a process change that resolves your problem.

Let's say you make light bulbs, and every fifth bulb comes out misshapen. You would use the 5 whys to trace it back to the molding station, where you discover that bulbs cool at a different rate in one of the machines because the mold uses a better insulator. You could stop there and replace the insulator. But if your job is to increase yield, then you can save the company a lot of future money if you figure out how that mold got there in the first place. You might find that purchasing subbed a cheaper replacement based on an incomplete spec. Or you might find that the supplier recently switched materials.

The point is that when you're trying to establish a controlled, repeatable process, you need to understand where your controls break down.

Once you understand the process problem, then you make a business decision about what to change. It was never meant to be applied to R&D problems. R&D processes are not as concerned with repeatedly doing something correctly. They're concerned with making sure something can be done in the first place. It's a different class of problem.

This reminds me of what we've been doing for a while, the "blameless postmortem." The technique is championed by Etsy, and you can read more about it in an article that introduces their debriefing guide:

https://codeascraft.com/2016/11/17/debriefing-facilitation-g...

From the linked PDF from the article:

> “Adaptability and learning. We learn through honest, blameless reflection on lessons and surprises. We believe that traditional root cause analysis makes learning from mistakes difficult. Our blameless post-mortem process is a widely-cited technique that we believe is becoming best practice among organizations that value innovation. Blameless postmortems drive a significant percentage of our development as we analyze what about our production environment was less than optimal and rapidly make corresponding adjustments.” (Etsy, Inc., 2015)

The idea, boiled down, is to inspect timelines, procedures, and actions and develop a narrative of how an incident came to be. The goal is for everyone to walk away with a (better) understanding of everything. With this, people are better armed to put into place solutions.

One example from the text is where an engineer pushes out a change because they thought the build system had zero failures. The push breaks the system and causes a regression that should have been caught in the tests. During the postmortem, the engineer says, "I thought the tests had zero failures. I guess I need to be more diligent in the future." Upon further timeline investigation, it is noted that the tests actually had eight failures, but the font had eights and zeros looking very similar. The fix was not "be more diligent;" the fix is maybe to have a better font or use colors for pass/fail.

Overall, I like the ideas proposed in this blameless postmorem style. It runs counter to the natural tendency to "find a problem and fix it" because it feels like we are talking less about the problem and the fix and talking more about the narrative of the failure. But what I've seen is folks gaining better understanding of how everyone else works, learning about tools and tricks, and about assumptions. And knowing more about the narrative leads to better solutions.

Shouldn't it be the cult of the single root cause? Most problems seem to have several contributing factors. You can almost randomly pick one factor, improve it, and the whole situation will get better.

You see this a lot in public debate like education or health care. Instead of fixing one of the many problems a lot of time and energy is wasted on finding THE root cause that will fix everything.

I wouldn't take such an advice. For me getting to the root cause of things is one of the core values of a good programmer (and for operations as well).

Cause #2 is also fictitious, the 5 whys never say anything about fixing the problem, only understanding it. In fact, for me, not fixing the problem is just as valid a solution - as long as you know what caused it, you can determine if it's worth fixing it at the root or even at all.

As to the linearity of cause and effect, while it's true that many problems have multiple causes, a solution to a linear problem will prevent alternative causes below it. Besides, the grand majority of issues arising in mature systems arise from a single cause and have linear cause and effect.

Reminded me of this entertaining clip:

http://vooza.com/videos/the-5-whys/

I doubt that most people who use the five whys really take it to be as simplistic as the author of this article suggested, but to those that do, it's a good wake up call.

In fact, life is even more complex than the article suggests, when you throw in effects of chaos, feedback loops, missing information, unknown influences - just to mention a few. Still, tracking down the order in processes has got humanity quite far (at least as far as being able to predict and engineer accordingly) so it's obviously effective.

Seems hand-waving, thin on value and promoting a consulting business.

It would’ve been better to talk about the real-world including TPMS and the NTSB investigation approach... making cars very reliable and very complex aircraft safer with strict regulations.

I think that 5W and 5W2H are means to develop a network of logical reasoning. These diagrams ( fishbone and Ishikawa) are used as a means to discuss what attack areas you have concerning a problem. They are used in many ways and in many different fields. They are very useful when designing experiments (DoE). 8D and 10D teams use them extensively in (high tech) manufacturing.

> There are often multiple causes for an effect,

I find that some of the toughest bugs to solve are the ones where the undesirable effect has more than one cause.

To be precise it's the kind of situation where the bug can be triggered by 2, 3 or more independent causes, i.e. each cause is sufficient on its own to cause the bug.

Often when attempting to solve a bug like that I'll find one likely cause, address it but because the bug persists, I end up undoing the fix for cause #1, then finding cause #2 and ping ponging between them till I realise that I have to address multiple causes to make the bug go away.

Looking at this (thanks anoncept): http://psas.scripts.mit.edu/home/get_file.php?name=STPA_hand...

I'm not sure it takes us all that much further (for general purposes) than John Stuart Mill writing on causation in the nineteenth century - he did very well creating a philosophical foundation for causation-talk. (And the STPA handbook is excellent, as well.) In any case, Mill is the original source for the modern understanding plural and complex causation.

This article feels very straw man to me.

First, the 5y I've learned and run in the last 10 years takes pains to identify a fix or mitigation at each step - not just the root.

In fact for small failures and big costs, the root cause is deliberately not worked on, because analysis says the finer grain fixes are better/cheaper.

Second, root cause trees have been quite common, because there's almost never just one chain, especially when you're running a system that has plugged all the easy holes and fixed all the obvious first level problems.

Straw man article IMO, but I can't figure out for what purpose.

Not a fan of this term "the root cause". It appears to be causing some semantic trouble for the author as well. If you never get it in your head that you're going to find the "root cause" of something (initial conditions of the universe? lol), then you won't be looking for it. You'll be looking for the most promising point of intervention, which is what we do.

The examples in this article are all straw men; where's the compelling specific case where 5 whys lead to some demonstrable mistake?

Nothing to see here.

Whilst I agree with the article being that context matters a lot. But there are a lot of great things about RCAs.

A few of the things:

1. It shows visibility about the engineer's abilities

2. It attempts to show weaknesses of technology choices that happen above the people who support it

3. It attempts to show a weakness in the process. (Similar to a retro) Yet, in practice, rarely is this addressed.

The problem usually is not finding and fixing the causes but that you could have spotted bugs earlier when the damage would have been smaller. People will always make mistakes, but catching as many as possible and as early as possible lessens the consequences

This article makes me angry. I think author has some useful things to say (like, there's usually not a single root cause) but the tone is super arrogant, most of the arguments start with strawmen.

Ultimately, the root cause is always "you fucked up." At least that's what people always seem to be pushing towards when they start dragging out terms like root cause.

I recently had to do a five-whys write up after typo-ing a config file. I'm pretty sure it was some kind of shaming exercise.

Chains are chains are chains. Can you construct a sail?

(2011)