Root cause of failure, root cause of success
surfingcomplexity.blog
surfingcomplexity.blog
Perhaps I agree with the idea in an academic way, but in practice seeking the deepest root cause possible is valuable.
OK - the boulder fell because enough people were sick that day. How did we end up with a staffing policy that allowed this? OH - because the person in charge of this is a junior manager How did a junior manager get to be in charge? OH - because our top leadership didn't realize how important this boulder is.
Oh - so the real thing we need to do is educate top leadership on how important this mission is. Then they will find a good leader for it. He will then design a robust staffing policy.
The point here is that this fixed other problems along the way - getting a better leader in place solves more than just this one instance of a problem.
I get it, it's all recursive systems of balancing the boulder all the way down. But in practice you can make things exponentially better by digging for the deepest root and fixing there.
----
If I had time, I'd type out the steps more explicitly, but important questions that an analysis process must raise:
- Why is this rock here to begin with? Did it need to be before? Does it need to be now?
- How far can we move it?
- What do we do to salvage the situation when it tumbles down?
- Why do we need people to hold it up? What other solutions were explored? Why were they rejected? Has the context changed?
- Why isn't there a system that monitors how well the rock is held up? Why do we not have systems that warn about faltering assumptions, like cliff slippiness?
- Can we make it cheaper to roll the rock back up?
Etc. These questions come up when you dig wider, not deeper. Any process that doesn't produce (and ideally answer) these questions is suboptimal.
- Was there another way to satisfy customer demand without balancing a rock on a hill?
- Must the rock balance on the hill, or just a hill? Must it be the rock or just a rock?
- What level of modifications would cause the hill to no longer count as the hill? What attributes would need to be transferred from the hill to another hill to make another hill count as the hill?
- Does the rock have to be made of this material? Does it have to be this shape?
- If the rock tumbles when no-one is looking, did it really tumble?
- What regulatory bodies are concerned with this rock balancing act? What are their opinions on our performance?
- Are records kept of our performance? Does it vary or is it constant? Is it acceptable? If not, why not? Who is responsible for this judgment, and who is responsible for ensuring our performance? What tools do they lack?
- What are the operating procedures of the balancing team? Are they followed? Why not? Does circumstances require not following procedures?
- Are people getting tired balancing? Why? Are there sufficient breaks? Why not? Do people need nutritious meals and are they getting it?
- Who is responsible for updating operating procedures based on operational experience? Are workers' opinions on improvements taken seriously?
- Are new people receiving sufficient training? Why not? Who is responsible for training? What qualified them for that?
- Does the culture value successful rock balancing? Does management? Is the motivation there? What are the levels of absenteeism? Why not? Why?
- Does communication between workers happen? Do they have a shared language? Are misunderstandings common? Why?
----
There's just so many things that can (and do) go wrong, and so much that needs to be clarified in regards to how the system works that blindly strolling down a single "causal chain" is bordering on malpractice.
That point is, again, the attempt to get to root cause (whether you go deep or wide) is valuable. The idea that you won't get the ONE TRUE ANSWER shouldn't stop that effort. An organization that landed at either your or my answer is better off than one that landed at neither.
I strongly oppose that picture of failure.
Be sure to watch Richard Cook's video presentation linked on that page.
C functions have one return code for success, and a thousand for the different ways something can go wrong.
You can very clearly root cause a failure. In the reverse case you can identify aspects of the success whose absence would have caused a greater chance of failure, but there are still many more ways for a project to go wrong than there are for it to go right.
The research community on complex system safety disagree with you here.
I love things that work well, and I think that despite the world of computing being criminally suboptimal, we seem to be making progress! Slowly. And I'm grateful for it.
The world is a surprising beast of incredible complexity, and I'm in love with it.
I agree that success is silent, but I don't think that means people don't appreciate things that work well. It's just that expressing that appreciation almost always sounds shallow. Of course it works well, that's obvious. Why bring attention to things we've solved, things we take for granted.
It's easy to notice things that don't work and use that observation as a tool to improve. It's much harder to separate the things that worked well by accident from the truly good designs.
Praise is a much more fraught art than criticism. You will see more of the latter.
Thomas Edison is supposed to have made hundreds of failed attempts to get the filament of his light bulb working reliably before his first success.
A smart aleck reporter asks him; "Mr. Edison, don't you think you are more of a failure than a success given the ratio of your failed attempts to successful outcomes?.
Edison replied; "No, because i am the only person in the world who knows hundreds of ways of how NOT to make a light bulb filament"!.
Toyoda was not shaken in the least by that, because in his view, the important information were all the failed attempts that lead to the design, and those weren't in the blueprints. "By the time they've taken their copy of that model to the market, we'll have innovated past that, and they won't be able to keep up because they haven't learned from our mistakes."
Maybe there is a way to make those things more visible and clear?
That's a gross simplification of course. But even simplified thought experiments can be useful.
I'm puzzled by your reply though, about "having a good time" and "get a manager with high standards". How is that relevant to dbg23's point?
> I'm puzzled by your reply though, about "having a good time" and "talking to a manager". How is that relevant to dbg23's point?
I believe dbg23's point is coming from frustration of what needs to happen to get to this "success" as a software engineer ("to create higher quality software"), if one side believes it's good enough to have "meh quality software - but good sales team", I was sharing my thought on how to deal with this frustration..
So your advice can maybe translated as "fire your customer".
And I partly agree. There are two sides though. For one, as I'm growing and learning, I see more potential to make things better, more robust, more usable and long lasting. On the other hand we (small team) have a principle that we don't sell things our clients don't need, and we make that very clear and have said "no" in the past to adhere to that principle.
So it is really a balancing act of really understanding and communicating what technical and design quality means vs. understanding how providing it will positively impact our relationships and their goals.
I've been thinking more and more about this issue in recent years, because I realized that I/we cannot compete on quantity or speed, but rather on trust and quality if that makes sense.
You absolutely nailed it!
This is why when people ascribe Success to any one factor, i simply roll my eyes. Personal Characteristics/Motivation, Timing, Circumstances, Random Chance etc. all have a role. The relative proportion of their weights will vary for any given event but all are necessary. As a learning and adaptive organism we do have some control over some of the factors i.e. ability to change their relative weights but everyone of them still count and cannot be neglected.
On the other hand, Failure is the general status-quo because it is so easy to be unaware/miss/neglect any single factor which might be essential (i.e. has a higher weightage) to achieve the goal.
Looked at from the above pov, it becomes clear that Failure is unavoidable and not something to be afraid of but is an essential learning method, thus; trial and error methodology. The importance of personal initiative and self-effort also now becomes evident; it is the only way we have to control and tame the various factors to the extent possible in our favour.
That's one of the fascinating things about software, we still have no reason to trust it to the degree that engineers who build bridges trust their work. There are almost never a backup bridge, and we quickly dismantle ferry routes as bridges are erected.
As developers and systems administrators we expect things to fail, if we're any good at our jobs, and implement redundancy, write restore procedures and in some cases create manual backup processes. Yet it's not uncommon to have multiple links fail and still bring down our towers of complexity and instability.
It is not that there cannot be multiple points of failure; it is that it takes just one point of failure to bring the entire system down. As long as there are dependant and moving parts failure will be inherent in the system; it cannot be "inspected/tested" away. Adding redundancy does not remove failure it merely transforms/postpones it. Failure is the "natural" state; we intuitively understand this through the concept of entropy always increasing (or more colloquially; "everything is going to shit") unless we do something about it i.e. expend energy and do work.
Therefore there has to be a single cause of failure, because all else failing the CEO has mucked up. And practically speaking there is probably a root cause somewhere closer to the problem.
"There are multiple causes of failure" style arguments are technically true, but also the sort of argument that people make when they aren't going to do anything to improve. There are many things that could change. There aren't many reasons why there was a failure.
> Why was the chain manufactured with a weak link?
I don't think it is feasible to manufacture a chain without a weakest link. The materials science requirements to produce truly identical links are extreme.
The people who truly improve systems are, in my experience, the ones that do consider multiple causes of failure. The ones who latch onto narrow chains tend to fix symptoms and then think they're done.
Making a strong chain is easy, finding a use-case for a strong chain is much harder. It doesn't matter how strong the chain is if there is nobody there who can pull it. And if you have a good use case and a person to pull it you can just get a standard chain, few fail because they failed to get a standard chain.
The "chain analogy" is just that i.e. a Metaphor. You need to map it to your studies in Systems Science and Complexity Science.
Start with the works of Henry Petroski. Then look at the works of Nassim Taleb, in particular Antifragile. Finally, do a search for "Failure in Complex Systems" and you get tons of journal articles one of which i have listed above.
Oh and btw, all these scholars consider Taleb like a wanker for a reason :)
I agree with this basic premise… if a single fat-fingered SQL command can bring down a service, for a random example, it’s not the operator that is really at fault but the system that has the operator typing SQL directly at the database to do maintenance.
However, unless you identify the specific causal pathway that led to the failure you can’t improve the systems! Root cause attribution strategies like “5 whys” do a pretty good job of teasing out the layers of a system failure to identify both the specific causal pathway and the systematic failures, if you keep it going long enough.
What makes a cause a root cause?
What is the root cause of, say, scurvy? Some millions of years ago, our monkey ancestors lost the ability to manufacture vitamin C in their own bodies. Since then, anyone who stops eating enough vitamin C faces scurvy.
But this is not actionable, as another commenter says. We cannot yet fix our genomes to bring internal production of vitamin C back online. So we say that root cause of scurvy is lack of vitamin C in that person's diet, which can be fixed very easily now.
If/when technologies like CRISPR mature enough that playing around with genes and their expression becomes a routine thing, we might change our genomes so that vitamin C gets produced internally again, thus shifting the perceived root cause.
Deeper causes include education about nutrition and availability/affordability of vitamin C rich foods. The causes of problems here are deeper, sociological and cultural, and probably to some extent lack of money and other more primitive needs.
(Douglas Adams)
Remark that this does not mean we made it happen less. Just that we soothed the part of our brain that was scared.
Great viewpoint!
You are technically right, so let's just pick a pragmatic and useful definition of root cause: the preceding event or circumstance that is at right the level of abstraction and proximity where knowledge of this will be maximally helpful to people in similar situations in the future.
Left with this, people will still revert to the status quo because they’ve been offered no true alternative that provides more value.