Why Blameless Post-Mortems | Hacker News Reader

Why Blameless Post-Mortems

medium.com

61 pointstptacek26 comments

Expanding on the post, one corollary of "people are rarely the cause of the outage" is that "you can't fix people". Given that Dave accidentally caused an outage by pressing some button he should't have, explaining to him why he shouldn't press the button won't prevent anyone else from making the same mistake.

Instead, you need to look at the reasons underlying why Dave pressed the button. Is the button confusingly labeled? Is there no playbook on what the process for pressing the button should be? If you fix those underlying problems, you will prevent other people from making the same mistake Dave made.

Dave actually wasn't the one who pushed the button. I was the button-pusher in this particular outage :)

It sucks being the one who caused an outage, but having a culture where people put blame on the process rather than the person takes away a lot of the stress and naturally motivates people to solve the core issue as quickly as possible. The points in the post about psychological safety and underlying systemic causes of an outage are spot on.

If your response to every problem is "we need a process to make this idiot proof," pretty soon you are going to be up to your eyeballs in CYA training webinars and permission requests and committee reviews. Small groups of competent people who can be trusted to wield powerful tools will start to run circles around you.

My employer does a reasonable job with this. Our tools usually make the path of least resistance the right thing. We have guardrails that help prevent accidents. But at the end of the day, our controls presume competence and good faith, and there's always an "I know what I'm doing" button. Actions taken after pressing it incur a bit more personal responsibility.

When parent poster says “look at the underlying reasons why”, you jump to untrustworthy idiot employees hampered by CYA red tape.

When your employer does the same thing, it leads to “good tools with sensible affordances and guard rails” for competent responsible people.

Why is that?

It's a balance. "Harden the process" is good and necessary in moderation for the reasons already mentioned, but can be taken too far, to the point that the process is so rigid it's paralyzing.

A finding of "the process was reasonable and the individual wasn't" or "a process guaranteed to stop this kind of action would be unreasonably heavy" may sometimes need to be on the table.

Small groups of competent people (e.g., startups) can also usually do less damage. One quite reasonable approach is to give people small environments they can meaningfully work with freely, but require change control to go outside it.

Also, a good postmortem process ought to treat "Why didn't we do this in time" as a problem to solve, too. (To be fair I don't immediately know of good examples of this.)

I'm not reading it this way, illustrated by:

> explaining to him why he shouldn't press the button won't prevent anyone else from making the same mistake.

How I interpret the point is: don't fix people, fix tool.

You can't tell someone not to break the system, you need to make the system less easy to break, labels on buttons, confirmation dialogs, no double negatives, approvals if need be.

Reminds me of a deployment with a customer (some 15y ago), I wasn't allowed to "touch prod" myself so I was sitting next to a button pusher doing what I was telling him to do (irony...). At some point there was some trickery involved to reboot HA instances one at a time rather than simultaneously, and you had to press "No" when prompted if you wanted to reboot. Before the dialog showed up, I pre-conditioned the guy, telling him "it's going to ask if you want to reboot, say no. Whatever you do, do not click on yes". This wasn't the first time he deployed. He said "oh, ok" then sure enough clicked on yes, and the IT police was in our room 2 minutes later because of a prod outage we just caused.

I have thought many times of what could have prevented that - allowing me to deploy myself, doing more training to that guy, finding a competent replacement to him, more documentation. There is no process that would have worked better to fix the issue than finding a way to remove this dialog box - today that would be done through automation.

This is where netflix memo hits it right. When you burden creative people with processes, you take away the alpha in innovation...

I like to think SWEs are in creative businesses... But this doesn't always hold true...

You don't seem to have learned the lesson at all.

The whole point is CYA doesn't work. Its not trying to make it idiot proof, it's trying to remove the idiot from being involved at all.

Ties to Warren Buffet, someone probably smarter than me ;) "Criticize generally, praise specifically" or "Praise by name, criticize by category". In my experience the biggest problem with criticizing specific actions is that it destroys loyalty and undermines trust.

My $0.02:

Despite being a strong believer in it, I've worked at places that did blameless post-mortems and it was sometimes frustrating because remediation often felt like a second-class citizen. Remediation is often much more difficult in these environments because systematic changes can obviously much more complex than "fixing" an individual's behavior.

If someone pushes the big red button accidentally I'm fully on-board with the "well what led them to push it and why were there no safeguards?" but it also kinda sucks if you spend so much time on that conversation that there's no time left for "ok, who's going install the confirmation screen for the big red button?"

Sometimes the "remediation" was even just "well, that's just the way it is and we can't do anything about it easily, but hopefully the post-mortem itself will serve as a way to inform people not to push the big red button" which is obviously frustrating because memories are short and people come and go quite frequently in the tech industry.

Anyway, and this might be obvious to some, but remediation is important and sometimes incredibly difficult in these environments. Adopting such an environment requires a commitment to actually fixing the systemic issues. It can sometimes feel more frustrating than "blameworthy" post-mortems especially if it's the same individual(s) over and over again.

> I've worked at places that did blameless post-mortems and it was sometimes frustrating because remediation often felt like a second-class citizen

Maybe I'm missing something, but that seems orthogonal to the blamelessness. Blame won't necessarily bring remediation front-and-center either (and will probably do far more harm do that end).

> can obviously much more complex than "fixing" an individual's behavior.

My favorite "hate" on the blameless post-mortem's is when it comes down to altering a process that someone owns, because then you're not only fighting for time-budget to fix and issue but you're potentially fighting what is, to them, part of their job and existence.

To get tangential, this is also why I hate big-A Agile, because if you have a person who's job is "do Agile," the scrum (master|lord|bag|dumpster), any changes, deviations, or complaints end up feeling like you're attacking that person's job. Never a fight you want to be in.

All good reasons. I'll take the first one a little further... when such an event happens, it may cost the company actual money. If you fire or create an environment where someone wants to look for another job, then whatever that event cost you is no longer an investment in the education and experience of your employee -- you've now invested that money in someone else's employee.

A person who plays a part in a costly error is going to be much more cautious in the future and be very helpful in improving systems so it can't happen again. You want to foster that approach rather than discourage it.

> A person who plays a part in a costly error is going to be much more cautious in the future

Unless they're just a fucking cowboy all the time and you should just fire him now because he's going to do it again. Maybe gauge the level of surprise of his collegues. Are they shocked at how this could happen or are they going "sounds like Jerry".

Fair enough. Sometimes I forget that these cowboys are out there (we got rid of our last one almost 10 years ago).

To add to this - if there's a pattern of having to take part blame/ownership for people who should know better I'm going to move on to greener pastures.

Some people work more carefully than others. This is absolutely a thing in software development/operations (my fields). When I was young, I had more screw-ups because I was still learning "how to work carefully" with certain things... It's hard for me to admit this but it is 100% the truth. When things went wrong, I felt the need to take it on my shoulders vs. to put it on the rest of the team who likely had nothing to do with the error.

As with all things this isn't black and white, which is why I'm making this comment.

I want to minimise the number of times that a single person "being careful" is the difference between a smooth experience and an extended service outage.

I also favour responding to mistakes quickly over preventing them completely.

I've seen a lot of teams where there was the one keeper of the systems who knew how to do all the deployments and migrations. In every situation they became a bottleneck for the team, and prevented any real improvements being made.

In my experience you literally can't idiot-proof ops. In every operational role I've ever been in there's still very trusted people hammering out commands with very privileged accounts.

Even at my last role people would get angry at me when I'd ask for "command confirmation" and/or "an extra sets of eyes".

In my world, I favor getting it right vs. responding to mistakes. It's the difference between "scalable" and "firefighting".

Having a SPF asset who holds the keys to the entire kingdom is a completely different problem, and a different conversation.

If you keep making mistakes that get caught in testing, you need to be careful. If you keep making mistakes that get caught in production, you need to write better tests.

> If you keep making mistakes that get caught in testing

Ha, yea. I can get behind this. There are times that one dumb thing causes the entirety of my tests to crumble and I'm always admittedly laughing saying "that's why we have tests!"

To expand on the CYA reason why blaming causes problems, people who want to avoid blame want to say there's nothing they could have done that would have prevented the incident. In a blameless environment people are more likely to figure out a way they can contribute to prevention.

Honestly, I expected the article to be hot trash -- considering how I saw the implementation at our shop. I've changed my mind. We just go about it wrong. What went right, what went wrong, where did we get lucky... If we answered those questions rather than dodge responsibility... this could be useful.

Think of it this way: Blameless means looking for solutions (which involves identifying root causes), not scapegoats.

Sometimes, it really is one asshole and they need to be fired. Most of the time, it's not.

> As a rule, people are not the cause of an outage.

But leaders are.

Most enterprise post-mortems I've read will deliver pages and pages explaining (or justifying) poor system architectures. But they will never, ever criticize the executive or tech lead who decided on that structure to begin with.

Everyone makes mistakes, for sure. One stray outage is likely technical in nature. A simple oversight that can be corrected. But if an org's systems are constantly failing or breaking, then it falls to the leader of that org to reshape its structure.

Or they can pretend that blame doesn't exist. The system is fine. A band-aid will suffice.

Without commitment to fix systemic failures, "blamelessness" enables the abdication of leadership.

> Most enterprise post-mortems I've read will deliver pages and pages explaining (or justifying) poor system architectures. But they will never, ever criticize the executive or tech lead who decided on that structure to begin with.

Certainly explains their popularity lol

Reply on news.ycombinator.com