A few words about Blameless culture
gybe.ca
gybe.ca
I have seen a lot of incidents and I cannot think of one where a single person was to blame. Sure, one person ran a command or made a bad commit. However, someone else granted them access, someone trained them. A manager either reviewed the process performed or never considered a process was required. A lot of company cultures do not promote proper risk management in technical processes.
I get a lot of satisfaction from technical post-mortems. There is always something that could have been done, a process that could have been in place or a software/infrastructure change to mitigate the problem.
A couple of companies have worked for have had individuals that did not subscribe to this culture. They would want a name. I never knew why exactly. Maybe it was to block a pay bump or defer a promotion. As the tech lead or dev manager any team failings are my responsibility and in companies where bullets are fired I take them. I find this protects the team and helps a blameless culture thrive amongst engineers.
Generally, I find this culture leads to fewer incidents and problems as the openness when things are going wrong allows for faster response times and software/process changes in review.
One of my worst and most consequential work experiences was with the opposite type of culture.
There's lots to be said about it, but in reflecting on things I've often felt like one of the worst parts of that type of culture ( "blameful"?) was a profound lack of trust, a feeling that the blame belied some lack of real interest in improving the situation, as opposed to being punitive against certain individuals for unrelated reasons. That is, the individuals who were targets of the most blame as far as I could tell were generally being targeted for cliquey reasons that had nothing to do with performance or anything of that sort. Conversely, people in the "in group" were given a free pass for all sorts of serious problems. Getting a free pass wasn't really the problem, it was a sense that blame wasn't really about the ostensible blameworthy act, it was that it was being meted out as a kind of superficial leverage for some other thing.
Also, because of these types of issues, serious, legitimate administrative, communication, and other systemic problems never got addressed, because the blame was being used as a kind of social ostracizing mechanism, and the actual underlying problems weren't fixed in their entirety. So in many ways a lot of the problems got worse, not better.
In contrasts, in other environments (even the same place at a different point in time) where there's an emphasis on problem solving and figuring out what could be done differently by everyone, and assuming the best intentions and basic competence, trust underlies everything. You're motivated more because you believe that actions have fair consequences, and that everyone has each other's best interests in mind.
There have been many folks like this, in my career. I am humbled and grateful.
I worked for a Japanese corporation, where they didn't "blame," per se, but did figure out if someone was Responsible. If so, that someone was expected to cop to it, and the team would then close ranks, and fix the issue. Lesson learned.
I know that part of it is lawyers and police. If you say "I'm sorry," in many contexts, that puts you on the hook, legally. Most lawyers will advise you to keep your mouth shut, and never, ever admit a thing.
I was taught to promptly admit when I'm wrong; regardless of the consequences. Also, to not cop to stuff that I'm not responsible for.
I have often had arguments with (American) people, where they are 100%, without-a-shred-of-doubt, wrong. Often, embarrassingly so, yet, they Will. Not. Admit. It.
I could bring the spirits of Carl Sagan and Stephen Hawking in, to show them they are wrong, and they will still never admit it.
It's actually kind of sad.
My ability to apologize worked extremely well, in the Japanese corporation. They respected it, and I was given a great deal of deference.
On the other hand, the Americans tended to be pretty much dicks about it. If I apologized, they usually tried to use it to gain some leverage, and bully me.
I don't think there's some general "(American) culture of blamelessness," and I think your comment might misunderstand the point of it.
My understanding about "blameless culture" is it's all about avoiding counterproductive things like scapegoating, blame-shifting, defensiveness, etc. Removing those things encourages the full cooperation of the team in resolving the issue, by removing the anxiety of punishment for being the guy who's being blamed. Everyone's supposed to take responsibility.
At the same time, we have made the words "you screwed up" almost a swear in the workplace. If you can normalize "you screwed up" so that everyone hears it (and let's face it, everyone screws up once in a while), those words don't hold nearly as much power. In a sense, the ideals behind giving every child a participation trophy have been extended into the workplace under the guise of avoiding counterproductive things like scapegoating, and I'm pretty convinced that's a bad thing.
I can't. There's no value in it. What did we, as a team, do that allowed the incident to happen? Yes, John Smith shouldn't have dropped the tables on production, obviously, but does he really not know that as part of the incident response that he's (presumably) also dealing with?
If he's truly not aware that was a mistake, there's an underlying transparency issue that goes way beyond telling an individual they screwed up.
"Negligent" doesn't just mean "made a mistake." It means something more like "their carelessness led to a mistake."
That person hearing "you screwed up" will cause significant behavior change. I daresay it will encourage them to make the prod tables very hard to drop, and since they are presumably a smart person, when combined with the postmortem of the incident, it will encourage them to look for and proactively fix similar problems, and generally align the team with good DevOps practices.
It is important in all of this that the right person gets the message. I assume you expect that to not happen, since that is one of the theses of "blameless postmortems."
Ideally a blameless post mortem allows the freedom to identify any of the potential fixes that could've stopped this, and empowers anyone who could've dealt with it to deal with future issues. If you blame the manager then that can implicitly absolve everyone else in the chain.
With that said, I would agree that having a primary owner of things does matter. For that reason, sure, making the manager more aware might help in future. I still think it's a bad idea for org culture though because many managers will respond to "You screwed up." with trying to ensure future blameables find their way to another target. Instead, I'd prefer approaching the manager with "We could've caught this in [any of the ways we could've caught it].", and if the manager doesn't care at that point they're just fully incompetent.
I agree that what you are describing is the ideal, but I don't think it happens often. What I have seen more is that the postmortem turns into 10 different open bugs, 5 of which (usually bandaids that stop that particular failure mode) get closed quickly and the rest of which (often including the real solution) get put off until the next crisis.
Ownership is really important, but hand in hand with ownership is responsibility. That step is missing in many orgs today.
With that said, I think your comment describes kiss-up and corporate fiefdom culture. There are plenty of examples of that. I wonder if non-American cultures are different in that respect.
One very key distinction is freedom to admit error vs "the baby is not ugly." I don't think blameless cultures are the norm, where people volunteer to write post portems and voluntarily submit after action plans..
I think it'd be a little more potent to just say hey, there's a difference between saying someone is partially responsible for an issue, and pestering them about it (e.g. telling them they did a bad thing when they likely already know - I suspect that is what people want to avoid when they go "blameless"). Blame is blame, and disrespect is disrespect. I like to see them as orthogonal.
The article's fire suppression story doesn't even seem like a blame problem to me. It was a jumping to conclusions problem. But again, maybe I only think this because the definition of "blame" is being played with.
Most of all the author's gratitude toward the patient fire chief whose leadership example he praises.
We could do with more of this in cybersecurity.
We're all surrounded by insecure software and defective devices churned out by companies chasing a quick profit. We're mired in complexity, non-determinism and broken systems. The experts who designed them throw up their hands in confusion. And we're hamstrung with laws and policies, which despite their laudable intentions only beget twisted systems filled with perverse incentives and loopholes.
How do we solve this? Blame employees! And then humiliate them! Make our cyber team into a punitive, even vindictive internal police force?
Karen from HR is sobbing into her cardigan sleeve because ICT caught her out in a "Phishing drill". Now she'll have to go on a humiliating "this is how a computer works" course - despite the fact she was programming before her IT manager was born. No doubt she'll get her own back in time.
As ICT and cyber teams we could learn a lot from more traditional organisational culture.
Should I DM you from info on your site?
PS: your website is so great. Love it.
Would love to know how this was explained. I wonder if the author remembers how it was put.
Blameless does not mean it is ok to break stuff. Or that you are not supposed to feel shame. Or that doing shit job comes with no consequences.
Blameless culture was supposed to mean that it is more important to figure out why something broke and find solutions to fix it than pursue penalties for people who did it. That pursuing penalties and assigning blame tends to turn into this nasty political circus that frequently makes it impossible to figure out what really happened and if somebody is found to be responsible, it frequently is the person who is least adept at politics.
So we say: "The only way we know to be able to maximise improvements based on factual analysis is to stop clouding the process with politics of blame finding."
So this is more like getting away with murder on a technicality.
Not blaming is not the goal. It is the means to uncovering the truth and figuring the way to prevent it in the future.
My last company had a culture of blameless post mortems, and also a strong culture of accountability, so definitely no one would fit the first definition. And the company was too small to have non technical managers.
My current company (big corporation) officially supports blameless reaction to incidents, but internally has a strong culture of cover-up and blame ping-pong, but I still cannot think of anyone who would believe "blameless means no accountability". It just comes from deeply ingrained habits (lots of people do their whole career inside the company, lots of them even getting their masters at the company's own "university". Behavior is hard to change when everyone comes out of the same mould). I also think it comes from inadequate processes: where in the previous company the postmortem was mostly a meeting done by and for the technical team, my new company has a strong culture of "escalation meetings", where the highest managers quickly get drafted in, which obviously has a strong influence on the capacity to conduct an open and blameless process.
I fully understand that some people are not looking for that kind of thing at all. Many would rather be nameless cogs, and some would simply rather not do the work (technical or soft skills). That’s fine too. For that latter group, whatever the management buzz words de jour are are likely to be employed where you work sooner rather than later.
My take is that it is probably better to accept bad workers in big orgs rather than trying to reach some kind of competence utopia, to get the incentives right.
No, that's still somewhat blameful.
If you had stuck with "blameless does not mean no accountability", I think you're right.
But the emphasis of the article specifically does not assume people do bad work on purpose:
> assume the individuals involved had the best of intentions, and either they did not have the correct information to make a better decision, or the tools allowed them to make a mistake.
I'd like to hear other takes on how accountability plays into blameless culture.
Blameless does not mean "you scan screw up as much as you want". If you start being careless, you will be shown the door.
I did know arctic wolf was big about being friendly to Trans and Autistic people, which I did like.
For me, long ago I was busy enough at work, I embraced blamelessness because I don't have time to waste on that crap. I immediately ended up with results where people who were expecting blame and didnt receive it were suddenly willing to get better. I learnt almost counter-intuitively or unexpectedly that you must have a blameless culture. Been blameless for at least a decade.
Props to arctic wolf, and well my current employer who is awesome.
Some years ago, we had an "incident" where one of our developers discovered he had admin access even though he was logged into a different account of his. It was a multi-tenant SaaS. Since he was supposed to be admin anyway, I wondered where the harm was.
Now, though, I think he was just practising the blameless incident response in case something serious did eventually come up. Once that happens, you don't have time to practise in peace anymore.
Somehow I have a hard time believing that. The author using "individual" in prose is a give away for militaristic leadership style, where you quickly learn to cover up any mess up so the direct superiors don't lose face.
Blameless ...
Unless the sysadmin got 1000 mails because the fired guy did "sudo rm *log" trying to guess the admin password I don't even know how you find out if something is a cover up, a mistake or a set up.
"shortly thereafter termination of their employment"
There was no due process there that is for sure.
For me and the teams I'm on I try to instill an attitude of humility that at its core says "Any of us on any given day have the possibility of making mistakes."
Mistakes happen. Anyone can make one. Finger pointing is usually a sign of either overconfidence ("I would never be that dumb!") or insecurity ("I need this person to take the heat so I look better.").
To me that's what is important in blameless culture. It encourages people to approach things in a humble manner, which in my experience leads to better outcomes.
Why it should not potentially affect compensations on progressions is beyond me. It might not as a singular event but but patterns of risky or disorganized behavior are commo rather than individual mistakes. Errors come in bulks but they might not express at the same time. In my experience there are people that do zero,every blue moon or infinite mistakes, and there are people that are good in zero,one or very many domains.
What an organisation needs for growth and coworkers need for sustainable stress reduction is the minimization of mistakes/output ratio.
If people get blamed a lot they switch to stereotypical behaviors and cant work in an innovative, creative fashion. A mistake took place? Is it institutional, personal? What other mistakes would present themselves if that mistake was a pattern? What is better for morale than finding another mistake due to analysis before it expresses itself?
Note that you bring in the example of a court, and this is how a court works: a suspect is presumed innocent until proven otherwise.
The main point is that a blameless culture fosters honesty and team work. A blameful culture fosters covering up and lying to look good to the manager. But if someone is careless, they will get into trouble in both cases. See the example with the person who deleted the production DB: they would likely never have said "guys, drop everything you are doing, I deleted the prod database and need help" in a blameful environment - even though this is the best reaction to have for the business. But even in a blameless culture, it takes a lot of courage to admit such a mistake and ask for help. Assuming that people are not afraid to do such mistakes just because they will not be immediately fired for it is quite absurd.
They are essentially filtering for the very same "C players" they despise and mock.
Like, the output disparity in bigger teams is extreme in programming if you let people loose. And you better not stick your head out in bad workplaces and the advice should be to do as little as possible to avoid being fired.
quality is a thing and forward movement with quality is possible.
What we are avoiding here is a CEO asking, "What caused the outage" and your manager responds "Bill wrote some code that caused a deadlock". It's possible Bill is working on fragile parts of the system, or writes most of the code. Blame at an individual level for an incident is unhelpful and, as OP points out, possibly creates a team with worse delivery incentives.
avoidance of negative quality is not quality.
Suppose employee E makes an error. Management decides to fire them or not. Firing someone for one error (of an accidental, non-malicious variety) makes no sense whatsoever.
The set of companies who do fire for a single accidental error is vanishingly small and very likely the least desirable of employers (as they’re likely making other colossally bad decisions as well).
If you’re working for a sane employer, at-will’s overlay with blameless transparency is fine. If you’re working for a terrible employer, you want to change that anyway.
A sane employer could decide to act on a repeated, long-established pattern of errors, which is frankly good for the organization and the remaining employees. In that case, the transparency around “most everyone’s error rate is a couple per year, while Pat’s error rate is 4 per week for the last 3 months; it’s time to part ways with Pat” is helpful.