I have been feeling cynical about the "blameless" culture seeping into entire organizations recently. At the end of some quarters where my team has fallen behind, I run away from the question of "who was really at fault here?". I probably should not do that...
If people know there is a risk of blame, it changes their behavior. It creates incentives to destroy evidence, or avoid creating it in the first place. It creates incentives to stay inside your organizational and political boundaries rather than reach out to solve problems. Whereas if you have a successful blameless culture it is much easier to find out what actually happened.
The NASA ASRS system: https://asrs.arc.nasa.gov/
FAR 91.3: https://www.law.cornell.edu/cfr/text/14/91.3
The first of those is the aviation safety reporting system, which provides explicitly for immunity from sanction for inadvertent violations of FAA regulations (with certain limitations).
The second of those is basically the first aviation law clause for flight crews (after 91.1 which is the "this is what this section of law applies to" and 91.2 which doesn't exist) and gives broad discretion to deviate from any rule [in this part of federal law] as needed to address an emergency.
ASRS is directly and strongly supportive of the overall blameless culture in aviation. 91.3 is moderately supportive of it.
Those are often confused.
As stated earlier, blameless postmortems are for RCA of a particular incident. If you shoot every engineer who causes a incident you will succeed in having no incidents because no one will bring them up or making any changes for fear of getting shot.
That doesn’t mean people aren’t accountable for their actions in “blameless” cultures, but problems are first assumed to be the result of systemic issues, deserving to be fixed, rather than individual issues, deserving to be punished.
Two of them relevant to this story were:
webservers.regenerate.all.cache.files
webservers.release-prep.stop.all.services
Guy's first day; I'm showing him the ropes; we push the marketing data update and set about regenerating all the cache files by manually picking the correct file from the folder of all possible files. I'm sure we can all guess what happened to make this a story remotely worth telling...
Complete site outage. Completely unnecessary. Completely human error.
Should we blame the guy who clicked on the file that was directly adjacent to the one he intended? Should we blame me as the guy overseeing the training? Or should we change the system so that files that we use multiple times everyday and are safe/innocuous are't right next to an E-stop/EPO button? Or maybe we should change the system so that pushing marketing data refreshes the caches files automatically?
Blameless culture favors the latter actions over the former and tends to make your operation stronger and more resilient over time. The experts (and the novices) who made the mistake can speak freely about what happened and how we might prevent it, without fearing reprisal.
If someone repeatedly kills the site by mistake time after time, despite reasonable safeguards being in place, they should face disciplinary action. But when they make an honest mistake because we left an idling chainsaw laying around on the workbench, it makes no sense to blame them for grabbing it by mistake.
If you promise people that there will be no blame, no punishment, no nothing, then they may speak more forthrightly. Do you want that honesty or do you prefer to retain the option of punishment?
Of course you can go deeper and ask why was this process not done correctly etc but at the end of the day, it’s about actionable changes that ensure mistakes don’t get repeated.
Unless you just call everything a process: "His childhood was difficult and that's why he fouls people. Let's call his childhood a process."
I think at some point you have to say that some things are not processes, or cannot usefully described as such. Or they devolve into Zeno's paradox.
Well... attempting to prevent stuff like bullying or domestic violence is a worthwhile goal in itself.
For the example of the soccer player, one might also question other incentives (e.g. bonuses tied to winning games no-matter-what), or why a player with a known history was hired in the first place.
Obviously a worthwhile goal, but not a process failure on the part of the football team.
I hope we aren't using football as an analogy for avionic engineering with 1000's of people.
The process that hasn't cut you from the football team.
I mean in theory, you're absolutely right but so rarely does it ever actually shake out that way. It seems like corporations continue to make the same mistakes while simultaneously being held less and less accountable.
Literally hundreds of people have died due to Boeing's negligence recently and the prevailing opinion seems to be that no one is ultimately going to be held accountable. (at least not meaningfully)
Ideals are great and all but if it results in zero accountability when hundreds of people die, maybe its time for us to re-evaluate.
Maybe we need a new way of expressing "blameless" that still expresses that personal responsibilty for quality work should be expected, but that punitive blame politics are simply a very ineffective way of encouraging that or finding the truth when problems inevitably occur.
Making it clear who did it does not mean firing that person or making them feel like shit internally, it's a learning process, but the higher up you go the more you should feel bad (presumably you went through the learning process and earning big bucks as a result, so wtf)
That may have been the case, but their are entire company values that insist on the blameless culture. This certainly sends a message organization wide to all levels that blame is not tolerated.
You can 100% blame people for doing bad things. But in an engineering incident you want to find the RCA.
In SDE you may find the single git commit that took down everything, it has a author. Fine, there is blame. But why was there no review, no unit tests, no functional tests, QA, etc....
So you fire the engineer but make no other changes and it happens again, how is that helpful?
In my current role I promote a blameless culture and the result is my team proactively raise incidents. We have more incidents than other teams but the incidents are generally controlled and problems are visible, prioritised and closed off. Coincidentally, my teams have record high Mean Times Between Failures and I'm now working with the other teams to fix their safety culture.
The lack of thinking about safety culture in this way is one of the reason that I don't believe most "Software Engineers" should be considered engineers.
Leadership is always to blame if something goes wrong, it's only fair as they will get the credit if things go right. That's the price of leadership.
Quite a logical leap to say that everyone in a corp deserves to be held 'blameless' except the CEO - imo, you can't have it both ways.