It's not your fault

I have some paradoxical feelings about "blameless" retro culture that I'll try to sum up.

In general, I'm in favor of the approach. I don't think singling people out and bullying or shaming them for their mistakes ever works. I think most well-intentioned engineers will already beat themselves up plenty for making a serious mistake, and they don't need any encouragement to do so. I know I do.

On the other hand, there is a red line. At a place I worked, a DBA was let go after he repeatedly brought production down for 45 minutes to an hour at a time by running intensive queries of his own design for data-gathering, in some cases, after being explicitly told not to do that against the prod database. This was a person whose job description required him to have access to prod.

There were process problems, maybe - being allowed to run whatever queries you want on production under your own authority, sure - but his cavalier attitude towards a production environment was still unacceptable. Process can only help when people are well-intentioned and doing their best; if people are malicious or negligent or just not good at their jobs, adding more process to get around that only makes things worse.

The question isn't whether it's your fault, it's whether you take responsibility for it. If no one takes responsibility for anything then you get nowhere. And if you take responsibility for it, it's your fault if it goes wrong.

> "Hero Programmer" is a derogatory name for a programmer who chooses to fix problems in epic, caffeine-fueled 36-hour coding sessions that frequently just kick the can down the road to the next heroic 36-hour coding blitz. Hero programmers would rather react than plan. Projects with hero programmers working on them often make a lot of progress initially, but never arrive at a stable state of completion

Maybe there are workplaces where people get together to collaborate on a design and then break the design down into tasks and assign those tasks to programmers to implement. Maybe this process is performed until the project is done. Maybe. But I've never seen it. I see people taking responsibility for small and large tasks, and the large ones sometimes involve a single person re-implementing entire systems spread across thousands of files (though not necessarily in "36-hour coding blitzes").

The way to attribute accountability is protocols. Create and maintain protocols that are known ways to do something safely. If you broke something by disregarding the protocol, then you've fucked up. If you broke something by complying with the protocol, then you discovered the protocol needs to be updated.

This may be true, but when things break people will look for a scapegoat. So when things break, and you are mostly-responsible for initiating the failure, use collective language ("we" and not "I"), frame the failure as a systems failure when you are talking to management or the executives, look cool even if you are feeling stressed out. Manage the narrative! Sure, you flipped the switch or whatever, but try and survive the event. Just because you think its a system's failure doesn't mean other people share this belief, don't volunteer to be thrown off the bus.

I feel like, for me at least, being able to say "That was my fault" makes me remember what I did. It makes me document that failure. And makes me tell others to avoid what I did.

I've been training a JR lately and I will always say "Here's all the things I did wrong when I did this, so don't do this things"

If I did it, he could easily do it, and if we can all avoid my mistakes, so much the better.

It WAS my fault quite a few times, and I'm ok with that. And luckily, everyone else around here is too. I'd hate to work somewhere that punishes honest mistakes. (there are limits, of course)

It being your fault is a good way to go through life.

It focuses your mind on what you could do to avoid those situations in the first place.

If it’s your fault prod broke you can fix the process or you can look for a new job where the process is already fixed. Or you can find a role that doesn’t involve pushing code to prod, maybe in R&D, etc.

> When you look at things from this lens, all the successes of a website, an application, or an organization flow from the talents and genius of a few individuals. It’s a compelling outlook because, well, empirically it can definitely appear this way, and it’s naturally aligned with the other dominant societal ideas we have about individuality.

In many large organizations, much of the success comes from the foresight, insight, and hard "work" comes from a few benefitting many. The reality is, it is individuals and not some collective group or "teams".

This is the default approach taken by the airline industry, just culture:

"Just culture is a concept related to systems thinking which emphasizes that mistakes are generally a product of faulty organizational cultures, rather than solely brought about by the person or persons directly involved."

https://en.wikipedia.org/wiki/Just_culture

I've been at multiple companies with formalized postmortem processes that involved final documents that were widely available within the company. And at each of them, I spent a few hours one afternoon reading down the last 30 or so. And I was struck by how many of them had the same root cause (different at each company, because each companies culture and tech-stack is unique, which creates unique failure modes). So at $firstCompany, 75% of their errors were "Dev made a change in the big ball of mud that is our product to fix a bug in feature A, didn't realize that code was reused in feature B, and since we don't have sufficient automated testing feature B was never checked before it hit prod." At $secondCompany, a software consultancy, all of the failures were because each deployment was a unique special snowflake with no backups or dev-test environment (they had originally been created with hardware for those, but that hardware was inevitably cannibalized to keep prod going or used to add a new feature to prod, without back-fill). At $thirdCompany, they had a massively complex roll-out process with larger and larger rings of internal dog-fooding, and developers kept screwing it up, advancing software that was known bad or hurrying things along too quickly or missing roll-out dates or whatever.

Reading a bunch of these documents in a row made the failure mode for the corporate culture clear and obvious. The difference between effective and ineffective teams is whether they decided to fix the underlying problem or not. At $firstCompany I asked my bosses to take a dev team and fix enough stuff so that we could run a full automated test suite reliably before every deployment, was told it wasn't a good idea, and left the company shortly afterwards. At $secondCompany they tried things to get that to work, but had major cultural problems with getting individuals on each team to have the discipline to backfill the hardware and actually keep prod and dev-test identical. At $thirdCompany they looked at the same problem of individual discipline and created a team of professional non-developers who were full time focused on controlling software roll-outs so that every new roll-out went to the right level at the right time. We devs gratefully handed over this complex process to that team, and the professionals did a great job.

Be like $thirdCompany: figure out your larger internal pain points and fix those; you actually need to be hit over the head with the data on this to make it clear how your culture and tech stack are failing.

It is your fault.

If you introduce a bug and let's say in part due to insufficient documentation in the code, code reviews that didn't catch it, your change passed all tests and you even made some new tests to exactly test corner cases of the change you were making, it ran fine in beta testing. Then it is still (in part) your fault!

Everybody makes mistakes, and systems should be resilient to mistakes. That doesn't mean nobody is at fault for anything. It means everybody has faults, and if somebody doesn't think they have any faults then why should they think they have anything to learn or improve?

Fault finding should be done. People who wrote the documentation should be given notice and opportunity to improve. People who reviewed the change should as well. And certainly the person who introduced the bug. If people can not cope with accepting the blame for a problem they had responsibility for introducing, they should not be in that line of work.

The point (which many people missed) is that there should be a process or tooling in place to make it hard or ideally impossible for a mistake like that to completely disable the system.

And that it is normal for humans to make mistakes every once and awhile. If the system can't deal with that, then it's not a robust system.

> would rather react than plan

Can’t help but notice that the people who throw around accusations like this are… still blaming the individual.

perhaps the best example i can think of was the gitlab incident.

and if we’re honest it’s because the engineer’s name is Yorick.

"Great Men are the cause of everything!"

"No, you're wrong, it's Systems!"

"No, it's Great Men!"

"No, it's Systems!"

The author found out there's another way to think about events other than "someone did it". Now he thinks he can apply that thought to everything. Sometimes it's the system, sometimes the person. You can't generalize easily.

> breaking prod is not your fault. Put differently: breaking prod is a systems failure, not an individual one.

False dichotomy.

Sorry, sometimes breaking prod is your fault. As a senior engineer there is no excuse for forgetting a where clause on your SQL delete. And doubly no excuse for not having somebody else look at it.