Hacker News Reader

Comment by cletus | Hacker News Reader

One tradition I really appreciate at Google is the post-mortem. Any serious outage or bug will be followed by a detailed post-mortem detailing a timeline of what went wrong and why.

Sometimes this will go as far as establishing a "war room", possibly for weeks, dragging in people from a number of different areas to address a particular issue.

It's not about pointing the finger or otherwise apportioning blame. It's about learning from mistakes and preventing them from happening again.

That being said, I've worked at other places where shame is attempted to be used in such situations (which is what this is). In my experience it just fosters a hostile environment.

View on HN

At a prior job I was a key member of a team that did a 2 week long, intense root cause analysis of a prod issue that threatened a multi million dollar contract. I consider it one of the most interesting and rewarding things I've done in my career. All obstacles were removed, we could follow the clues wherever they took us, use any resources etc. In the end it took careful analysis, many simulations and hard work to find and fix the root cause, and no fingers were pointed, instead we learned some good lessons and were better for it.

When done right, RCA is a great thing.

A good programmer will feel shame from that whole process even if nobody points the finger at them. The blame game just isn't needed.

A bad programmer shouldn't work there in the first place, and if they need to be blamed, they're probably doing other things to destroy morale.

s/programmer/worker

If we stay in common sense ranges, a bug is not something you have to shame or proud. It is just part of the working process.

I find it interesting that you'd feel shame over a bug. Why?

This is a very well-known phenomena. If you don't feel anything after a bug in your code cost your company lots of money and your team members many days in debugging and fixing it... then you're like nobody I've ever worked with.

Really? My first major software release took the AOL mail system down for 34 hours. My later work kept it up for 8 years. I consider both equal fodder for bragging, self-immolation, and war stories over drinks. I couldn't tell you which taught me more.

I think what the previous commenters are trying to say is that you shouldn't be OK with producing lots of bugs. Sure, you can learn a lot when you make a mistake, but you shouldn't be making the same mistakes.

I think the bragging part of the bug that brings the system down for hours and requires significant efforts in RCA is that the initial work was important and the bug likely to be non-trivial. Non-trivial bugs happen. I will not be proud of small stupid bugs I make but non-trivial bugs, the discovery of which brings sort-of an excitement - that is something I can be proud of.

That sounds like you feel something about both of those things.

You're right. I feel sheepish and proud and part of a bigger effort, but I do not feel nothing.

I feel shame over bugs. Of course, it depends on the nature of the bug, but there are definite moments of deep self reflection, which I think that I internalize as shame. While I reflect and move on, as a professional, I should be making as few mistakes as possible. When a star hockey player screws up a play that could have won their team the game, I imagine that they feel shame - not that they will get fired for the incident, but self relection is how you improve, and in many people, that manifests as a healthy amount of shame. (Not that I'm the Gretzky of software development).

Recently, a piece of code that I wrote crashed after running for over a month because I forgot to lock one of several methods that was concurrently accessing a collection. I was very ashamed of this - this was an amateur mistake, and one that someone with the amount of experience that I had should not have made. I didn't get any any trouble for this, but I was pretty ashamed. "I am better than this, and I should not be making this sort of mistake," I thought and have been very careful to avoid this sort of thing since.

I'm not dogging people who feel bad over bugs, I just don't feel that way myself. I've never written a bug-free piece of software, so I just accept that bugs are part of the deal. When I screw up, I just fix it. On the bad ones, I feel a tinge of regret (as in "oops, my bad. Let me help correct that.") if I inconvenienced anyone else, but nothing I would call shame (which to me implies internalizing negative emotions about one's self).

I don't write perfect software, and I never will. I'm choosing to not feel bad about that. It's all matter of perspective and expectations, isn't it? My colleagues can think less of me if they like - that's their baggage, though, and I won't pick it up. Shame is dangerous, and leads to a lot of the communications errors that developers make, which are often worse than the bug itself, or even the cause of it.

That sort of sounds like how Morbidity and Mortality conferences at hospitals are supposed to work. It isn't about assigning blame(unless someone really screwed up) but examining the mistake in front of the whole department so that everybody learns from it.

Right. Ideally, everyone is willing to take responsibility for the work they do and are proud of it even when it contains flaws.

If you can't handle having bugs in your code without getting all defensive, you probably shouldn't be coding. The only way to get better is to get feedback.

My company, LiveOps, does the same thing. Every customer impacting issue gets a post-mortem, and some non-customer impacting problems get one.

We have good stability (we hit four 9s recently), but we're always pushing for me. The post-mortem process is very positive and affirming. That's because it's transparent, as it should be.

Interesting. That reminded me of this book "Start-up Nation: The Story of Israel's Economic Miracle" which says their military runs the same procedure in a similar way after missions.

As an Israeli, I can vouch for the massive ass-covering that goes on there. In fact, we just had a big public ass-covering demonstration regarding the awful flotilla blockade. Regardless of what one thinks of the political aspects and/or morality of that action, the purpose was to examine why were several soldiers injured and Israel's relationship with Turkey, previously a strategic ally, deteriorated. Both these results were seen as failures of the mission. The ass covering was plain for all to see, on the national news. I agree that the Israeli army has a very substantial contribution to the high level of entrepreneurship and technological achievement in the country, but effective organizational behavior is definitely not something you want to pick up from the IDF.

> effective organizational behavior is definitely not something you want to pick up from the IDF.

(1) The IDF is "ineffective" on an absolute scale. (2) The IDF is significantly more effective than most other organizations.

While better isn't perfect, you can learn from both.

It's also a lot easier for large organizations to examine issues of a less sensitive nature. AKA missile detonates in the middle of a field vs school.

Why do you claim that the IDF is more effective than other organizations?

http://www.globalfirepower.com/ Compare that list to the population size of each country, Israel is a very large outlier.

National will is important, but you don't rank on that list by being ineffective.

> Why do you claim that the IDF is more effective than other organizations?

I didn't claim that the IDF was more effective than all organizations, I said that it was more effective than most.

For example, I'd claim that the IDF is more effective than almost every failed startup.... (You can argue that many fail for reasons outside their control, but I'll respond that such vulnerabilities are a failure.)

Almost all armies do this, and they do it really, really well.

As a veteran I can tell you that everything you do is critiqued, and everything your boss does is critiqued, and so on.

At certain levels it's peer review, at others, hierarchtical. The whole point of it all is to increase your chances of survival.

It's a great process, and I've noticed that most military folk are great at speaking plainly and accepting critique, much more so than civilians. I beleive these processes are a huge part of the reason.

I don't know... maybe it's where I work, but the post-mortems I've been to all too often mandate some useless extra layer of review or process on top of an already byzantine change control scheme. Sometimes, "shit happens", and I wish our availability teams appreciated that.

And even if there is justification for additional process, they should let the ops/dev teams come up with ideas/solutions themselves: we know our tools and teams best.

Reply on news.ycombinator.com