Hacker News Reader

Comment by tgb | Hacker News Reader

The most impressive thing I've read about NASA's software development isn't mentioned here. It was the system where bugs and defects were (at least in principle) not treated as the failing of an individual but rather as a failing of the entire development process. So when a problem is found, the question isn't "Who did this and how can we punish them?" it's "What can we as a team do so that this never happens again?"

I don't know how this works in practice, but it sounds like a great system even when dealing with code in a lower-risk setting.

View on HN

As a software person at NASA, one caveat to note is that this is in part due to the intricacies of the federal civil servant system. Frankly, it is very hard to punish or fire government employees or some prime contractors, so most often (even in non-software contexts) accountability is applied to a process rather than to an individual because it's a path of less resistance for management. Even when an individual approach would be more appropriate.

The other nuance here is focusing on process gives you a place to put less capable people that you're stuck with. You can create tightly defined QA tasks that are easily verified, give people a checklist, and keep them away from places where they can break things.

This is the standard in manufacturing techniques. A failure in quality control is considered failure of the system for allowing it happen in the first place, not the person making the mistake.

See: http://en.wikipedia.org/wiki/Poka-yoke

If a bug got through, and you don't test. Then obviously its because you don't test. Not because the programmer isn't superhuman, and doesn't make mistakes.

Everybody makes mistakes, design the system to handle it.

Related: root cause analysis - http://en.wikipedia.org/wiki/5_Whys

More on (no) root causes: http://www.kitchensoap.com/2012/02/10/each-necessary-but-onl...

This ... is how it should be? Is this not obvious to everyone?

When push comes to shove, it doesn't matter who made a mistake, all that matters is that things didn't work. And that's all you really care about. The result.

People make mistakes. But systems let mistakes into production.

Bizarrely, I've found that most people agree this is obviously the better solution. And I have never seen it in practice (manufacturing/industry - not the shiney robotic kind, either). People often claim to want to fix the process, but it gets political fast, and because processes/systems are hard to engineer, blaming someone is just faster, easier, and cheaper.

Which is awful.

The stakes are different. If pushing an error to production means that an email doesn't go out or a $500 sale goes through, you have limited motivation, as the benefit of pushing out code fast is greater than pushing it out right.

If you have a billion dollar program where a failure will kill people and create a political storm that will get your management chain fired or sidelined, you can justify spending $100M on the process (ie lots of people + less ability) to push quality.

This is just an attribute of good leadership. It's applicable to teams in any context, and can't be enforced except by leaders who themselves are led by strong leaders.

The technique is basically an after action debriefing. In settings where people die from mistakes such as military operations and space flight, a political process which seeks to assign blame to individuals is often of little utility.

The actions of the deceased often contribute to their death. The only lesson from assigning blame to the pilot for pilot-error is "Try not to make mistakes". Placing the system at the center of the investigation makes for better checklists that reduce the probability of pilot error for many pilots.

"Don't wear red and march in a straight line" is a better tactic than "Don't get shot."

Actually, the use for that doesn't seem restricted to software in general. It seems like you could apply that to any team and get better results - for example, if a big sales deal falls through, don't punish the sales person, use the situation as a learning experience.

I try to do this with my team as much as possible. I am limited by my superiors, who don't hold the same beliefs about maintaining morale and intrinsic motivation.

It is also known as kaizen, with the Toyota Production System[1] as the best known example.

[1] https://en.wikipedia.org/wiki/Toyota_Production_System

I really can't think of a situation where it wouldn't work. Maybe if the codebase is extremely modular and each developer works mostly alone? Even then they should probably still have things like code reviews.

This, in theory, is what "QA" is. What everyone actually does, and calls QA, is usually just QC. Hardly anyone does QA right. Except NASA.

egoless programming

Blameless postmortems work phenomenally at Etsy which is a pretty low risk setting (after reading Sidney Dekker's book The Field Guide to Understanding Human Error [highly recommended] I would say that blameless post-mortems are even more important in a high-risk setting). Except failure isn't the correct word - the book makes the case that these are natural artefacts of complex systems.

1: https://codeascraft.com/2012/05/22/blameless-postmortems/

2: http://www.amazon.com/Field-Guide-Understanding-Human-Error/...

Nice article. Something it doesn't point out, which can become one of the elements in the cycle of name/blame/shame, is the creation of 'heroes'. This is the figure who throws themselves at the issue and brings it back from the brink of disaster. If the culture isn't fixed and failure easy to talk about, then this can play out repeatedly. Since it's usually a celebrated event ('we survived!'), it creates poor incentives for fixing the culture. One can also play this game by noticing a potential failure mode, figuring out how to fix it, but only leaping in once it's struck. I'd rather find another employer though.

We do this to pretty good success at the Big 4 I work at as well. The process by which these post-morterms get assigned can occasionally get political -- e.g. manager A dislikes or is trying to make a move on manager B so he pushes for a detailed post-mortem for every little issue that B's team has -- but the post-mortems themselves tend to work very well in exposing and correcting process failures without placing individual blame.

Reply on news.ycombinator.com