Comment by jevinskie | Hacker News Reader

Comment by jevinskie | Hacker News Reader

Parent Full thread

"We informed the foam technicians at our plant in Michoud Louisiana that they were the cause of the loss of Columbia..."

That to me is pretty disgusting. In an incident like the loss of Columbia, there is no one, true "root cause". To assign blame to those foam technicians was disingenuous and just another instance of "passing the buck" that seems to happen so often in the post-mortem of NASA failures. NASA knew of earlier foam strikes (STS-112) yet chose to continue flying without diagnosing the problem. Even during the tragic STS-107 flight, engineers knew of foam strikes and their concerns were ignored. Even though they would have been almost completely powerless to remedy the situation on STS-107, the higher-ups decided to continue on with the mission instead of addressing the concerns with the heat shields. The article author states later in the article that he apologized to the foam technicians. Commendable, but I am still bothered by the fact that NASA was initially so eager to place the blame on a single contractor instead of owning up to their own culpability. Leadership and responsibility needs to come from the top, especially in such a prestigious organization!

Understand the type of organization NASA is. First, accountability is everything. It's not that there are big internal political struggles (there are), but more importantly accountability is required for high safety. We aren't talking about a 10 person startup where a bad commit to production takes down a site for a few hours. We're talking about people's lives, careers, and the safety of millions of people that could be negatively affected by a crash or explosion.

When he talks about it being their fault, it's not that those engineers are being singled out for punishment and derision. They had to find out where the problem existed that led to the loss of Columbia, and after extremely thorough testing they believed it existed with the foam team. It's simply a matter of finding where a problem is and doing everything you can to fix it.

So it's not a personal, vindictive "your fault", its an impersonal "the problem is here, lets fix it".

> accountability is required for high safety

It's not. That's complete bullshit, and the On-Board Software Group demonstrated it by being as flawless as can be during the whole history of the Shuttle: as far as I know there was no personal accountability in the OSG, the only thing accountable was The Process supported by a strong culture of adversarial testing.

Personal accountability in such a system brings politics and career advancement in focus and leads to issues being shoved under the rug when inconvenient and energy being expended in finger-pointing and blame games rather than fixing problems.

> So it's not a personal, vindictive "your fault", its an impersonal "the problem is here, lets fix it".

No, it's not. Accountability is very precisely "your fault", that's all it is. That's pretty much the definition of it.

There may not have appeared to have been any personal accountability, but make no mistake if anything had gone wrong with the On-Board Software the EXACT same accountability process would have been initiated to ensure that the same human error did not occur for a second time.

Very unlikely, unless that had been imposed on the group by an external, blame-oriented entity.

Left to its own devices, the group would most certainly have operated as it did every time it found fault in its output: find out how The Process had allowed for a fault to be introduced and reach output, find out how to make The Process prevent the introduction and/or release of such faults, fix The Process.

So no, the "exact same accountability process" would most definitely not have been initiated within the group, a very different one would have taken place.

> No, it's not. Accountability is very precisely "your fault", that's all it is. That's pretty much the definition of it.

That. Accountability (worse: "personal accountability" and variants thereof) are a tool provided by the law to determine who to recover damages from after a failure has occurred. It is entirely unsuitable for failure prevention because it is entirely orthogonal to rigorous testing and a culture of workplace safety.

Anyone who insists on accountability on their project does not know what they need. What they get is an extraordinary amount of ass-covering, finger-pointing and blame deflection, though.

That's a really good summary of mission critical systems accountability problem.

As additional color here is what he posted in a reply to a reply of his blog entry:

Perhaps I was a little too brief in my writeup. Nobody that I know (certainly not me) went to MAF and downdressed any of the workers. What happened was that the conclusion was reached in engineering and management meetings and the word filtered out to the workers that poor workmanship was the proximate cause of the loss of Columbia – as if there wasn’t enough blame to go around in many other areas. I really regret the erroneous conclusion, the impact it made on the workers, and the way the whole scenario played out at MAF. The people there were very hard working, dedicated, and proud of their involvement with America’s space program, many of them second or third generation workers at that location. Now, of course, they have all been laid off and the MAF plant is virtually a ghost town with very limited work going on there for other NASA or commercial projects.

That was my assumption by the end of the story. The bluntness of the explanation was more for effect or maybe how things were eventually interpreted by the workers, not what was actually said.

> "We informed the foam technicians at our plant in Michoud Louisiana that they were the cause of the loss of Columbia..."

This is stupid, perhaps wilfully so. The fact that foam can come off the external tank and strike critical parts of the shuttle is a fatal design flaw. Insulation has been used on rockets for sixty years now, and it has been observed to come loose every now an then. The difference is this: on a rocket it just falls off, but if any comes off the tank and strikes the Shuttle orbiter, there is disaster.

The fault was at an early stage of design.

I'm just impressed he tells the story so honestly. He could have characterized what was said differently.

Yes, I recoiled when he said the line quoted above.

But I think this direct wording was a rhetorical device to set the hook for the pivot in the story, when it turned out the problem with the foam wasn't inclusions during installation at all.

And I wonder if there is a bias in his telling... that he words it harsher than it was presented in reality out of a sense of guilt.

To me, it's pretty refreshing. Leadership and responsibility should certainly come from the top and any public statement about this should make it clear that the leaders take responsiblity. Direct and clear communication about all components that you think contributed to the failure is just as critical. Saying "we failed as an organization" is not helpful, and saying "I failed as a leader" is nice but not informative. Failures can have root and contributing causes; organizations may have many shortcomings; calling them out in order of importance (as best you can determine it) and emphasizing what must be fixed is the only way to remove a poisonous kind of uncertainty that lingers in a situation like this.

The problem with the investigation into foam loss on STS-107 was that when they tested small sections of foam, it was free of material defects. At the time, investigators concluded incorrectly, by exclusion, that it must have been faulty installation that caused the loss of the shuttle. NASA management blamed those workers for the loss of Columbia. When Discovery's External Tank had cracks and found it was cracking from thermal stresses. This meant that the blaming it on faulty installation was wrong, and for which Wayne Hale apologized.

Like you, I felt some serious disgust at this line given all the other people who contributed to the problem, but also very worried that innocent people were fired, suffered long-term grief, or committed suicide. You better be damn sure before you tell people at the bottom that it is their fault and it better be for their personal incompetence and not a failure of process or design that they have no control over.

I got sick to my stomach when I read that.

Reply on news.ycombinator.com