Hacker News Reader

Comment by stareatgoats | Hacker News Reader

Yea, accidents and mistakes should obviously be seen as learning opportunities, and the ones making them should really be treated as heroes for demonstrating how things can go wrong. But when millions (billions?) of dollars and/or lives are put at risk you better have a good explanation, that's for sure. Putting glue in an accidental hole of a space vehicle or similar and hoping no-one discovers would probably not help your case further (if it were to be discovered ...).

The guy who miscalibrated the machinery that made the Hubble telescope (1/50th of a hairs breadth error) may have had a good explanation for all I know- care to share the details?

View on HN

My grandfather worked on the HST mirror. It's both what he was most proud of, and the most ashamed of, at the same time.

He talked about this not really as a single person's fault (except in terms of a senior manager's failure), but as an organizational problem. He pointed out two things:

1. In particular, the organization was poorly designed, with QA being too close (in org chart terms) to production. If there's one finger to point, it's at the guy who was the common manager of both departments. And I think this is something that the engineering profession has taken to heart since then.

2. People at the time thought of things with somewhat different priorities because the context changed. At the time the mirror was being built, it was understood that the space shuttle would make it relatively simpler to fix serious problems (well, as "simpler" as any space missions could be). As it turned out, the space shuttle was less effective in that role than its initial requirements led HST management to believe. This probably led to a more relaxed attitude about the ultimate result of any problems.

> In particular, the organization was poorly designed, with QA being too close (in org chart terms) to production.

I see this with great regularity in my practice and it is really strange that this lesson has to be learned all over time and again. Latest incarnation: make developers responsible for their own QA. That's about as logical as asking the butcher to grade their own meat.

> QA being too close (in org chart terms) to production.

Can you explain in a bit more detail what this means in practice? I haven't worked with a QA team before and I'm curious.

Does it just mean the QA checks happen well after the specific work was being done, making it harder to go back and fix things (due to mental context switching, tooling, etc)? Or just too many last-minute fixes?

My positive experiences with QA were always when the QA team was strictly separated from the production team.

Ideally, no friendship whatsoever.

Doing QA is hard. Bouncing issues & QA reports between teams is hard. Bouncing issues between people that know each others too well or have a common manager leads to all sorts of issues.

In this specific instance, it could be as simple as the manager trying to balance things between teams and thus letting things slip past the due date or with sub-par quality.

My worst experience was when a married couple worked together on a project I was involved with: one on the sw team and the other in QA. Working together is hard, working on "antagonistic" teams is hell

> Latest incarnation: make developers responsible for their own QA. That's about as logical as asking the butcher to grade their own meat.

No, you're wrong. Self certification works when it's done properly. As with any QA system it fails if it's done poorly, but any QA will fail when done poorly, and self certification is less likely to fail.

I built safety critical equipment for coal mining, and some aerospace test equipment, and saw a variety of QA systems from very good (MTBF 14 years) and piss poor (product broken when leaving the factory).

Coal mining and aerospace have a completely different attitude to safety to begin with. Maybe my 'sample size' is too small but I'd figure with close to 100 dd's under my belt if DIY QA would work I would have seen it by now.

The only place where it sort-of worked that I looked at is where there still was another QA dept further downstream.

Yup, I knew the person who was the supervisor. From what I know it was one piece that was attached to the grinder that was actually put in upside down. No one noticed it and it wasn't labeled. It took 10 Years and even had extra time to see or fix any problems. Here is a good read - http://www.scienceclarified.com/scitech/Telescopes/Hubble.ht...

The person who put it in wrong didn't get in trouble. The manufacturer of the part that didn't label the part didn't get in trouble. The upper management didn't get in trouble. The Quality Assurance didn't either. Just seemed stupid where the blame sits.

"This probably led to a more relaxed attitude about the ultimate result of any problems."

had a former co-worker who used to say as a general rule, the quality of a code drop was inversely proportionally to the ease of updating the code.

IIRC a calibration rod had some paint missing from one end, which affected measurements for grinding.

Well I heard that it was actually installed upside down. Well at least that was what the local rumors were.

Reply on news.ycombinator.com