Hacker News Reader

Comment by baldfat | Hacker News Reader

> According to Russia Today, the person responsible for the hole has been identified.

That was pretty scary sentence for me. Short story, I know the guy who made the mistake on the mirror for the Hubble Telescope. https://www.nasa.gov/content/hubbles-mirror-flaw

He was a great guy and was the President for the Trouts Unlimited when I was a kid. That one thing defined him for so long. Was pretty sad story to me. He did one mistake but the quality control department were never troubled like he was even though they had it right in front of them for years.

View on HN

"An unnamed source told RIA Novosti the person responsible for drilling the hole filled it with glue instead of reporting the mistake."

A mistake is one thing. Mistakes are lessons and it's good to have people who have some solid lessons in their backgrounds.

Trying to patch it with some glue without telling anyone is a character flaw and the person needs to go, so everyone else gets the lesson.

True, but apparently still a quality control should be in place to find such problems.

It's precisely this kind of bodging which caused great loss of life with the de Haviland Comet. Installing the radio antenna opening with punch rivets instead of drilling out the holes ended up killing two planeloads of people.

https://en.wikipedia.org/wiki/De_Havilland_Comet

Ya. That's how things operate at my work and it works well. As much as it sucks mistakes happen, things get damaged, sometimes things worth lots of money. If feels really shitty to have to go tell someone you made a mistake, that a bunch of time got wasted, we're going to have to redo work, that you just destroyed a couple thousand dollars worth of things. It really sucks...

But... it sucks a whole lot less than when someone tries to cover up or hide a mistake. It always ends up coming out in the end, it always makes things worse than they could have been in the end and everyone ends up even more pissed off than they would have been. It's not worth it.

Actually, my parents forced me to learn this lesson young. Me and two other kids did something fairly terrible when I was in elementary school. My parents took me and made me personally apologize to the person who detrimented from our actions. It wasn't the most pleasant of experiences but in the end t was the right thing to do and it taught me, if you fuck up, the best thing you can do is admit it, try and make it right and learn from it.

I've been trying to teach the new guy at my work this. He started off worried about making mistakes, would beat himself up and try to cover them up. It's been a few months of talking to him and telling him to focus on learning and not worrying about mistakes. If they happen, I get him to work through what went wrong before anything too bad happens and i've found his attitude has changed quite a bit since he first started. He doesn't worry so much and he's been learning a lot more quickly. When something does happen, instead of panicking he just comes to tell me and i'll help him work though figuring it out.

We're all human, mistakes happen. They just need to be dealt with in a reasonable way and if you make a mistake admit it and if you can't figure out how to fix the problem there's no shame in asking for help.

Kudos for creating a work culture where it's easier to make good choices that lead to better character.

That's easy to say from our position, but I'd bet that the person responsible had a pretty good idea what would happen if they reported it.

At my current machine shop, I'd have a half dozen people documenting what happened, helping to design, implement, and rush out the repaired part, adjusting customer expectations, and making sure that the process that allowed me to make the mistake is fixed so that others repeat it.

In my previous job, it depended on the day. Sometimes you'd get called into the boss' office and cussed at for a while with no consequences, sometimes you'd get something capricious like confiscating the offenders' drill bits and requiring check-out for each hole from the foreman's office for a week. (It only lasted 3 days, but that literally happened.) Sometimes the boss was having a really bad day with customers or vendors he couldn't punish, and you could have a half dozen guys bodging a repair out of the boss' view and keeping him distracted. Or fired on the spot, if it didn't work. I lasted 5 years and 37 hires and fires at that 20-man shop, should have left so much sooner.

The official story is:

> It is a technological error by a specialist. It was done by a human hand -- there are traces of a drill sliding along the surface. - http://tass.com/science/1019791

Drills should not be sliding along the surface of a part. You locate the hole by marking with calipers from a known location and hit the intersection with a punch. Especially in curvy aluminum aerospace work. And there's certainly little opportunity for anyone to go get some glue, install it, wait for it to cure, and paint/cover it up without inspection. I am surprised, though, that they didn't just weld in some filler and grind it smooth.

I suspect that NASA and some of their suppliers are nominally more like the former but practically there's still some risk to admitting error, and that Roscosmos/Energia is closer to the latter.

>You locate the hole by marking with calipers from a known location and hit the intersection with a punch.

Okay, and now do that in microgravity in a spacesuit with the thick gloves. Good luck.

this wasn't drilled in space....

Where did they have to repair it when the plug came loose and began leaking again?

It said the astronauts used an epoxy to plug it, but I don't think it required a space walk. I assume it was doable from inside.

> Drills should not be sliding along the surface of a part.

https://cdni.rt.com/files/2018.09/article/5b8d416cfc7e932e72...

"Another source said a worker apparently accidentally drilled the hole, but instead of reporting it, simply sealed it. The sealant held for at least the two months the Soyuz spacecraft spent in orbit, before finally drying up and being pushed out of the hole by air pressure."

This American Life had an absolutely amazing story about how a dropped wrench in a nuclear missile silo caused a cascading series of failures that ended in utter disaster but could have been even worse: https://www.thisamericanlife.org/634/human-error-in-volatile...

It just goes through the story in bone chilling detail and shows how the natural human desire to cover up their mistakes is absolutely anathema to safety culture.

Yea, accidents and mistakes should obviously be seen as learning opportunities, and the ones making them should really be treated as heroes for demonstrating how things can go wrong. But when millions (billions?) of dollars and/or lives are put at risk you better have a good explanation, that's for sure. Putting glue in an accidental hole of a space vehicle or similar and hoping no-one discovers would probably not help your case further (if it were to be discovered ...).

The guy who miscalibrated the machinery that made the Hubble telescope (1/50th of a hairs breadth error) may have had a good explanation for all I know- care to share the details?

My grandfather worked on the HST mirror. It's both what he was most proud of, and the most ashamed of, at the same time.

He talked about this not really as a single person's fault (except in terms of a senior manager's failure), but as an organizational problem. He pointed out two things:

1. In particular, the organization was poorly designed, with QA being too close (in org chart terms) to production. If there's one finger to point, it's at the guy who was the common manager of both departments. And I think this is something that the engineering profession has taken to heart since then.

2. People at the time thought of things with somewhat different priorities because the context changed. At the time the mirror was being built, it was understood that the space shuttle would make it relatively simpler to fix serious problems (well, as "simpler" as any space missions could be). As it turned out, the space shuttle was less effective in that role than its initial requirements led HST management to believe. This probably led to a more relaxed attitude about the ultimate result of any problems.

> In particular, the organization was poorly designed, with QA being too close (in org chart terms) to production.

I see this with great regularity in my practice and it is really strange that this lesson has to be learned all over time and again. Latest incarnation: make developers responsible for their own QA. That's about as logical as asking the butcher to grade their own meat.

> QA being too close (in org chart terms) to production.

Can you explain in a bit more detail what this means in practice? I haven't worked with a QA team before and I'm curious.

Does it just mean the QA checks happen well after the specific work was being done, making it harder to go back and fix things (due to mental context switching, tooling, etc)? Or just too many last-minute fixes?

My positive experiences with QA were always when the QA team was strictly separated from the production team.

Ideally, no friendship whatsoever.

Doing QA is hard. Bouncing issues & QA reports between teams is hard. Bouncing issues between people that know each others too well or have a common manager leads to all sorts of issues.

In this specific instance, it could be as simple as the manager trying to balance things between teams and thus letting things slip past the due date or with sub-par quality.

My worst experience was when a married couple worked together on a project I was involved with: one on the sw team and the other in QA. Working together is hard, working on "antagonistic" teams is hell

> Latest incarnation: make developers responsible for their own QA. That's about as logical as asking the butcher to grade their own meat.

No, you're wrong. Self certification works when it's done properly. As with any QA system it fails if it's done poorly, but any QA will fail when done poorly, and self certification is less likely to fail.

I built safety critical equipment for coal mining, and some aerospace test equipment, and saw a variety of QA systems from very good (MTBF 14 years) and piss poor (product broken when leaving the factory).

Coal mining and aerospace have a completely different attitude to safety to begin with. Maybe my 'sample size' is too small but I'd figure with close to 100 dd's under my belt if DIY QA would work I would have seen it by now.

The only place where it sort-of worked that I looked at is where there still was another QA dept further downstream.

Yup, I knew the person who was the supervisor. From what I know it was one piece that was attached to the grinder that was actually put in upside down. No one noticed it and it wasn't labeled. It took 10 Years and even had extra time to see or fix any problems. Here is a good read - http://www.scienceclarified.com/scitech/Telescopes/Hubble.ht...

The person who put it in wrong didn't get in trouble. The manufacturer of the part that didn't label the part didn't get in trouble. The upper management didn't get in trouble. The Quality Assurance didn't either. Just seemed stupid where the blame sits.

"This probably led to a more relaxed attitude about the ultimate result of any problems."

had a former co-worker who used to say as a general rule, the quality of a code drop was inversely proportionally to the ease of updating the code.

IIRC a calibration rod had some paint missing from one end, which affected measurements for grinding.

Well I heard that it was actually installed upside down. Well at least that was what the local rumors were.

I always wondered how this happened, given that that the HST was far from a "one-off" product - would love to know if this was an error that affected all of the KH-11 mirrors up to that point (and went undetected), or if there was something about the Hubble mirror that was different?

Unfortunately, that information is probably still classified.

The focal length of the Hubble was certainly different, but maybe other things too... I'd be surprised if this wasn't a copy/paste error, but maybe it really was some huge miscalculation that nobody caught.

There were a lot of KHs (Keyhole) flying long before Hubble so they didn't have unfixed issues and their test fixtures were all set up (and frankly, probably passed Hubble).

https://en.wikipedia.org/wiki/KH-11_Kennen

The mirror was too flat in the middle and there was a slight wave. It was a mechanical issue when the polished the mirror starting in 1979 for over a year. It wasn't launched till 1990.

They state that the error in the mirror if you enlarged to to the size of the earth would not be any higher than 6 inches.

http://www.scienceclarified.com/scitech/Telescopes/Hubble.ht... is a great article on it. Since this all happened in my area I was always interested. The fix of the mirror is one of the greatest feats I have ever read. They weren't sure how much could be fixed. It ended up being better than anyone could ever expect.

Reply on news.ycombinator.com