Comment by themodelplumber | Hacker News Reader

Comment by themodelplumber | Hacker News Reader

Parent Full thread

themodelplumber

Wow, if you look at the PDF there were issues all over the place, not just in the binary representation. Patriot battery operators could have easily worked around the computer math issues in particular, but various leadership-system-level mistakes were made, and then procedural details added up on top of that.

Edit: To be clear, this was not just bad code, as that issue was already measured and known, but not communicated well, for one. It was military leadership systems-level issue in addition to a Raytheon/coder/etc. issue.

Software was corrected on February 16.

> On February 21, 1991, the Patriot Project Office sent a message to Patriot users stating that very long run times could cause a shift in the range gate, resulting in the target being offset. The message also said a software change was being sent that would improve the system’s targeting. However, the message did not specify what constitutes very long run times.

So Israel was running their units for 8h, but the US unit kept theirs running more than 10x longer with no idea what was appropriate. Phew.

Then, on Scud-day, February 25, there were _six_ batteries operating around the vicinity, and it so happens that the 100h-uptime Patriot battery was tasked to this missile. (How many scuds flew that day? I'd like to know)

And finally on February 26, the next day, "the modified software, which compensated for the inaccurate time calculation, arrived in Dhahran."

Ouch. :-(

The PDF report is a really nice effort. Things like the table on page 17/print page 15 made it easier to understand the range error situation compared to Patriot battery run time.

I once worked with a person who had an advanced degree in nuclear physics. They were sharp as a tack, a skilled programmer in multiple languages and on several platforms, and grossly over-qualified for the job they had taken but Congress had just killed their position with the SSC in Texas so they needed to switch gears for a while.

Though they were super smart and a rapid typist they were also dyslexic. To make it worse, they would never take responsibility for any mistakes resulting from their failure to double-check the figures entered into the software they needed to run. As a result, we were constantly forced to rerun processes they had worked on because the person never reviewed the parameters they entered into the processing modules before running the jobs. Very simple mistakes like entering 1005 instead of 1500 would cost us a day or more rerunning jobs. It took me weeks to detect the pattern of failures and link them conclusively to this individual since other jobs by other people I supervised also failed but for different reasons.

Once I saw the problem and could easily document the issue I took a few moments to sit with them privately to show what I had found and made the recommendation that they should double-check everything they typed before submitting the jobs so that we could eliminate that class of failures. This person was adamant that I had to be wrong and that they were a perfect typist and somehow someone else was responsible.

Try as I might I could not get them to adopt a simple review-before-submit plan so, since they were a valuable team member who didn't need to lose their job during a period where good jobs were not easy to find, after consulting with my superiors it became my job to review everything they submitted, correct any input parameter fat-fingers or dyslexic inputs, and resubmit those jobs so that we could eliminate those failures.

It was easy enough to do so we made it work and they were never informed I was bird-dogging their work so closely. They remained a valued member of the team until one day they put in their notice.

They had been applying for any positions anywhere that would allow them to use their programming skills on problems that a physicist might find challenging. They had been offered and had accepted a position as a programmer with Raytheon working on missile guidance systems. After their exit, we joked that one day we would hear about a fat-finger mistake sending a missile to an unintended location.

This person was not connected at all to this event since their work with Raytheon was several years after this event occurred (SSC shuttered in 1993). I have no idea what they worked on while there but I hope that someone noticed the issue and insured that two sets of eyes hit everything they typed. Those were the days.

Can you really be a skilled programmer if you refuse code reviews and constantly ship buggy code?

I might be an outlier here, but I absolutely think that makes a shit developer. No amount of code knowledge is worth hiring someone that won't work with others. I absolutely hate the trope of the brilliant genius coder who has to be left alone. The reality of that is exactly what you're describing.

>Can you be a skilled programmer...

The answer is no. I agree with you and there is a chance that I could've been less generous in my description.

I think they were ashamed to be confronted since they had a more advanced education and experience level than most of the rest of the team. They worked well with everyone, would gladly help others improve skills, had a great personality and sense of humor but their main defect was the unwillingness to accept the fact of their own fallibility, causing production lags and extra work fixing problems that shouldn't have happened. We handled it and it all worked out in the end.

Can you be a good musician if you always make mistakes while performing?

Maybe you’re a better composer or producer then, but it’s on whoever leads the group of musicians to organize that.

Jerks who randomly push buggy code straight to master might be the reason you have a job. Most of them probably aren’t, but some of them are the reason a bunch of other people have a job running around fixing stuff and being mad about it.

It’s really hard to see yourself as a regulating component in some messy system, but nothing is better for velocity than pissed of engineers heroically fixing all the stuff they hate from some other engineer that’s pumping out mostly working high impact stuff left and right.

This was a good read, thanks.

> but I hope that someone noticed the issue and insured that two sets of eyes hit everything they typed.

I think you mean ‘ensured’.

>I think you mean ‘ensured’.

Yes. That is the correct word.

It's funny to me to make this grammatical error in a wordy post about a dyslexic person who fat-fingered things. The character differences, "i" versus "e", between the two words are both typed by the middle fingers of opposite hands.

Your post shows the value of having a second set of eyes on your work. Thanks.

As you say, it’s amazing what you find when you review your work. I’m currently trying to implement a checklist system for reviewing work to get output quality up.

I’m consistently disappointed to see what I have done. Having worked at places where the solution was to tell people off when they made errors, that doesn’t work.

People then just watched their manager make the same error and took pleasure from the situation.

In my own career doing QC work I found that so many people would never ask for assistance understanding why their work was rejected. They would simply blame me for noticing their errors. That was literally my job, I was the second set of (trained) eyes.

I worked with a couple of people who would regularly see more than 90% of their work rejected for processing errors that were easily recognized if they had bothered to look at their own work but they consistently missed these things and suffered because they knew that I was waiting to look at it and if there was a problem they would know about it soon enough. They did no QC at all even though this regularly set them and the project back sometimes by weeks.

I would make a point to highlight the problem, describe the source of the problem and detail how to prevent it in the future yet inevitably I would see the same issues all over again from the same people. There was no effort to use any of the QC processes or tools that I set up for them so that errors could be minimized.

One person's position was put in jeopardy when they fell so far behind that a big client had to be informed that deadlines would slip. Only then did that person catch me in the hall and ask me to show them exactly how I checked their data, what I looked for, and how they could fix their processes so that their data passed QC. I ended up teaching that person all the things that their job required anyway and they became one of the best performers in the group.

I was not hard to reach, difficult to work with, unwilling to teach or assist - people were just unwilling to ask for help since it could imply a weakness in their skill sets and put some of their other work under closer scrutiny.

It was a depressing job overall since I always felt like I was the bad guy whose only purpose was to point out all the defects in someone else's hard work.

With the final data products they were producing becoming the basis for a client's decision to spend multiple millions of dollars it was important to me to get things right.

Good luck with your checklist.

Thanks. If nothing else, it has vastly improved my work.

Seems almost criminally negligent.

It's interesting that Raytheon's name is nowhere to be found on this. It sounds like Raytheon's bad code killed 28 US soldiers.

The Iraqi SCUD killed 28 US soldiers. Raytheon failed to prevent it.

Yes, the Iraqi scud was the primary factor, but it almost seems like if your seat belt snaps in a car accident with a drunk driver and you go through the windshield and die. The drunk driver is culpable but the car company doesn't get a free pass.

Due to a code defect in their product*

Missile interceptors are incredibly complicated. Even with perfect code there is a probability range for successful intercept. The probability was much lower in this case as the software had a defect, and the battery was online for 100h.

That was being used outside of its original design specifications.

Reply on news.ycombinator.com