The Patriot Missile Failure (2000)
www-users.cse.umn.edu
www-users.cse.umn.edu
Edit: To be clear, this was not just bad code, as that issue was already measured and known, but not communicated well, for one. It was military leadership systems-level issue in addition to a Raytheon/coder/etc. issue.
Software was corrected on February 16.
> On February 21, 1991, the Patriot Project Office sent a message to Patriot users stating that very long run times could cause a shift in the range gate, resulting in the target being offset. The message also said a software change was being sent that would improve the system’s targeting. However, the message did not specify what constitutes very long run times.
So Israel was running their units for 8h, but the US unit kept theirs running more than 10x longer with no idea what was appropriate. Phew.
Then, on Scud-day, February 25, there were _six_ batteries operating around the vicinity, and it so happens that the 100h-uptime Patriot battery was tasked to this missile. (How many scuds flew that day? I'd like to know)
And finally on February 26, the next day, "the modified software, which compensated for the inaccurate time calculation, arrived in Dhahran."
Ouch. :-(
The PDF report is a really nice effort. Things like the table on page 17/print page 15 made it easier to understand the range error situation compared to Patriot battery run time.
Though they were super smart and a rapid typist they were also dyslexic. To make it worse, they would never take responsibility for any mistakes resulting from their failure to double-check the figures entered into the software they needed to run. As a result, we were constantly forced to rerun processes they had worked on because the person never reviewed the parameters they entered into the processing modules before running the jobs. Very simple mistakes like entering 1005 instead of 1500 would cost us a day or more rerunning jobs. It took me weeks to detect the pattern of failures and link them conclusively to this individual since other jobs by other people I supervised also failed but for different reasons.
Once I saw the problem and could easily document the issue I took a few moments to sit with them privately to show what I had found and made the recommendation that they should double-check everything they typed before submitting the jobs so that we could eliminate that class of failures. This person was adamant that I had to be wrong and that they were a perfect typist and somehow someone else was responsible.
Try as I might I could not get them to adopt a simple review-before-submit plan so, since they were a valuable team member who didn't need to lose their job during a period where good jobs were not easy to find, after consulting with my superiors it became my job to review everything they submitted, correct any input parameter fat-fingers or dyslexic inputs, and resubmit those jobs so that we could eliminate those failures.
It was easy enough to do so we made it work and they were never informed I was bird-dogging their work so closely. They remained a valued member of the team until one day they put in their notice.
They had been applying for any positions anywhere that would allow them to use their programming skills on problems that a physicist might find challenging. They had been offered and had accepted a position as a programmer with Raytheon working on missile guidance systems. After their exit, we joked that one day we would hear about a fat-finger mistake sending a missile to an unintended location.
This person was not connected at all to this event since their work with Raytheon was several years after this event occurred (SSC shuttered in 1993). I have no idea what they worked on while there but I hope that someone noticed the issue and insured that two sets of eyes hit everything they typed. Those were the days.
I might be an outlier here, but I absolutely think that makes a shit developer. No amount of code knowledge is worth hiring someone that won't work with others. I absolutely hate the trope of the brilliant genius coder who has to be left alone. The reality of that is exactly what you're describing.
The answer is no. I agree with you and there is a chance that I could've been less generous in my description.
I think they were ashamed to be confronted since they had a more advanced education and experience level than most of the rest of the team. They worked well with everyone, would gladly help others improve skills, had a great personality and sense of humor but their main defect was the unwillingness to accept the fact of their own fallibility, causing production lags and extra work fixing problems that shouldn't have happened. We handled it and it all worked out in the end.
Maybe you’re a better composer or producer then, but it’s on whoever leads the group of musicians to organize that.
Jerks who randomly push buggy code straight to master might be the reason you have a job. Most of them probably aren’t, but some of them are the reason a bunch of other people have a job running around fixing stuff and being mad about it.
It’s really hard to see yourself as a regulating component in some messy system, but nothing is better for velocity than pissed of engineers heroically fixing all the stuff they hate from some other engineer that’s pumping out mostly working high impact stuff left and right.
> but I hope that someone noticed the issue and insured that two sets of eyes hit everything they typed.
I think you mean ‘ensured’.
Yes. That is the correct word.
It's funny to me to make this grammatical error in a wordy post about a dyslexic person who fat-fingered things. The character differences, "i" versus "e", between the two words are both typed by the middle fingers of opposite hands.
Your post shows the value of having a second set of eyes on your work. Thanks.
I’m consistently disappointed to see what I have done. Having worked at places where the solution was to tell people off when they made errors, that doesn’t work.
People then just watched their manager make the same error and took pleasure from the situation.
I worked with a couple of people who would regularly see more than 90% of their work rejected for processing errors that were easily recognized if they had bothered to look at their own work but they consistently missed these things and suffered because they knew that I was waiting to look at it and if there was a problem they would know about it soon enough. They did no QC at all even though this regularly set them and the project back sometimes by weeks.
I would make a point to highlight the problem, describe the source of the problem and detail how to prevent it in the future yet inevitably I would see the same issues all over again from the same people. There was no effort to use any of the QC processes or tools that I set up for them so that errors could be minimized.
One person's position was put in jeopardy when they fell so far behind that a big client had to be informed that deadlines would slip. Only then did that person catch me in the hall and ask me to show them exactly how I checked their data, what I looked for, and how they could fix their processes so that their data passed QC. I ended up teaching that person all the things that their job required anyway and they became one of the best performers in the group.
I was not hard to reach, difficult to work with, unwilling to teach or assist - people were just unwilling to ask for help since it could imply a weakness in their skill sets and put some of their other work under closer scrutiny.
It was a depressing job overall since I always felt like I was the bad guy whose only purpose was to point out all the defects in someone else's hard work.
With the final data products they were producing becoming the basis for a client's decision to spend multiple millions of dollars it was important to me to get things right.
Good luck with your checklist.
Raytheon people are all over the place there, to the point I bumped into them on every drone industry event I been (3 in total)
>Second, there is no technical evidence that a single TEL was actually destroyed during the war, despite the claims of some 100 “kills” by aircrews and special forces. [...] The exact impact of coalition operations against mobile systems is more problematic. Iraqi launch operations never stopped and only diminished somewhat over time, although during the last week of the war launch operations increased in tempo. At best, it can be said that counter-Scud efforts only maintained “pressure” on Iraqi missile operations and that Scud CAP operations apparently were successful at harassing but never halting Iraqi launch operations.
https://media.defense.gov/2019/Apr/11/2002115481/-1/-1/0/15C...
Everyone calling for war with Russia seems to ignore the fact that the Patriot batteries that are 'simply defending a NATO members territory' are switched to attack weapons with a simple software change, meaning that the difference between safety and catastrophe is, as always with the Patriot technology, just a matter of software ..
[1] https://en.wikipedia.org/wiki/Aegis_Ballistic_Missile_Defens...
This list needs to include Russia themselves, who've been ramping up the hostile internal propaganda and moving their troops to the front.
Compare that to a patriot which has a different purpose and was optimized to carry almost nothing.
One can wipe out a large area, the other is like a large arrow... might not even penetrate a tanks armor?
You said about Russian military on Ukraine border, there was: “Nothing at all wrong with that, one bit.”
I’m not sure where you fell off, but if you ever considered yourself anti-war, I’m here to tell your younger self that you’ve been led far far astray…
Yes, I consider NATO an aggressor, and it has finally accomplished the breech in European peace that has been planned since 2008.
Here, I'll spell it out for you:
Russian military in their borders: okay.
Russian military in Kiev: not okay.
American military in Poland, Lithuania, Latvia, Turkey, Greece, Croatia, Hungary, Slovakia, Czech-Republic: not okay.
American military in Iraq, Syria, Afghanistan, Pakistan, Somalia, Yemen: not okay.
Get the picture? If so, please color in red crayon the states above that are currently on fire, emitting endless streams of war refugees, and press hard if those states have been that way since the Western coalition started WW3 in 2003 .. speaking of 'being led astray'.
Side-note: search on youtube is a mess. 'For you' and 'Previously watched' collectively took up 16 of the 28 videos displayed on the first page of results, not including '+3 MORE' and '+6 MORE' messages.
Also any automated threat detection system that looks at satellite photos / night vision camera feeds / radar / sonar and misinterprets enemy military equipment for something innocuous or vice versa could be really bad. Especially if weapons are used on the targets on the basis of the system's recommendation.
The most terrifying point in all of this is that multiple systems were all coming up with confirmation that the attack is really happening, he had his entire department checking and checking and checking everything over, and it was coming back as 100% positive.
I don't think Soviets ever publicly said what the issue actually was, we can only guess.
https://en.wikipedia.org/wiki/1983_Soviet_nuclear_false_alar...
Strange definition of velocity!
Glad there was a failure so more R&D could be done to improve.