Comment by probablypower | Hacker News Reader

Comment by probablypower | Hacker News Reader

> "A Single Line of Code Brought Down a Half-Billion Euro Rocket Launch"

Blaming a system failure on a single point like this dooms that system to repeat similar failures (albeit in another element) in the future.

There are numerous testing, quality and risk controls that could've been in place. There are probably even a few people who didn't do their job (besides the one person a decade ago who wrote the 'single line'). The point isn't to pin blame on any one point, but to look at the system (people, processes, technology) and try to understand why the system is fragile enough that a single person's error is able to escalate into a half-billion euro error.

By focusing in on the point of failure, you end up falling victim to survivorship bias [0]. It is how you end up with developer teams swamped with unit-testing requirements and test coverage metrics, but still somehow end up with errors that impact the end-user anyway. It is how you get company surveys that always seem to miss the point, saying that the measures they implemented to improve company culture worked, yet everyone is burning out and miserable.

[0] - https://en.wikipedia.org/wiki/Survivorship_bias

They're not focusing on one line of code, they cover the failover systems that also failed as well. It's also a mistake to try and fix bad tools, languages and programming practices with higher level processes. Just use better tools that do bounds checking (and unit checking which has also caused failures), preferably checked at compile-time and the problem is fixed without all the rigmarole you describe.

Also remember this happened in the late 90s which means the code was written in the 80s.

sounds like you're saying that being unable to write in-bounds code is the single-point-of-logic failure for a coder, and if you correct that part of the bad tool, all their other algos will be great...

I think people who can write in-bounds code and type correct code with no safety rails have a leg up to write really good code.

A bounds check wouldn't have helped. The value would have saturated instead of rolling over, resulting in a similar failure.

The mistake was an incorrect specification. A programming tool can't identify that you've made the wrong thing, which is why we need "rigamarole" to validate the spec. That's what the systems engineers are for.

Saturation might have worked, actually.

> the main computer interprets the data as real navigation data and considers it as an indication that the rocket is wildly off-course

Not clear what sort of magnitudes we are talking about but saturation could have worked here and avoided the problem.

But an exception handler could've helped too.

> The code wasn’t necessary after takeoff, it was only part of the launch pad alignment process.

A supervisor for this task could have decided to ignore an overflow fault after launch.

Saturation would have been fine; actually anything would have been fine, since the result was not actually used in flight. However, Ada traps, and the trap was not handled (because resources were tight and overflow was physically impossible on the Ariane 4, for which the code had been written), and the specification required that the system shut down entirely on an unhandled trap.

True. I should have said "if it was needed".

Had they validated more, they would have realized they made the wrong thing.

According to the article the code was designed to run on the previous iteration of the rocket, the Ariane 4, who's first flight was on 15 June 1988. Conceivable that code was written in the mid-1980's, better 'tooling' might not have been an option.

Not at the time, but those features are available now, and that still isn't the lesson commonly taken from these disasters, unfortunately.

> single line of code

The funny thing is that you could pick and choose to attribute any error to a 'single line of code'

Also, an issue like this going unnoticed points to a lack of proper QA. There were probably a fair few more than a "single line of bad code" that could have fucked the launch of this one didn't do it first.

There is actually a lot of research on how catastrophic failures in highly complex systems happen. Here is a brilliant article that summarizes the main findings:

https://how.complexsystems.fail/

I cannot read that one without thinking in the descriptions and analyses of disasters like the sinking of the MS Titanic, the Chernobyl disaster, the loss of the Challenger space shuttle, or the Fukushima disaster. Many, many points in the article seem correct for all of them.

Reply on news.ycombinator.com