Hacker News Reader

Comment by mizzao | Hacker News Reader

This seems to follow a general pattern in today's devices: expensive hardware ruined by crappy software.

Why is it that good software quality is generally an afterthought in so many systems?

As an ex-hardware engineer, reliability and thorough testing has been drilled into the heads of every single hardware team. Not because we care more about quality, but because of simple economics: finding a single major bug after tape-out is going to be extremely expensive to fix, and will push out your product launch by months. Finding a single major bug after release just might doom your entire company. Hence why reliability and testing is treated like a life-or-death priority.

In software on the other hand, we're spoilt by how easy to is to fix bugs (compiling your code to generate a binary takes minutes/hours, not months, and it doesn't cost you millions of dollars), and also how easy it is to deploy fixes to customers (just release a patch and when someone complains, tell them to download it). Hence why if you're working at Google/Amazon/Apple/99%-of-software-teams, the rational thing to do is to treat reliability/testing as a 2nd class priority, not a 1st class priority.

Except that there's a small number of software applications where this doesn't hold. Where testing really does need to be treated as a 1st class priority. Problem is, the engineers and managers working in these teams belong to an industry that has a completely different mindset. A mindset that translates itself into all parts of the software stack, which these engineers are then forced to rely on. A mindset that will creep into everyone's minds, by osmosis, no matter how much they try to fight it. A mindset that works fine 99% of the time, but every now and then, leads to events exactly like this.

> Problem is, the engineers and managers working in these teams belong to an industry that has a completely different mindset. A mindset that translates itself into all parts of the software stack, which these engineers are then forced to rely on. A mindset that will creep into everyone's minds, by osmosis, no matter how much they try to fight it. A mindset that works fine 99% of the time, but every now and then, leads to events exactly like this.

The famous Worse is Better, aka. 80% solution that isn't even correct (it's fine as long as it doesn't segfault in that 80% of cases) today is better than a proper solution next week.

There's a lot of software out there. Every car, every phone, every plane, computer, TV, washing machine, factory robot, etc - every thing works pretty much flawlessly on a day to day basis. Occasionally things need a reboot or a patch but that's all. The number of times software fails in our daily lives is pretty low considering we interact with it so often. I'd contend that software quality is actually staggeringly high. It could be higher, but there's the law of diminishing returns and all that.

That is not satisfactory. Some instances of software are more crucial than others eg. software running money transfers must have less tolerance for bugs than a game I play on my phone. The question is not whether software works most of the time, but how badly things go wrong when mistakes happen.

Here's an analogy: A weather forecaster can predict "no hurricane" every single day and have a near-perfect success rate. Needless to say, that's next to useless. (False positives and false negatives have wildly different costs in this context.)

At a guess I'd say that the probability of a fault in a piece of software does approximately match the cost of it failing. A phone game is very likely to have more bugs than a banking system. My point is that both applications actually work really well. In my experience premium phones games crash maybe once in every thousand runs. Bank money transfer software crashes perhaps once in every few billion runs (that's a total guess but we'd hear about it if it was higher and there are a lot of bank transfers every day). I think that's quite good.

>Every car, every phone, every plane, computer, TV, washing machine, factory robot, etc - every thing works pretty much flawlessly on a day to day basis.

What is your definition of "pretty much"? In my experience none of those things you mentioned have ever worked pretty much flawlessly.

I'd argue the opposite: software quality is generally about as low as it can possibly be while having any value at all. I do my best to interact with as small a quantity of it as I can manage, since it generally frustrates me so much that my life is better if I continue to do most things manually.

A couple decades of work on compilers, dev tools, drivers, firmware, and other sorts of system software may have warped my perceptions somewhat.

(Edit: I don't say that to pretend some kind of superiority; my code is just as buggy as everyone else's. I've just had many occasions to dig around in other people's code and figure out why it stopped working, after we tried to change something that shouldn't have affected it. Hardware, too, you can't trust that either.)

"Flawlessly" is definitely not the word I'd use. "Adequate" maybe. But just about every piece of software I use day to day is frustrating to interact with, and those that aren't might be 'upgraded' at any second.

Because the hardware folks are king in those shops, not leading to proper respect and care for software. It has to be a yin/yang situation, not a pyramidal one.

It's more that software is becoming a dominant factor in everything. Which is both good and bad. It's bad in that insufficient effort and process is going into ensuring software quality. In a very real sense we're in another "software crisis". Software complexity is growing beyond the bounds of the previous era's "best practices" (of which only a subset were ever actually applied) to manage. While at the same time the impact of software problems is growing, especially because the potential for problems due to interactions of software systems is also growing.

Though in truth that's mostly unrelated to this instance. Real-time control systems for satellites have always been a tough problem, and there have traditionally been growing pains for organizations tackling new designs. Even NASA has had its fair share of stumbles in that regard, for example the loss of the Mars Polar Lander and even the Mars Climate Orbiter (the famous metric/english units fuckup) were substantially due to problems with the software and operational management of the spacecraft.

Every hardware platform has unique abilities and constraints. The code that provides the "business logic" part of the device is often one-off and domain- or application-specific. Even if you're running on some general purpose platform, that part of the code is going to be about as battle-tested as anything running on some new architecture - not much.

Edit:

Someone alse pointed this out but it's worth reiterating - hardware systems are simulated extensively and analyzed in various ways for performance, cost, reliability and other factors. Software less so, and test cases, if they exist, might not necessarily capture all the power loss/inconsistent state problems that can occur in the real world. Maybe in the future embedded applications could be analysed with a fuzzer or something to help weed out that kind of once-in-the-field-never-in-the-lab bug.

It's not exactly under a plain software/hardware dichotomy. Rather, this incident indicates a system design problem - no preplanned failure modes, redundancy, cross checking and thorough testing. Good system design regime should overcome both software and hardware issues, either by detecting the issues in advance or by implementing measures to withstand them during operation.

There's so much that can go wrong in a satellite that expecting software to function well is very naive.

Besides, it's not like testing a satellite is easy as running unit testing on a Python module - there are countless operational and environmental variables that are very difficult to synthesize.

I have failed to see evidence that the software was faulty.

If anything, it tried to do what it was supposed to: stop the satellite from spinning.

From the report:

> Feb 28, JAXA sent the commands to update the RCS control parameter based on the center of mass changes by deployment of the EOB. Through the post-incident investigation, it is confirmed the RCS control parameter on Feb 28th was not appropriate.

So the software worked as designed, but was fed crappy data.

Indeed. And the question of whether we consider software consuming data from a physical device that doesn't correctly account for the possibility of that device feeding it garbage to itself be crappy is a largely academic question.

The whole system has to be considered for success.

Would you say it's easier to spec out hardware than software? I don't work with hardware, but from a layman's perspective it seems like you can (and have to be) utterly exacting when it comes to product matching designs, with said designs themselves being very exacting, i.e. using physics equations.

With complex software, it's pretty dang hard to layout an entire program in specifications without going down the provably correct approach, which likely will run into limitations when the real world is involved. Waterfall dev for operating systems of the past come to mind.... Probably the best you can do is be very stringent with your coding standards, something space agencies are the masters of.

I get the sense that the difference often stems from the nature of software as "words you can run" vs. hardware as "physical manifestation of words."

For quite a bit of software these days, the correct program and the description of the correct program are pretty similar artifacts.

> For quite a bit of software these days, the correct program and the description of the correct program are pretty similar artifacts.

My brain was going mental mobius strip when I was originally responding and thinking of something along these lines. Maybe what I was getting at is that the an idea like the Halting problem doesn't apply as viciously to hardware?

With hardware if you are not careful you can get parts that meet spec, but are useless because you forgot to specify something correctly (eg: referenced the wrong dimension, error stacking, etc). Generally the only reason why hardware would be easier to spec is that the output can be simpler (eg: a plate out of this material with holes in these locations).

Then you get the stuff that lurks in the middle that could be a software, electrical or mechanical problem (or all 3), and that is where the bugs like this occur. I think its more a result of splitting up work into domain silos, so they do not think about problems that arise in the other domains that can be fixed easily in another. This can lead to one or more domains getting no time at all to implement their part of the system as they spent most of the time budget waiting on another domain to finish. Watchdogs and other "things are going crazy" sort of belt&braces protection are the first parts of a design to get dropped when in a time crunch.

One of the reasons is that people in the software field (and in a few others as well) have this urge to look for 'challenging' work, i.e. things that they haven't done a million times before to the point where they've become boring. Which is also why you see a lot of s/w engineers working on untested (anything that can't achieve a year of up-time) clever designs/implementations. Its exciting to be clever, but its boring to be conservative with your design to achieve high reliability.

Reply on news.ycombinator.com