Hacker News Reader

Comment by shellmayr | Hacker News Reader

Wow, how can something like this happen? I thought airplanes had triple redundant software systems using 3-version programming [1] in order to avoid such bugs/problems. Can anyone familiar with flight technology shed some light on this?

[1]http://en.wikipedia.org/wiki/N-version_programming

View on HN

Not an airplane programmer, but I seem to remember that the literature says that it's not generally a cost-effective way of finding bugs. In particular, you multiply the cost of development by (say) 3x (which is fine, on its own) but also the places where bugs are inserted are typically the hard parts; so you don't reduce the number of bugs as much as you'd like; it can easily be more cost effective to invest the few million in static analysis etc.

As much as we'd like plane manufacturers to test things to death, it'd become too expensive too quickly. For all we know, this software could be written by a contractor, or the firmware for a third party part.

As far as I know, N-version programming was effective when software systems were small (shuttle ran on 50k lines of code) and where poring over every single line was possible, because the hard part was coming up with the spec.

Nowadays a big plane like the A380 might be expected to have 100M lines of code in its subsystems, and it's simply too expensive.

> Nowadays a big plane like the A380 might be expected to have 100M lines of code in its subsystems,

Why does an airplane require 100M lines of code?

Linux kernel alone comes in at 15m SLOC. Now add an userland subsystem and you're at 20-30M just for one device.

Multiply by all the little and big subsystems, the embedded chips, in-flight entertainment, network gear... 100m SLOC is too low, I think.

Erm, the article I got my numbers from is: http://www.aerospacelab-journal.org/sites/www.aerospacelab-j...

where it states that the airbus A380 has more than 100 million lines of code in its avionics systems.

Sorry if this is incredibly ignorant, but I can't believe flight control systems are running Linux?

Do these systems not have hard real-time requirements about the execution time and periodicity of tasks which can't be guaranteed by the time-sharing scheduling algorithms in Linux?

There are several real-time Linux variants.

There are - to my knowledge, feel free to correct me - no real time linux versions (or even any version of linux) that are currently certified for avionics (DO-178B certification is required for that, there are multiple levels and I don't know of any linux distro (or just the kernel) with that certification).

Avionics certifications are well past the extent of my knowledge and I will gladly accept your point. I was generally addressing the idea that the need for preemptive scheduling precludes the use of a Linux-like kernel.

I've never heard of anyone getting Linux running under hard real time constraints. You can get it pretty good (excellent for real time audio, for example), but can never be 100% sure you're going to meet the deadline. The people I've talked to who tried said by the time you strip out enough of the kernel to approach hard real time, you've lost enough of the advantage of using Linux that you may as well switch to an RTOS.

Real time audio only if you use a large enough buffer and a hardware component to clock the data out.

It's all about the latency guarantees and that means that you're going to have to inspect each and every path through the code for length. With the complexity of the linux kernel that's a pretty tough job and I suspect that anything lower than a few hundred milliseconds (guaranteed!) is out of the question.

I built a little controller using linux that required hard real time during a long (multiple minutes) of operation and the way I hacked it was to simply disable all interrupts and recover the various drivers as good as possible once that phase was over. It worked well but was mostly deaf to input during that time except for polling one 'stop' switch which would cause the machinery to coast down to a halt after which interrupts would be enabled.

Good enough for tinkering but I certainly would not bet anything in production on that strategy.

Real time is hard, soft real time is hard enough (without guarantees but with a best effort and a very large fraction of the deadlines satisfied), hard real time (no misses at all, guaranteed) is hard for a kernel of any complexity.

Real time systems will be running a RTOS: VxWorks or QnX or something equivalent to that.

They'll definitely build a prototype using Linux but they won't get that certified so it literally 'won't fly', it's just a means to speed up initial development.

Is the order of magnitude of lines of code in QNX different from that of linux? At a first approximation, I don't see why it would be.

The QnX kernel is very small compared to the Linux kernel.

Small enough that I could-reimplement it in approximately 3500 lines of code + another 850 for the virtual memory management.

Wow, I had no idea. Since their source is closed and untouchable I had no way to check either. Is there any reason there aren't several certified open RTOSes around?

I don't know if there aren't any open certified RTOS's around, but I can explain the 'why' part easily: if you pay for the certification of an open RTOS then everybody that can use one will say 'thank you' for the effort and that's that, since the certification would apply to any and all copies of that particular version. So you're essentially paying for the privilege of cutting your competitors a break.

This could only work if the entity paying for the certification had a way of making that money back somehow and I don't see how that could be done.

Not an RTOS, but seL4 is a correctness-proven open-source ARM microkernel: https://sel4.systems. Looks like a mixture of public and private funding. It's part of the L4 family, http://en.m.wikipedia.org/wiki/L4_microkernel_family#Univers... which includes OKL4 (deployed on 1B+ ARM-based mobile phones) and http://genode.org (x86/ARM) from Dresden.

That's super cool, thank you for that link.

Not to detract from the fine work done by the sel4 folks, but there is a large gap between what they have and what DO178 C requires for level A software. Like many other bureaucratic organisations, the FAA (and other regional equivalents) have a process with it's own set of rules (MCDC testing, requirements/design traceability artifacts, etc).

It would cost a significant amount of money to develop the necessary artifacts and engage the FAA to obtain a certification.

That's absolutely true but something like this could be a good starting point.

What I think the whole thread above misses is that the economics simply aren't there, cost isn't the limiting factor for the OS licenses for avionics but an extra certification track (especially for a fast moving target) would be, besides, it is not just the OS that gets certified but you will also have to (separately) certify (usually) the hardware that it runs on (unless you're going to use a design that has already been certified).

That means that modifications are expensive and that 'known to be good' trumps 'could be better' or 'could be cheaper in the longer term'.

Someone would have to come up with a very good reason to see open source trump the existing closed source solutions.

In theory, while certification would be done on a binary derived from seL4, any improvements resulting from the certification process could benefit the open-source core and derivative binaries. Compared to a proprietary OS, improvements would have ecosystem-wide benefits.

In addition, a modular microkernel architecture could use reproducible builds to generate identical binaries from identical source. This would enable binary components to be certified both separately (akin to unit testing) and as an integrated system (mix and match components). This could reduce overall duplication and certification costs, even among competing commercial products derived from seL4 components.

3/4 of that 15M is drivers and filesystems you won't be using, and this is not including the entertainment systems.

They should be able to keep it under 10M lines pretty easily if they actually cared about bloat.

A complicating factor is that you really should include your compiler as part of your codebase (if you're doing lots of formal method work, you will usually do it on the source code, and the compiler can invalidate all of the guarantees you carefully program in), and this will be millions of lines of code (where you can't simply factor a tonne out).

Think GCC 5 is about 15 million or so now.

it's always easy when you're not the one making the changes on the system you don't understand. is there a name for that? I feel like it's a common enough thing people do that it should have a catchy name.

Eh. To say that something that used to be mechanical should be possible with a mere few million lines of code seems pretty obvious to me.

Or compare to the Apollo missions and the space shuttle being well under a million.

I'm not saying that it would be easy to redo the entire system from scratch now, I'm just saying that if it was a design goal from the start it wouldn't have been very onerous.

Because each discrete component that you can eliminate (and replace with code) is a weight saving. Because once you tip over a point in complexity, you just keep adding more code to guard against more edge cases - edge cases you can't avoid because they involve crashing into mountains. Because you want to offload as much possible effort from the cockpit crew, while still allowing them full control over the automated feature

All of these things just add more and more code.

"independently generated from the same initial specifications"

For all we know, if the spec required a counter of 100ms intervals, all N implementations could contain the same bug.

N-version avoids random, but not systematic bugs.

The complexity of the software that manages the N versions seems terrifying, and decision-by-committee in the implementation seems problematic. 3 implementations could select different sequences of responses to a situation, each of which individually is correct, but which are disastrous when combined by vote. E.g. say they're voting on 3 different numbers that have to add to 1; proposals are (1 0 0) (0 1 0) (0 0 1). Consensus is (0 0 0), oops there goes a billion kilowatt dam.

Reply on news.ycombinator.com