Hacker News Reader

Comment by CGMthrowaway | Hacker News Reader

On the Qantas 72 flight (2008), the ATSB report showed the same power spike that upset the ADIRU also left tidy 1-word corruptions in the flight data recorder. Those aligned with the clock cycle, shared the same amplitude and were confined to single ARINC words. That is pretty much exactly the signature of a failing solid state relay or contactor on the shared avionics power bus (upstream of both FDR and fly by wire).

Radation-driven bit flips would be Poisson distributed in time and energy. So that is one way to find out

View on HN

Do you think they're using the guise of "its solar radiation" as cover to do a software update to fix a more problematic "bug", and perhaps tangentially there are some changes in said-update to improve some error correcting type code (eg: related to detecting spurious bit flips).

Yeah I don't buy it either.

If it was really 'solar radiation' there would be more small details.

I would say its pretty detailed -an unknown interference caused a single crc protected 32 bit word to be corrupted simultaneously, by timestamp, in both the flight controller hardware and the black box data recorder.

My concern would be what error correction mechanism did or did not catch the corruption in memory and why did it not recover without critical impact to operations?

> corrupted simultaneously

This sounds like a software bug.

Something like - {copy a to b, checksum a--b}

Instead of - {copy a to t, checksum a--t, copy t to b, checksum a--b}

I bet the fix is along these lines, with the caveat of real time systems/etc.

Reading the Airbus press release, I wonder if this is what happened:

Solar radiation event led to alpha particle induced data corruption in a flight control computer memory (could be DRAM, SRAM, on-chip cache, registers...). These failures are supposed to be transient (reboot and all is well).

This is an anticipated failure mode. Only one (of three?) computers should be affected by such a failure and therefore the remaining two keep on running the plane.

But what happened is <something> went wrong with the failover/voting mechanism (as often happens with one-off seldom-executed failover code). The result was no flight control computer functionality until the entire system was rebooted. Hence the emergency landing.

The fix is to address that software error, with perhaps a secondary fix TBD to harden the hardware (add some shielding perhaps).

The fact that they talk about data corruption and not just a malfunction suggests alpha bit flip rather than latch-up.

Then send the whole statement through a French to English translator to make it a bit more confusing.

My guess is they haven't managed to point to the single memory bit which was flipped to cause this result.

The software update is probably more along the lines of 'lets just introduce a watchdog task which resets the system if the output deviates too far from the input for too long'.

No, that would be straight to jail.

Remind me who from Boeing went to jail?

Airbus is in Europe where the Rule of Law still exists

That’s what we naively thought here too.

Look at how US government treats financial behemoths which actively harm whole mankind vs how EU treats them. There is way more to this topic obviously (who wants to harm their local company), but generally US is pro-companies while Europe is pro-people.

Deutsche Bank and HSBC, two major European banks, have repeatedly admitted they have engaged in money laundering activities for Russia, drug cartels and terrorists and have consistently failed to meet their AML obligations. The US is the only entity that’s going after these banks for these issues winning significant judgments and even with that backdrop you don’t see any EU enforcement.

Not in aviation.

Does the 737-Max not count as aviation anymore?

It does. It is but the Max issue was well different to this one.

Counterpoint: Boeing MCAS tho

No, because aerospace is not garden-variety Silicon Valley webshittery.

There is a slightly different level of discipline and engineering ethics at play.

After reset, it went away. If it was this kind of hw issue, it should still be present.

Considering those units were designed back when they did not have EDAC mandated, I can believe it could have been a bit flip (along with some other stuff they will probably address to take into consideration this failure mode). Nowadays, most MCU's have ECC on them so the time of this excuse is mostly gone now. :)

> Nowadays, most MCU's have ECC on them so the time of this excuse is mostly gone now. :)

That's kind of a misleading statement. Assuming you mean on planes built nowadays, as we clearly see that nowadays planes still flying (6K of them at least) still have issues. We don't need hand wavy comments trying to make it sound like modern day aviation is no longer susceptible, especially when it's in a thread on an article showing how that's just not true

I think you and gp may be speaking about different stages. Gp seems to be saying that a plane being designed and specified today would use technologies hardened against this type of error.

That even though they’re in widespread operation today, the aircraft types in question were designed (and certified) many years ago, before ECC was the norm. My impression is that, once their type is certified, new airframes are built to pretty much exactly that specification even all these years later.

> I think you and gp may be speaking about different stages

Yes, that's my point. Just because new aircraft are designed with improved hardware does not automatically mean the issue is resolved industry wide. Existing equipment will still have issues. So the statement is misleading. Is the number of aircraft with ECC "most" of the equipment in the skies?

Ok, I can see how my statement can be confusing. I wanted to say that on newly built things this is mostly gone today, although I'm certain freakish accidents can happen. Yes, if your hardware does not have ECC[1] that is something that can happen. I was initially surprised because I did not expect them to not have error correction, but I guess it makes sense for systems designed a long time ago and still in use, so that was new info to me.

[1] Technically EDAC is the correct name of the whole sybsystem, and ECC is the name of the algorithm. But I've only heard it refered as ECC in my industry. I was even initially confused when I read EDAC, so TIL.

I'm very surprised that a plane doesn't have voltage, current and glitch monitoring on every power rail, logged to the data recorder.

You would pretty much be logging, every millisecond, the minimum, maximum and mean voltage for every 1ms period (and the same for current).

Then any failing solid state relay would be obvious in the collected data, far before you start to get word corruption!

What do you think it could be ?

"That is pretty much exactly the signature of a failing solid state relay or contactor on the shared avionics power bus (upstream of both FDR and fly by wire)."

Thanks. I didn’t realise what that meant in context.

Reply on news.ycombinator.com