Hacker News Reader

Comment by chris_overseas | Hacker News Reader

> Pilots found that jets’ systems would shut down midflight about once every three or four hours and have to be rebooted.

> But the new, improved version of 3i is a significant improvement, according to the JPO. The new version showed approximately twice the level of stability as the previous load, Block 2B, and three times better stability than the original 3i software

Does that mean it "only" needs to reboot once every 6-8 hours? If that's considered stable enough to sign off for production, does that mean they will simply reboot before each flight and just hope for the best?

View on HN

According to earlier articles about this issue, planes currently in wide use need to be 'rebooted' at a similar frequency, it's just specific subsystems that get rebooted (I think the F-35's former problem was a sensor), the planes keep flying, and missions don't often last that long anyway (IIRC all the details).

Here, I found the link:

https://news.ycombinator.com/item?id=11260927

EDIT: Heck, I'll just copy the meat of it here:

http://breakingdefense.com/2016/02/bogdan-predicts-f-35s-for...

The toughest problem the program is having is matching the timing of the aircraft’s fusion software with its sensors’ software. “As we add different radar modes and as we add different and capabilities to the DAS system and to the EOTS system, the timing is misaligned,” and then you have to reboot it. Bogdan said he’s aiming for eight to nine hours between such software failures when a radar or DAS or EOTS needs to be rebooted, which is what legacy aircraft boast. Right now they are at four to five hours between such events. “That’s not a good metric.”

I guess if you demand a lot of interaction between almost all systems, it's harder to ensure high levels of independence.

Welcome to the perils of globalism...

It's not hard at all - you buffer it and make the communications paths asynchronous. You allow dropped "packets" when necessary. I don't think you can enforce hard realtime constraints across subsystems and expect anything but tears.

Buffering is not such a great idea when your getting shot at.

I'm not talking about buffering that breaks a time line. I'm talking about buffering to deal with something that can be modelled as essentially clock uncertainty.

Holy friggin crap. Can I buy a hardware timer, pls?

I have to be reading this wrong - they can't even enforce the time budget for different tasks/threads under varied configurations?

Don't nobody here know how to play this game?

> Don't nobody here know how to play this game?

I guess all they need to do is hire "ArkyBeagle" from the comments section of hacker news, right?

Maybe! What do we know?

We tried that. It didn't work out :)

Real issue is different company's building different systems and then trying to integrate them after the fact. Chances are good that several systems are involved no one team has source code for even 1/2 of them. So, this is probably a reasonable trade off to prevent something else braking or massively delaying the project.

Don't forget this is a weapon decreasing latency by 0.0001 seconds might be worth a lot of pain.

They actually will have most of the source code for everything non-classified. However, they won't be allowed to change any of the software from other companies. This means getting a single bug fixed involves several con-calls with various levels of management at each company over the course of somewhere between two weeks and a few months. Double this at a minimum if the software wasn't developed in-house at the company that delivered it to you.

In such an environment you very quickly get used to living with workarounds rather than bugfix.

[edit]

Oh, another thing I forgot that isn't universal, but is unfortunately too common. Dealing with managers after the bug is fixed asking why you spent time, and thereby the company's money fixing a bug in code that is the responsibility of another company.

I realise there's a difference between military and passenger, but the 787 Dreamliner had a problem that required a reset every 248 days and it made the headlines: http://arstechnica.com/information-technology/2015/05/boeing... Based on the raw numbers the F-35 problem sounds a LOT worse?

My intuition is that occasional software crashes like this probably follow a Poisson distribution statistically speaking. Like occasional cars appearing on a lonely street. One car appearing doesn't affect the expected arrival of the next car. So rebooting before the start of a mission is not an effective strategy. Just idle speculation based on a stats and probability class 36 years ago. Happy to be proven wrong.

I somewhat doubt that model is valid for these issues. A large number of software issues relate to run time of a system. In particular, this one seems to be clock drift from other comments. That's more of a random walk; which crashes a system at some given distance.

Unless the crashes are caused by things cosmic rays, they're very likely influenced by the internal state of the aircraft and therefor not independent events. If that car showing up causes the world to reset - then you won't likely see another car for a few million years.

If I read the thing right, the system is in sync at reboot and effectively, jitter renders it unstable over time. So events are not independent.

When you start a mission you are literally starting up the plane. Along the way you turn on the avionics, authenticate, turn on the engine, ask the plane to perform a comprehensive set of built in tests, etc. You are booting up. It does not make sense to reboot before you've even booted up in the first place.

In general I would agree.

But there could be indeed some SW failures that are more likely after a given time of running. E.g. memory [de]allocation problems, memory fragmentation - in case the use some kind of dynamic memory. Or other types of resource leaks.

> Does that mean it "only" needs to reboot once every 6-8 hours? If that's considered stable enough to sign off for production, does that mean they will simply reboot before each flight and just hope for the best?

Ferry flights must be an experience.

F-35s have ferried across the Atlantic without difficulty, and across the continental US on numerous occasions.

That's continuous delivery for you.

Would a jet be flying missions for 6-8 hours anyway?

Yes, with aerial refueling, missions of 12 hours or more are not unheard of. The pilot's endurance is the limiting factor. Or, as I've heard it said, "You worry about running out of ass, not running out of gas."

Reply on news.ycombinator.com