Hacker News Reader

> The toughest problem the program is having is matching the timing of the aircraft’s fusion software with its sensors’ software

I don't know if I'm interpreting that correctly but what I'm getting out of it is clock skew between sensors and the fusion system, which I assume to be referring to sensor fusion.

That makes sense, analogous to a typical distributed systems issue. I would imagine the skew needed for the type of sensing they do is quite small. I doubt they can just toss an atomic clock on the plane :)

I would imagine the skew needed for the type of sensing they do is quite small. I doubt they can just toss an atomic clock on the plane

I don't think a single atomic clock would do it -- sounds like they'd need a high accuracy clock for each separate subsystem.

But unlike distributed systems, it seems like an aircraft is small enough to have one clock that everyone uses - any idea why they don't do that? Single (or too much of a) point of failure? Disparate vendors can't agree on a standard? Too much extra wiring or other complexity?

The standard way to do timing in a system like this is for each subsystem to run its own binary clock, with the main system sending period time-of-day messages to keep everything externally synchronized. In that sense, all the systems do have one clock. Many do not need to worry about wall clock time or synchronization, though.

> I doubt they can just toss an atomic clock on the plane :)

They totally could. Some atomic clocks are quite small. The one pictured on the below wiki page doesn't look that much bigger than a 3.5" hard drive.

https://en.wikipedia.org/wiki/Rubidium_standard

I wonder how something like that stands up to a gunfire vibration test.

That might have detection implications though.

> They totally could. Some atomic clocks are quite small.

Are they mil spec? In other words do they operate correctly over a very large temperature range, with large input voltage variance, under high G-forces, in high vibration environments, etc?

I don't know anything about atomic clocks past what I can read on wikipedia but, I can say there is generally a large difference between mil spec ICs and commercial or industrial grade ICs, I would assume this would extend to atomic clock scale devices as well, but just a guess.

Who needs atomic clocks on individual planes when the US government has spent billions of dollars to place a fleet of atomic clocks in orbit?

Someone whom wants their planes to work when the enemy jams GPS and/or shoots the satellites down. We're talking about war here after all.

So you freewheel once GPS goes down. You might drift a few microseconds. This is not necessarily a problem.

If you're traveling at high speeds and relying on this timing to adjust flight control surfaces, or trying to use radar to get position of combatants and/or the ground, a few microseconds clock skew would be a major issue.

I didn't say clock skew. I said drift. As in, you lose GPS timing signals so your own internal master clock freewheels. That has nothing to do with clock skew (different components within the jet 'seeing' different time due to delays or other issues). The jets are designed to function with degraded or no GPS functionality. But, if it's there, they will take full advantage of it.

For those downvoting, see my comment below, where I explain why this really isn't a problem.

Also, if I'm understanding the link properly you'd need an atomic clock for every sensor to make it work.

That is one explanation. I've worked in avionics telemetry & remote sensor fusion and there is actually a push to go from synchronous archs (using PCM on I2C buses) towards modular and networked archs (using some variant of IP/etc. protocols).

Problem is you need a reference time clock. On monolithic arch, is easy : the backbone bus timestamp the paquets it sees. On networked arch you need a time server a la NTP, otherwise you have time skew and loopbacks.

Even with a accurate time clock, you may also need to correct for timing issues due to cable length.

Check out PTP, and then the WhiteRabbit research project. It's not exactly easy to implement, but you can get down to ~10ns sync on an ethernet network using it.

I'm curious what you mean by 'synchronous' here, for example an I2C bus is clocked by the master, ignoring clock stretching and that clock need not be in any way synchronous with the processor clock on the slave.

And to clarify, we're talking about PHY level protocol clocks here, not time clocks.

I doubt they can just toss an atomic clock on the plane :)

As I understand it, an early variant of the TCAS system for collision avoidance did just that. Every aircraft would have a Cs or Rb standard that was synchronized to a master clock, and they'd broadcast their position and altitude at an assigned time slot.

This would have been very expensive at the time (late 60s-early 70s?) they were proposing it, and it would only have worked if everyone had the necessary equipment. So it didn't fly. But in later decades it was no big deal to put a high-quality atomic time/frequency standard onboard an aircraft, and I wouldn't be surprised if most large airplanes actually do carry a Rb standard for one reason or another.

Edit: the proposed system was called EROS: http://www.allstar.fiu.edu/aero/tcas.htm

I think the image a lot of us have is that an atomic clock is a PC-to-dishwasher-sized box. Rubidium standards are smaller than that but less accurate than the lab grade equipment.

You don't really need to toss an atomic clock on the plane. Just make the plane use GPS.

Where one of the tricky parts comes in, is in distributing this clock signal to all portions of the avionics system that need it. But that's beside the point.

It doesn't really sound like a timing skew problem (as in, a problem with making all systems recognize the same time simultaneously) but rather with making sure that all relevant information needed to construct a coherent 'picture' from all sensors reaches the core processors in a timely manner.

You can't provide a real time display if, say, your radar data is delayed for a significant period of time due to problems or instabilities with the radar's onboard processor.

> You don't really need to toss an atomic clock on the plane. Just make the plane use GPS.

These are warplanes - you don't want critical systems reliant on satellites that could be jammed, or downed, during wartime, which I suspect is why they don't do this already.

But fighters do use GPS. Daily. They use it for accurate navigation. But they do not absolutely depend on it.

> But fighters do use GPS. Daily. They use it for accurate navigation. But they do not absolutely depend on it.

But...

>>> Just make the plane use GPS.

...implies you would make them absolutely depend on it. Military aircraft use GPS daily, because it's cheaper and better. However, they also have accurate inertial navigation systems as a backup, so they can still operate effectively if the enemy makes GPS unavailable.

How many fighter plane flights last longer than 8 hours? It's a mean time after all, it could simply be a reflection of the length of the sortie.

Probably not a lot but it wouldn't be uncommon for an American plane to be in the air for 12+ hours with in-air refueling.

That mean time still leaves room for that doesn't it?

(2 + 2 + 2 + 2 + 12) / 5 = 4.

And so on. This whole statement is meaningless without more information about the distribution of lengths of flights.

Forget fuel, and radar crashes... can you imaging flying a fighter jet for that long? The stress on the pilot must be insane.

Thats why they're fed a cocktail of amphetamines pre-flight for longer sorties.

It is relatively uncommon for a single-seat (& single-pilot) fighter jet to fly more than 8-9 hours. If they do, then AFI & OPNAV duty rules restrict how these flights are conducted (including combat ops) and how soon the crew can fly their next mission. Aircraft like the B-2 have two pilots and planned sleep periods.

According to references, 13.5 hours [0] is the record for the F-16 combat mission, and this was actually a SEAD (Suppression of Enemy Air Defenses) mission in a single-seat F-16CJ. Similar missions in the EA-18G extended to 13+ hours with a single pilot, along with a WSO/EWO in the back seat.

Various unofficial reports indicate F-16s do fly 14 hour ferry flights, but that is highly unusual.

[0] http://www.f-16.net/f-16-news-article4413.html

What I'm wondering is how these fighter jet pilots deal with needing to go to the bathroom. 14 hours stuck in a tiny cockpit kinda makes it hard to relieve yourself...

Not sure why you are being downvoted. It is a serious question, if a bit gross.

At the Evergreen Air & Space museum (home of the Spruce Goose), they have an old SR-71 on display. Part of the exhibit has a TV playing a video loop with pilots and crew talking about their experiences. One of the pilots recalls how they'd have a "high-protein, low-residue" breakfast like steak & eggs before a long mission. The idea was to provide long lasting energy without having to go "number two" in the suit. He says that some of the crew who would help them out of the flight suit at the end of a mission were female, so it'd be especially embarrassing if you'd pooped yourself. I got the impression that, while rare, it did happen sometimes. :-\

Imagine the Falkland War's Vulcan march with hard reboots...

This is what happens when you have an entire industry (defense/general gov't services) that treats software development as a cost center instead of a profit center.

I think it is time for the incumbents to be interrupted. Boring/Northrowup/... are fking up to much. Here is another defense industry f-up [0]

[0] http://abcnews.go.com/US/jet-skier-broke-jfk-airports-securi...

Likely refers to sensor fusion (https://en.wikipedia.org/wiki/Sensor_fusion), which is an algorithmic process.

A 4-hour MTBF window (even for a subsystem) is terrifying considering modern vehicles like this are likely all fly-by-wire.

From the quote it appears that it's 4 hours between reboots of any subsystem, not the whole plane.

Don't think of it as a failure. Think of it as a garbage collection pause at subsonic speeds.

Spot on.

This article should clarify what the radar reset that they are referring to is. Integer, origin, and timing resets are a function of radar tech, and are not specifically related to something like a system reboot. This would be something like restarting a process due to bad data getting into your process from a sensor and potentially leading to a bad result from a range or trajectory.

I have no experience with aviation, why do they need to be rebooted at all in less time than they can fly without refueling? It would seem with such a critical system the tolerance for rebooting would be exceptionally low. The F35 is not a dogfighter; does it have any meaningful capabilities while radar-blind?

> why do they need to be rebooted at all in less time than they can fly without refueling?

I don't know much about Navy / Air Force ops, but it's possible that a sortie may consist of taking off and returning to rearm / refuel without shutting down, which could easily span more than four hours. In addition, lots of fighter planes do refuel in mid-air, so the requirement is still very prevalent for the subsystem to remain functional.

I'm actually curious what possible technical reason would prohibit them from running for 48+ hours (or indefinitely). IE some kind of garbage collection issue. I'm unaware of any reason they should ever have to be rebooted aside from software upgrades and curious why it might be necessary (technical reason). Unfortunately, from the article: "But the Air Force did not go into detail about when the instability in the radar system occurred..."

Edit: It is mentioned elsewhere in this thread that it is a timing issue, see: http://breakingdefense.com/2016/02/bogdan-predicts-f-35s-for...

Have to shut the engines off at some point. Any vehicle "reboots" when you shut down the power.

It is certainly possible, but not normal. Normally the engines are shut off during servicing and the aircraft is hooked up to a ground cart to provide limited systems functionality.

Most military jets can be hot refueled on the ground, with engine(s) running. Some fighter jets (F/A-18s etc) can hot swap crew on the ground, with one engine running.

With the F-35 and other jets, its possible to swap crew and perform minor maintenance with the jet electrically powered up and cooled with auxiliary air supply, although the engine is shutdown and then restarted. This actually saves significant time on the pre-takeoff procedures. I would think the Lockheed Martin engineers could also perform diagnostics after a failure, without interrupting power or restarting the sub-system.

The thing I'm struggling with is, "Why does rebooting solve the problem?"

I can only imagine such sad state of affairs comes from extremely quirky requirements and very tight and crummy process (from TPS reports to worse)

Yes, I can understand the problem is complicated, but come on.

Any clue to why it would take so long to reboot?

The reboot itself is not the N hours figure, that's the interval between reboots.