Multiple Boeing 787s in China experienced GPS 20 years rollover issue | Hacker News Reader

Multiple Boeing 787s in China experienced GPS 20 years rollover issue

twitter.com

126 pointsjcfrei59 comments

Someone on reddit said yesterday a KLM flight (apparently B777) was grounded because of incorrect date (in some Honeywell equipment?): https://www.reddit.com/r/sysadmin/comments/babyvv/anyone_hit...

We had some Honeywell engineers at my workplace for a system integration meeting. Over lunch I got the opportunity to talk with them. This came up in conversation. Apparently it's very much all-hands-on-deck at the moment - almost every system they've currently got on-board Boeing aircraft is being assessed, and they're in all hours of the day to get new revisions through. I do not envy them right now.

Is it because they didn't see the GPS rollover coming up (it was in the news though)? Or did they not think their products would be affected?

Partly that (I think there were some legacy systems that were missed), partly that every minor clerical and drawing error that got passed through because of how insignificant it was (nothing that would affect the integrity of the aircraft I should add) has been flagged by Boeing. They want to look proactive to their customers and shareholders I suppose. That means lots of tiny, insignificant changes to hundreds of drawings. It's nothing technically challenging for the most part - just lots and lots of paperwork.

I don't get how some planes were affected, but not all; Every 787 in China should be running the same software version, let alone the world, no?

Are there any reasons a fleet plane might be using a different version? I'm coming up short on reasons, other than language, or maybe special hardware modifications?

iPhones on China have interesting "hacks" to comply with laws (GPS has some restrictions: http://nowiknow.com/the-problem-with-chinese-gps/), and they're not allowed to have the Taiwan flag emoji.

I also noticed the other day that Google Maps shows static images for S. Korea (not the fancy WebGL stuff). Also due to laws.

How does this user-facing map change translate to avionics though?

Both are examples of laws messing with tech availability and inspiring custom workarounds. Those workarounds then tend to suffer from getting less attention than the mainline version with regards to updates and maintenance. Chinese aviation might be hit by both import restrictions (by the Chinese government) and export restrictions (by the American government) at the same time.

That's true, but that's also hardly a unique case - many other countries have special requirements and demand (e.g. Russia also uses metric system).

Also many other countries demand Google Maps manipulation (USA being one of them by demanding hiding of military bases, not to mention many other border disputes over the world). So I'm not sure how authors point really links up to the difference in avionics in Boeing JUST for China.

What are those different requirements?

China has different requirements than the rest of the world. It'd probably be good to have a pilot chime in here (one who flies both in China and ROW). For instance they allocate flight levels in metric units (most of the world still uses feet, IIRC).

They have regular software updates. Just like Windows update, nobody do that regularly

I concur. For example in telecom getting operators to update anything is just a pain. Unless they are seeing service affecting issues, they are totally fine running decade old software. Also sometimes if they have lets say version 5 and current is version 12, latest, and they need some fix, then depending on the size and wealth of the company they may force manufacturer to create version 5 plus fix, making it a new branch just for them, even if all versions 6 to 12 were already tested in production.

I can speak from the enterprise side of this.

Typically this is done because the work required to certify a new version (either internal or external certification) is prohibitively expensive.

For core, business critical systems, a lot of large, non-technology businesses simply don't have a mirrored QA setup with enough coverage / integration simulation to work up through.

So... faced with a heavy testing price to upgrade, risk due to incomplete testing environments, and known current version behavior... a lot of businesses punt and say "We want our version + only the fix for this issue we're experiencing."

It's annoying from the software vendor side (QAing old_version + patch_x2 + patch_q5), but it absolutely makes sense from the business side.

The summary reason is: because the customer doesn't have a flexible and testable infrastructure configuration. And that's not really something a software vendor can change.

Haha, that's surprising though, since these are important systems, if there is an update, doesn't it mean something really needs to fixed, i.e, gps rollover? Any bug left unpatched seems unacceptable

Maybe I have more faith in the 'system' than I should...

Sometimes, perversely, important systems are the _least_ likely to be updated. Operators require extensive testing before widespread application of the updates, which takes time.

We're talking about the same manufacturer that's responsible for the 737 MAX fiasco. I'm not sure how much trust I would want to place in their software.

You're putting trust in their software whether you upgrade it or not.

You forget that Boeing doesn't own these planes. These aren't Tesla's. We are currently in an era were it seems to be expected that the manufacturer automatically update their software without the end user intervention, ie Tesla, Windows, Apple, etc... Thats not how it was done in the past, particularly when the bandwidth didn't exist.

I don't see how this changes what I wrote.

Not necessarily.

Different components could be made different companies. While flight controls and handling fly-by-wire could have been Boeing, the FMS could be (e.g.) Honeywell, GE.

The FMS, AFCS ("autopilot"), and (primary) FCS are three distinct systems per ATA iSpec 2200:

* https://en.wikipedia.org/wiki/ATA_Spec_100/iSpec_2200

Things can be more blurred with fly-by-wire, but can still be conceptually separate.

Funny I got an old Tom Tom for $10 a few weeks ago, too old to get software updates, and I was pleased to see that it worked just fine after the rollover.

At the [dead] commenter: Timekeeping is an integral part of how the Global Positioning System works, to the extent that relativistic time dilation is a factor. You simply cannot use a GPS receiver with a broken time implementation, at least not without significant accuracy issues.

But internal time is not the same as display time.

Internally, the week counter just rolled over. That's fine; all the almanac data is handled in that same format internally. Some receivers may need a reset when this counter rolls, and may need to relearn the almanac/ephemeris from the sky, but once that's done, Sunday of week 0 is just the same as any other Sunday of any other week 0.

Navigation works just fine because it's all internally consistent.

But display time, ahh, now that's where things get interesting. Sunday of week 0 could be displayed as 2019 April 7, or it could be displayed as 1999 August 22, or it could be displayed as 1980 January 6.

Knowing which epoch you're in is actually considerably trickier than working consistently internally, since there's no information in the satellite signal to tell you this, by design. It comes from applying an external offset. Receivers have lots of ways to do this, and quite a few simply have it baked in and cannot handle a rollover, so as far as they're concerned, week 0 is in 1999, period. They'll navigate just fine but they'll display the date wrong, because they're applying the wrong offset.

The easiest way of doing this is to assume time will never travel and store the latest computed date right? If you've seen 2001, it's never going to be 1999 again, so it must be 2019.

I understand that will require either flash or battery backed RAM, etc to keep this stored, but seems relatively trivial in most applications... (side effect would be that if the backup battery is dead or all batteries are disconnected, you would revert to 1999)

Unless someone manually sets the date incorrectly or gets the timezone wrong (assuming the date isn't stored as UTC).

Yeah, you can only use the GPS source for that flag.

Cool, see, I knew this was worth talking about. While I've learned about GPS as a way to gather meteorological data[0] via radio occultation, I never had to deal with the raw data stream myself. Thanks for your expertise.

[0]: https://en.wikipedia.org/wiki/Constellation_Observing_System...

Funny enough my Tom Tom made some noise and seemed to boot with a different sequence than usual, in particular it insisted that I remove the USB cable that I use to power it from the car. I did that, and it found the signal in 50 seconds and worked normally, correct date and everything.

Any tips for someone looking for similarly priced antiquated GPS's to shove in the glove-box of all the cars in their fleet?

There were four of them here:

https://ithacareuse.org/

there also was a $5 pair of Bose headphones that I was stupid to not buy the instant I saw them.

The Tom Tom I have is too old to get updated maps, but it's not a problem driving around in upstate NY. In New Hampshire it might be another thing, but where I live they build new buildings but not new roads.

It isn't really the Nav/Positioning that has issue- but the time. Check the time on your Tom Tom unit. Is it 2019?

Sort of related: my Sony TV, circa 2013, ALWAYS has this opening message (even if there's been no power outage that I've noticed): "Power outage. Time needs to be reset." I've been ignoring it since 2014 because it takes about 5 minutes and a half-dozen menus correctly sequenced (but only after multiple FAILs) to reset correctly.

What's the reasoning for the current GPS time format and it's resulting issues? Can't we just use a 64bit integer timestamp? That would give us roughly 580 years in nanosecond resolution?

GPS was designed a long time ago, and the data rate for ephemeris and almanac data is.... 50 bits per second.

That's not an exaggeration for comic effect. There's literally 50 bits per second from each satellite. 6.25 bytes.

So the navigation messages are packed extremely densely. To the point that plenty of numbers don't even start on byte boundaries.

The age isn't the only explanation for the low data rate- it's not simply low tech.

GPS relies on receiving extremely weak signals, often with a very limited receive antenna. This means there's a real trade off- you can send a limited amount of data over a given communication channel. Increasing the data rate might require increasing the transmit power / antenna gain of the GPS satellites, or requiring bigger / better receive antennas on GPS receivers.

Here are the parameters involved: https://en.wikipedia.org/wiki/Shannon%E2%80%93Hartley_theore...

When you consider that you're listening to a ~500W EIRP transmitter that's >20,000km away, with a handheld receiver that usually has a pretty omnidirectional antenna... 50 bps starts to sound quite impressive!

More so when you realize the signal level is under the noisefloor!

I will never again get annoyed because my $140 Garmin Forerunner 35 watch takes up to a minute to lock onto GPS. That it does so at all is the real wonder, a là Samuel Johnson's remark about the dog that walked poorly on its hind legs.

> What's the reasoning for the current GPS time format and it's resulting issues?

GPS satellites and receivers last on the order of 20 years so it takes a long time to age out of legacy support. The latest generation of satellites, GPS-III, have a new week number format that allows for 13-bit week numbers (120 year rollover).

The reason for the low number bits in week number is simple, it's broadcast on a very low bit rate (50 bits per second) and every frame has the week number in it. So you get faster fixes if you don't pad out with useless zeros.

> The reason for the low number bits in week number is simple, it's broadcast on a very low bit rate (50 bits per second) and every frame has the week number in it. So you get faster fixes if you don't pad out with useless zeros.

It should be noted that the coders could have used data windowing and pivot dates to make certain assumptions:

> When ntpd(8) receives an unresolved timestamp from an upstream server, that timestamp could be based in any era prior to the current wall-clock date - but by definition we may not know the wall-clock date with confidence yet (not having achieved sync).

> To resolve this ambiguity, NTP also uses an internal pivot date. It chooses the era that would minimize the absolute value of the time difference between the resolved timestamp and its internal pivot date.

> This resolution technique is part of the NTP specification. It will normally resolve to the current era, but may in cases where the pivot is within a half-cycle of a rollover time resolve to either the previous or the next era.

> An ntpd(8) instance’s pivot date will be the date it was compiled and built.

> It is fairly easy to see that this guarantees correct resolution if the pivot dates of the communicating ntpds are within a half-cycle (88 years) of each other, even if there is a rollover between the pivot dates.

* https://docs.ntpsec.org/latest/rollover.html#ntp_pivots

* https://en.wikipedia.org/wiki/Date_windowing

The GPS signal format was was designed in the 1970s. Any change to the signal format must be backwards compatible with legacy receivers.

There have been some efforts to modernize the GPS signal format. In 2014, select GPS satellites began broadcasting an additional message (CNAV) with a 13 bit week number, allowing a ~157 epoch.

> What's the reasoning for the current GPS time format and it's resulting issues?

Somebody wanted to save money when building the original satellites in the 1970s.

Related discussion which explains the rollover issue:

https://news.ycombinator.com/item?id=19593227

Dumb question: What is the rollover issue?

Not a dumb question at all.

GPS used ten bits for datetime. With a week counter and seconds into that week since 6th of January, 1980. After 1024 weeks it resets back to 0. This has happened once already on 21st of August, 1999. This is the second rollover/reset of the week counter on 6th of April, 2019.

Newer versions of GPS messages use thirteen digits instead of ten digits allowing for a larger week counter (granting 8,192 weeks), causing the first rollover not to occur until the year 2137.

https://en.wikipedia.org/wiki/Global_Positioning_System#Time...

Instead of adding 3 bits, maybe they should have dropped 3 bits. A rollover every ~20 years is just asking for latent buts to blow up. A rollover every ~2.5 years is just business as usual. Go big or go small. Don't design things to break just after you retire but before your coworkers do.

OTOH, with the current setup a receiver will know the correct date as long as it is turned on at least once every 1024 weeks, since it can detect the rollover. More frequent rollovers might cause issues with timing on devices that are infrequently turned on. It's sort of a moot point for most of us once there's greater adoption of the L2 and L5 CNAV signals— those use a week counter that won't roll over until most if not all of us are dead. Then it can be someone else's problem :)

> OTOH, with the current setup a receiver will know the correct date as long as it is turned on at least once every 1024 weeks, since it can detect the rollover

Actually, probably every 512 weeks, as it needs to know (or assume) with side of the +/- it is on:

* https://en.wikipedia.org/wiki/Serial_number_arithmetic

* https://en.wikipedia.org/wiki/Date_windowing

* https://docs.ntpsec.org/latest/rollover.html#ntp_pivots

I believe that's what GLONASS used as a strategy:

Roll over so often that firmware/software implementations pretty much have to take it into account.

> GPS used ten bits for datetime.

Nit: GPS uses 29 bits for "datetime".

10 bits for the week since the epoch, and 19 bits to count the number of 1.5s intervals that have passed since the start of the week. Yes, 1.5s intervals.

See Section 2.3.5:

* https://www.gps.gov/technical/ps/1995-SPS-signal-specificati...

I had to look it up as well. I found this link: https://www.orolia.com/resources/blog/lisa-perdue/2018/gps-2... From the text comes: "However, the field that contains the week number is a 10-bit binary number. This limits the range of the week number to 0 – 1023, or 1024 total weeks."

https://news.ycombinator.com/item?id=19593227

The 787 didn't even exist when the GPS time last rolled over, so it's quite unusual to see it have this problem --- unless Boeing reused the software from a previous version, which is not surprising given the regulations of the industry, but that would imply other older models also have the problem.

> The 787 didn't even exist when the GPS time last rolled over, so it's quite unusual to see it have this problem

It's not as unusual as you'd think:

https://blog.fosketts.net/2019/04/06/gps-time-rollover-failu...

There you see a nice summary of the problem and an example a Mercedes car from, if I understood, 2008 experiencing the same type of problems. Also discussed on HN:

https://news.ycombinator.com/item?id=19593227

It's like having a Y2K bug software and saying "but that software didn't exist before 1900." It doesn't matter when the "epochs" are, it's just about what the design covered or didn't.

I've personally fixed the trading software which failed to correctly implement the correct rollover of the decades (i.e. the transition from year 9 to year 0 in the names), not to mention Y2K. It was simply tested before only "in the same decade."

It's definitely a failure in the development process, for the software that is supposed to work in some bigger time frame, and not only shortly.

I'm rather interested in what kind of software developed after Y2K still exhibiting Y2K bug, and how it affects the usage. Is it made with the assumption that it's going to record only post-Y2K dates, and somehow user manages to input pre-Y2K date?

A lot of software was never fixed for the Y2K problem. People just went from e.g. entering a date of 99 to 00 as the new millennium rolled around, and dealt with the pain of any time offset calculations being incorrect and needing to be manually fixed for a few months.

Some of it's doubtless still in use and people are entering "19" as the year today, and if it's still in use in 2099 we'll have the exact same problem all over again with the exact same software.

That may seem crazy today, but people in the 1950s would have thought you were crazy if you said B-52s were still going to be flying in 2020.

There's doubtless going to be people still running Windows 95 on some emulator in 2100 to use some legacy application that keeps chugging along. Some software we use today might be in use for thousands of years or more, unlike airframes it isn't subject to any natural decay or wear.

> It's definitely a failure in the development process, for the software that is supposed to work in some bigger time frame, and not only shortly.

This is why I've recently started abstracting time/date calls in all my work so that I can test using arbitrary dates/times rather than using the various "now" library calls directly.

This has already had the side-benefit of allowing me to deal with stuff like the time arbitrarily going backwards when NTP synchronises the clock. I don't typically rely on wallclock time for sequencing, this just keeps things a little more robust.

You don’t have to write that code yourself, you can just use libfaketime https://github.com/wolfcw/libfaketime

Can someone give me a source of it?

Reply on news.ycombinator.com