The July Galileo Outage: What happened and why | Hacker News Reader

The July Galileo Outage: What happened and why

berthub.eu

185 pointsshannietron59 comments

The most interesting part of the whole write up is that it sheds light on how dependent of the base stations these constellations are. I used to think that, other than sending orbital corrections and the occasional fixes, the constellation would run itself. Apparently this is not the case.

So a localized outage can in fact affect the entire planet. This has also implications in disaster scenarios.

Not sure how applicable this is for GPS.

This is one of the reasons for the ridiculously high perimeter security around Schriever AFB in Colorado, which runs the GPS network, and a number of other critical DoD satellite programs. It's guarded as thoroughly as the Pantex plant in Texas.

The GPS constellation is considerably more robust to a ground station outage. If the ground station goes out, the satellites themselves are a distributed network and they use each others' data to correct themselves. It can operate in this mode for something like 60 days before it's unable to continue correcting itself.

Of course, if the ground station failure was a soft failure, and instead of ceasing to upload, it began uploading incorrect data, the GPS location results would be arbitrarily bad to unusable. It's not clear to me whether the Galileo outage was because of a hard ground station failure or a soft one. But given that one of the contributing factors was that the backup system was not online, it would indicate to me that this was a hard failure that GPS would have been able to correct.

The other factor for the US GPS system is that there are actually over 24 ground stations, one for each time zone plus at least a couple of spares. A local failure in one time zone doesn’t take out the whole fleet — they can fall over to a backup ground station, and backups for the backups, etc....

Just wanted to give you a little bit more info. I work for the Spanish company that has developed the OSPF, GMV, and although I'm not currently involved at all in this project, from what i know, the following statement: "around 5% of the Galileo capacity is lost to software problems likely in the Orbit Synchronization Processing Facility (OSPF), run by GMV." is not fully true. The OSPF receives data from sources (From my limited knowledge, time stations in order to generate the ephemeris), and it looks like the problem was not in the OSPF code but in one of these time stations sending the data.

Hi - thanks for this (author here). What I mean is, currently, around 5% of the satellites on average broadcast NAPA status. I did not refer to the big outage with this statement, I mean to ongoing developments. Or perhaps you are referring to the same thing & in that case it is entirely possible that the NAPA flag gets raised because the ephemeris is being generated from stale data. Do you know? Thanks!

So that's 1 satellite on average. Even though 5% means the same it makes you think the number is much larger if you do not constantly think of the total available.

author here - if anyone has any questions or remarks, let me know!

No question, but thanks for the research and article!

thanks!

Thanks for writing this.

The title doesn't mention that Galileo is a Global Navigation System (GPS).

GPS refers specifically to the US system.

Indeed, the second paragraph spells out that this is a Global Navigation Satellite System (GNSS) and places it in context with GPS, GLONASS, and BeiDou

Unfortunately, anyone who knows what the acronym GNSS stands for probably also already knows Galileo is such a system. So changing the title to 'Galileo GNSS outage' probably wouldn't clarify matters :(

"Galileo" can stand for a lot of things.

Which is funny, because I've heard it both ways.

For a while, as new systems were coming online, GPS was being used as a generic term, and the US system was being referred to by its original name, NAVSTAR.

But only purists would do that, and everyone else kept saying GPS to refer to the US system, so the new term GNSS was invented as the generic.

I try to say NAVSTAR and GNSS to avoid the ambiguous GPS, just like I say "gridiron" and "soccer" to avoid "football".

But only purists would do that

Perhaps the same sort of space nerds who still call Dish Network "Echostar."

/Waves hand.

Or track the ISS as Zarya.

/waves a trusty Omnipoint phone

That's because Galileo is a Global Navigation Satellite System (GNSS).

Saying Galileo is a GPS is like saying Target is a Walmart.

I gather the US GPS system has "Autonav mode" where each satellite has long-term ephemerides, allowing 60 days of reduced-precision operation if the ground segment fails. Does Galileo have no such thing, or had the errors propagated into such data by the time someone sounded the alarm?

The 60 day long-term ephemerides were the older sytem. Autonav on the current GPS satellites is considerably fancier - basically, the satellites carry out ranging measurements between each other and compute their own updated clock and ephemeris solutions on the satellites themselves in a distributed fashion. Specced to provide essentially full accuracy for something like six months, minus ionosphere estimates which can only be measured from the ground.

As far as I can tell, Galileo has nothing like this. They have a small buffer of future ephemeris data, but the ground segment failure here was so long that it completely ran out. Their approach seems to be full accuracy or why even bother.

Any chance we'll see WAAS reference stations with more autonomy for uploading ionsphere correction data directly to the constellation, instead of having to aggregate at master stations, which then forwards to ground uplinks?

Why would you want that? For slightly more accuracy in a severe ground station outage?

Seems only marginally worth it...

More redundancy, less single points of failure.

> As far as I can tell, Galileo has nothing like this. They have a small buffer of future ephemeris data, but the ground segment failure here was so long that it completely ran out. Their approach seems to be full accuracy or why even bother.

It's probably easier to say "our system being down won't be catastrophic" when it's one of 4+ systems rather than one of one, like GPS was.

Also probably not inaccurate to say that the US military cares more about what happens after the nukes start falling than the ESA does.

Hi makomk, (author here)

Indeed, Galileo has nothing like this. The GPS solution sounds exceptionally fancy, and it may go a long way to explaining where all those billions of USD/year go :-) Do you know if it has ever been tried for real? Thanks!

Doesn't sound that fancy... Orbital calculations are hard maths, but when you've written code to do it, that code can run anywhere, including on the satellite. Very little compute power is required.

Also, satellites are in direct view of one another and can easily receive each others signals. A rather simple software radio could receive the signals from a basic omnidirectional antenna. The only challenge is subtracting out the very strong local signal before digitising, but considering the very strong coding gain in GPS, it should be doable.

> A rather simple software radio

I'd think that would be rather a phase accurate software radio.

Aren't all software radios phase accurate...?

I mean the system was designed as a military tool, with the primary scenario of surviveability being nuclear war, so the level of redundancy seen makes sense to me.

I'm not actually sure if it's been used for real, and I don't think it's really been needed - apparently it's rare for GPS ephemeris data uploads to be late by even an hour or two. In principle the idea is that it would be operating all the time and give slight accuracy and integrity monitoring improvements, but that doesn't seem to be a priority.

Incidentally, it seems like the EU is looking at developing their own version of Autonav for the next generation of Galileo, quite possibly with optical rather than radio links between the satellites.

Nice article! I used to work in this space (was formerly in charge of GNSS hardware at my company), and it's neat to see Galileo being put through its paces.

I see now that the US gets to take GPS (and the fact that the Air Force pretty much runs it single-handedly) for granted. We are seeing _some_ problems (like the GPS III block upgrade and OCX, ahah), but the org chart seems to be considerably simpler than what Galileo's got to deal with.

The operational and management aspects strike me as what happens when you have too many cooks in the kitchen - if everyone is responsible for everything, no one is responsible for anything

Is 1cm accuracy part of Galileo system already operational? If so, why our phones and other devices cannot utilise it yet?

Galileo has like a dozen different signals and I believe that most of them aren't broadcasting yet. I'm not sure which of those signals is supposed to give 1cm accuracy, but it could be one of the encrypted signals. Another consideration is that the 1cm accuracy would only be possible with fancy DGPS or RTK setups. 1cm accuracy is doable with GPS today, but unless you're a farmer or land surveyor you probably don't own any hardware capable of realizing that accuracy.

I suspect this is an extremely naive question, but why would one signal give a 1cm accuracy and another one not? Shouldn't the satellite just tell you its trajectory + identification info so you can calculate where you are based on that? Or perhaps by signal do you mean the physical electromagnetic properties (frequency etc.) are unsuitable for high accuracy, rather than the actual information content it's supposed to carry?

AFAIK each satellite runs two atomic clocks one being much more accurate. Completely different from GPS and way more advanced.

Unsure how it works singal-wise. I assume not only you need different receiver, but also its hardware has to have much higher clock rate.

Interesting, thanks!

To get good precision you need to remove the effect of electrons in the ionosphere. These basically change the index of refraction and therefore the propagation speed of the radio waves. The delayed signal makes your receiver think it is further away from the satellite than it actually is. One way to estimate this delay (instead of relying e.g. on the correction signal of a differential-GPS base station close to you) is to receive signals from the satellites at two different frequencies. The time delay is proportional to f^-2. Knowing the frequencies and the delay between the two signals allows you to estimate the electron content and and correct both signal to the undelayed "infinite frequency" arrival time.

Ahh I see! So they're referring to signals of different frequencies? Makes sense, thanks :)

It's a problem in the US. The Galileo high accuracy service (HAS) is broadcast on 1278.75 MHz (the E6 signal). But the 1240 to 1300 MHz band is not allocated to satellite radionavigation in the US. It's allocated to radar with a secondary allocation for amateur radio.

Because of that allocation, the FCC cannot guarantee that Galileo HAS receivers will not be interfered with.

https://docs.fcc.gov/public/attachments/FCC-18-158A1.pdf

orbits in space are geometrically perfect, you can reasonably talk about 1cm accuracy there.. On earth, variations in a long list of things, including physically moving land (due to seismic activity), make "1cm accuracy" something that does not actually hold.. no doubt it has a lot of positive measuring ability .. but the real truth gets messy..

The "instantaneous violations" link is dead -- I was hoping to learn what that means!

back up, thanks! It is on our experimental node. What is means is: the Galileo service definition regards a 'dilution of position of >6' to be problematic, but only if this happens for more than 77% of the time. The instantaneous bit is "well it is >6 NOW".

Maybe instead of 24*7 work, some manager thought it was 24/7 work, and sent everyone home.

we shoot for π uptime, or around 22/7...

Eight 8s

When was the last time GPS had issues like this? I can imagine that during the early years this happened there as well? Or maybe not, as military quality engineering requirements were stricter.

Or rather silence requirements were stricter. The military is not in the business of publicising failure and weaknesses.

This is a bit of the interesting aspect to this failure. Official channels have overall been pretty weak on details on what all happened in this error. Most public knowledge of what happened has been gleaned from the signals coming off the satellites themselves which anyone can see. This live data combined with understandings of how the system should work is most of the public knowledge of what has been going on.

I don't have a ton of knowledge into the deep technical aspects to GPS, but I imagine we would probably have some similar clues of an outage of this scale in GPS. Maybe a little less technical details of what is happening behind the scenes, but knowing a high percentage of satellites entering a no guarantee precision mode should be possible.

Exactly. And the galmon network already does that. (It should really be renamed gnssmon at this point, it's far outgrown its Galileo-watching roots.)

You can run a station yourself if you have a Ublox 8- or 9-series receiver.

As an amateur radio operator interested in space signals, it definitely intrigues me. Does this galmon project collect data collectively as a wider analysis of the satellite network, or does operating a node only really benefit the group operating a node? I'd love to operate a station to analyze GNSS traffic.

Perhaps in the cold war era.

In the current age, where GPS can be post-processed precisely enough to measure things like land subsidence, there are networks of base stations at fixed locations recording GPS observables and providing historical archives [1] - some operated by US government agencies, some by private companies, some by universities and suchlike.

Due to the use of GPS for things like aircraft navigation, the government also issues 'NANUs' ('Notice Advisory to Navstar Users') and provides an online archive of them [2]. So we can say with some certainty that GPS SATELLITE SVN31 (PRN31) WAS UNUSABLE ON JDAY 349 (15 DEC 97) BEGINNING 1300 ZULU UNTIL JDAY 349 (15 DEC 97) ENDING 2200 ZULU.

Someone who wanted to find the largest recent GPS outage could download the archived NANUs and parse them.

[1] https://cddis.nasa.gov/Data_and_Derived_Products/GNSS/RINEX_... [2] https://navcen.uscg.gov/?Do=gpsArchives&exten=txt

My current impression from reading this article is that SpaceX could create a better GNSS than Galileo in a year as a side project - not because of their cheap launches, but because of their diy development culture. I am stunned by the level of organizational inadequacy of allowing such critical parts of the system to be developed by not just a contractor but a chain of subsidiary contractors, with feedback cycles that move at the speed of corporate communication. My expectation now is that Galileo will never catch up to GPS and will eventually be forgotten.

Perhaps a good lesson that the opposite of “move fast and break things” is “move slowly - but still break things”.

Move slowly and be unable to fix things fast.

I think you're being downvoted because you made a pretty amateurish mistake: on HN you're supposed to say that you yourself can create a better GNSS over the week-end.

Galileo is a strategic asset in case of dispute with the US. It is not in a competition with GPS, it simply has to function as designed and continue to operate independently of US desires. The US objected to the construction of the Galileo system.

The US did not object to the concept of Galileo, the US objected to the EU's choice of frequency, which made it impossible to jam Galileo without jamming GPS. The EU changed the design to avoid this, and I am not aware of any follow-up objections.

Reply on news.ycombinator.com