GPS error caused '12 hours of problems' for companies
bbc.com
bbc.com
https://en.wikipedia.org/wiki/Single-frequency_network
The kit is supposed to accomodate loss of GPS, but evidently doesn't handle error so well: http://www.microsemi.com/products/timing-synchronization-sys...
The first question, of course, is "how do you determine when there's an error?" Half the GPS constellation was broadcasting the data, and for almost all purposes it was 'correct' data, other than being wrong. Perhaps ground stations could determine that they're getting different data from different satellites (if they have some good ones in view), or maybe they could just decide that the new parameters differed enough from the old ones to be implausible, but neither of these is a cut and dry "I definitely have an error" ... but let's say you have a foolproof way to determine that A0/A1 are incorrect. Great. THEN what do you do?
You could just decide to use the previous value for that parameter... which is great for a single clock, but what about clocks elsewhere that have a different value (the one being broadcast by the rest of the satellites) or an out of date value (say, a clock that was powered off while traveling in a broadcast van)? Barring external communication, you could have an arbitrary number of different interpretations of the current time -- a different one for each ground station!
You could also just accept that the new parameters are correct, but how do you then get your clock to the new correct time, without having a second (or other time period) that's significantly longer or shorter than a second? There's a lot of gear that expects a second to be a specific length, which is not gonna be happy if the length changes. (Think of playing back an audio file that's exactly 30 minutes long, and needs to start and stop at specific times, down to the bit. You then configure your system to send one bit every (1/bitrate) seconds exactly. How does that system recover when 13µs of bits go missing? How do you coordinate that behavior over multiple disconnected systems around the world which may have gotten the same information at different times (or not at all)?
From reading the article, it sounds like what happened was that the units didn't fail because of the GPS issue, but that they start throwing alarms (probably while going into a mode where they ran off only their internal (disciplined) oscillator. This seems like absolutely the right thing to do -- "uh, boss, things look really not-right here, and I can't fix it without breaking something, you really need to come deal with this yourself." (Note the comments about 'thousands of system warnings' but none about 'cellphones broke.') This is, from what I can see, really the only reasonable behavior these devices could have.
(disclaimer: I don't have any direct data on what broke vs. what complained loudly and woke people up. I'd love to have some solid, not-parsed-by-the-tech-media data on this topic, if anyone has any.)
However, regarding your question of how to programmatically handle partial failure scenarios, I would reduce the trust in individual components (eg. satellites, receivers, receiver firmware releases, etc.) via redundancy and a voting-type mechanism to evaluate individual component outputs and their changes over time versus probable reality. Such a system would possibly include multiple offline clocks to match prior signals and thus maintain increased fault/drift-tolerance.
https://www.febo.com/pipermail/time-nuts/2016-January/095692...
Position is based on relative difference between time signals received from different satellites, not absolute.
If you did have an atomic clock in every datacenter, you'd still need to make sure that they didn't develop a problem that caused them to drift apart, and also you'd need to make sure that they were in sync with each other in the first place.
GPS is a convenient way to get well-synced precise time nearly everywhere you can see the sky.
(and to find your position you use their also-broadcast orbital positions and math for the effects of relativity etc)
According to https://en.wikipedia.org/wiki/Atomic_clock#Evaluated_accurac... existing atomic clocks can drift by 1 second in 100 million or 150 million years. You won't notice that in your high-speed datacenter applications over the course of a few decades.
(unless perhaps if one is plain broken, and it's noticed by comparing to the others and taken out of service)
Edit: So far as latency - the on-board clocks get resynced today every now and again against a ground-based reference time as they pass overhead of a USAF transmitter (primary + backup). If they used only moderately-accurate clocks as a cost-saving measure (or were at the limit of late 1970's technology) drift would be more severe and the tail-end of each orbit would be less correct than at the start of the orbit. So additional ground stations would be required in parts of the planet not always so friendly to the US, and they would have to coordinate their time sources, allowing for the light-speed delay travelling around the earth. Much more expensive/harder solution from an operational aspect.
Edit Edit: Don't forget that the GPS system was designed around military requirements. All the satellites (even today) have nuclear weapon detonation detectors on board and are a primary input to NORAD.
> The 2nd SOPS has three missions: global navigation, time transfer and nuclear detection.
Based on what others have said and my own inkling I think it's about synchronisation at a fine timescale without having to trouble yourself with bothersome shit like relativity, and leaving that particular pain in the ass to the US military. That's what I meant by latency, but phone keyboard, walking, lazy!
If these people are upset over 13 microseconds, wait until the DoD turn selective availability back on during a war or start adjusting satellite positions to avoid incoming ASAT missiles.
Interesting fact: GPS time doesn't take leap seconds, etc. into account. As of last summer, it's ahead of UTC by 17 seconds.
The other problem is that there's not really such thing as a single, true time. Precision timekeeping is one of those places where relativity matters -- differences in altitude and the geology of the earth under your feet at a particular point change how fast any particular clock 'ticks'.
So to accommodate these differences, if you want events to be synchronized across multiple locations over a long period, you both need a single agreed-upon "current time", and some way to get information about that time information to the sites that need it.
You could surely do this with directly connected wires that were very carefully measured, but that gets expensive very quickly. The GPS system already requires super-precise time just to function, so it makes for a great distribution mechanism of "what time is it" information, where the people running the GPS system are bothered with all the details of figuring out and standardizing a single time, and you 'just' have to listen.
The GPS folks also keep track of things like how fast the GPS standard clock is drifting from the UTC standard clock (since no two clocks will agree over the long term, period). And this is actually where the problem happened -- Two of the parameters broadcast (A0 and A1) were incorrect, and it's these parameters that are used to specify the current time difference between the GPS master clock and the UTC master clock (the GPS master clock is disciplined to be as close to UTC as possible, but is always off by a minute amount, and that's what A0/A1 represent).
Pretty much it boils down to, much as it is in distributed systems, "there is no now". Even with maximum care and the highest quality clocks, there's really not a definition of 'now' that's consistent across more than a single location. GPS is currently the closest thing we have to a globally-distributed definition of 'now'.
Of course, they make use of GPS, but also add their own atomic clocks.
> The key enabler of these properties is a new TrueTime > API and its implementation. The API directly exposes > clock uncertainty, and the guarantees on Spanner’s timestamps > depend on the bounds that the implementation provides. > If the uncertainty is large, Spanner slows down to > wait out that uncertainty.
...so really, GPS and atomic clocks aren't for defining "now" they're for providing a keeping the window of uncertainty around "now" to a small value. The value mentioned in the paper for this is "generally less than 10ms", which is pretty good for a large-scale distributed system, but massive in terms of precision timekeeping. It is, after all, an uncertainty that's three orders of magnitude larger than the GPS glitch that started this discussion...
> atomic clocks in general is that they have very good
> short term stability, but not-so-great long-term
> stability
http://ivs.nict.go.jp/mirror/publications/gm2002/takahei/img... http://www.ke5fx.com/rb.htm http://www.thinksrs.com/assets/instr/PRS10/PRS10diag2LG.gif
By definition, anything steered by GPS will have infinite precision for infinitely long time-scales, as GPS time and UTC time ultimately are steered such that they are within a certain tolerance of each other.
http://www.leapsecond.com/pages/tbolt-tc/
And UTC time is the pretty elaborate average of ensembles of atomic clocks, hence atomic clocks produce the international reference time there is.
You can, of course, improve the long-term stability of any clock by locking it to a GPS (or other means of time-signal distribution). And for a good Caseium that might only require you to look at the phase, and slightly nudge a potentiometer (magnetic correction field adjustment) on the front by a tiny amount every week. But atomic clocks in general are pretty darn impressive.
UPDATE: This presentaiton has all the interesting plots combined, it seems: http://www.hipster.net/ShadNygren_Atomic_Clocks_for_Amateur_... (pages 45 and following).
The particular nit you pick here is a fair one, if you take my comment literally, rather than as the (over?)simplification of a complex concept I was trying to summarize.
Yes, in absolute terms, an atomic clock will have excellent stability over the long term in its own frame of reference. As long as you don't need your atomic clock to match any other clock in the universe, the long-term stability is excellent.
The particular topic being discussed here, though, is equipment that is designed to synchronize events across a large area. Even atomic clocks that are literally 100% accurate will diverge if installed in different locations. So if you want your atomic clock to actually be able to tell you what time it is (in a way that matches up closely with anyone else), you have to constantly be adjusting it to match a common standard (e.g. UTC)
These adjustments are a form of instability, since to correct for changes, you have to make your seconds either faster or slower than they actually are locally, on a constantly-changing basis, or you have to step your time and have less (or more) than (Y - X) time actually elapse between timestamps X and Y.
As soon as you move beyond a single frame of reference, you can't have both short and long term stability.
I suppose I could have used a different word than 'stability', but that seemed to get across the underlying effect best. I suppose another way of saying it (that is more correct) is that atomic clocks are great for telling you how much time has passed (in your local reference frame), but not so great for telling you what time it is. GPS is great for telling you what time it is, but not so great for telling you how much time has passed (in your local reference frame).
(The best clocks for timekeeping, of course, are atomic clocks disciplined by a common reference like GPS, which gives you the option of deciding your own long term vs. short term tradeoff however you'd like. But the original question was "why do we need GPS at all, and not just atomic clocks?".)
Disrupting GPS is less useful for military goals these days, since there are independent positioning constellations, like GLONASS.