Google Spans Planet with GPS-Powered Database (2012)

The recent paper on Spanner has some details of how it is used if you are interested.

https://static.googleusercontent.com/media/research.google.c...

They have attached atomic clocks to their servers let alone gps

Take a look at Spanner paper for details. There’s a set of time master machines per datacenter that have GPS receivers attached, and a set of “Armageddon masters” that have atomic clocks attached. Regular machines (including the Spanner servers) run a time slave daemon that talks to the master machines.

If you want consistency in a distributed system, you have to establish a consistent ordering of events across the system. In absence of synchronized clocks you bounce back and forth logical timestamps, essentially numbers increasing whenever an event occurs because you only really care whether A happend before or after B, not how much time elapsed in between. This adds latency for the additional communication and usually scales not well with the number of processes you have to synchronize. See for example Lamport or vector clocks.

With accurate enough clocks you can establish a global event order simply by time stamping events with the actual time. If you want to process events at really high frequency, you need a correspondingly accurate synchronization between the clocks. The idea is decades old, I guess it is more of an engineering challenge. With the number of systems Google operates at least a few things will break all the time and you have to be able to tolerate that. But I think I just skimmed the paper when it came out and might therefore be unaware of some important details.

GPS is "just" each satellite sending out a constant readout of its current time, very precisely. By reading these from several satellites it's possible to figure out what the actual current time is and then from that determine the time of flight of each signal. Which gives you the distance to each satellite, so if you know where each satellite is you can figure out your own position too.

The point is, it's time based, and it enables synchronization of clocks across the globe to sub-microsecond accuracy. Google uses that feature for keeping its distributed databases synchronized as well. It's not just an alternative to NTP, it's a way to keep systems in sync to a very, very precise degree. Which requires constant communication with time reference servers. They don't put a GPS receiver in every server, just a redundant cluster of them in each data center.

Consider a transactional write using Spanner, the way it works is that the system connects to all of the relevant database shards, it knows what all of their clock accuracy levels are and so it knows how long it needs to wait to negotiate a transaction lock. This is only possible due to the high accuracy of time synchronization. Meanwhile it's buffering write data/instructions to all the hosts. Once it gets the lock it then sends out commit instructions. The whole process can take mere tens of milliseconds for a handful of hosts and only a few hundreds of milliseconds for hundreds of hosts. Which is remarkable when you consider the scale of the problem.