Nice writeup. I disagree that we need chip-scale, atomic clocks. My idea was a dedicated, battery-backed piece of hardware that reliably stored time plus could sync other machines. Plugs into an interconnect with ultra-low latency. One for each datacenter.

You can plug them into each machine in the cluster periodically to sync them. Or you can plug it into a master node that connects with low-latency management interface separate from main data line. Occasionally, time server gets exclusive access to that line, assesses latency, and then syncs its time. Time server might be custom built to avoid its own skew or keep one of the timekeeping devices attached. Those are periodically shipped to a central location to resync themselves against an atomic clock or each other.

What yall think?