Graham: Synchronizing Clocks by Leveraging Local Clock Properties (2022) [pdf] | Hacker News Reader

Graham: Synchronizing Clocks by Leveraging Local Clock Properties (2022) [pdf]

usenix.org

59 pointsmlerner13 comments

I wonder whether there are clock generators with on-chip temperature sensing...

Yes: "an advanced, low-power, high-performance mobile clock generator with four clock outputs. The device integrates a MEMS resonator, temperature sensor and a temperature-to-digital converter (TDC), which eliminates the need for external crystal and temperature-sensing crystal resonators." -- https://www.sitime.com/products/clock-generators/clock-gener...

The paper mentions TCXOs and OCXOs, respectively temperature and oven-controlled crystal oscillator. You can get PCIe cards that have these on them, and devices for audio studios with a rubidium oscillator like the Tascam CG-1800.

Jane Street has a podcast that lifts the veil a little bit on how they keep their gear synced up, which is pretty interesting: https://signalsandthreads.com/clock-synchronization/

The CG-1800 uses "a high-precision OCXO (oven-controlled crystal oscillator)" https://tascam.com/us/product/cg-1800/ only mention of rubidium is: "An external input connector that supports a 10MHz signal enables the CG-1800 to be connected to a rubidium clock or GPS clock for even higher precision."

On the other hand, Antelope Audio 10MX is a rubidium word clock source: https://en.antelopeaudio.com/products/10mx/

Surplus rubidium frequency standards are cheap on Ebay: https://www.diyphysics.com/2012/02/14/d-i-y-10-mhz-atomic-cl...

Cheap second hand rubidium frequency standards may be worthwhile, unlike new rubidium oscillators, which are not that good in comparison with good OCXOs to make their much higher price acceptable.

Traditional rubidium frequency standards still have an aging rate that is high enough to be non-negligible. While from the datasheet of a rubidium oscillator it appears that it is better than an OCXO, that can be misleading, because many OCXOs have very predictable aging drifts. If you compensate the aging in software, you may reach a similar performance to a rubidium oscillator.

In recent years there have been invented several new kinds of rubidium frequency standards, with much better performances than the traditional models. However these are either not available yet as commercial products or they may have a single provider which demands a steep price.

The Jane Street podcast was a very approachable explanation, and was interesting to me for a few reasons: (1) that EU regulation requires their timestamps to be within 100 microseconds of UTC, and (2) they claim to achieve 20 microsecond accuracy using hardware-timestamped NTP synched to local NTP time servers, the NTP servers then being synced to off-the-shelf GPS masters using PTP, (3) they really didn't like the idea of running PTP on all of their switching infrastructure (even though running PTP boundary clocks on all of your switches is the obvious way to go), (4) they weren't doing anything fancy or hardcore. Very pragmatic. I was expecting lasers, custom FPGA systems, and a dev team dedicated to time synchronisation.

They did mention that to go below 20 microseconds they'd need a different approach. And mentioned white rabbit https://ohwr.org/projects/white-rabbit/ From NTP to white-rabbit sounds like a big jump to me.

In the context of digital audio, 20 microseconds is an entire sample period at 48kHz. AVB using gPTP is capable of locking up all devices on the network to some small fraction of a sample period. That requires all network switches to propagate time information. Start here: https://en.wikipedia.org/wiki/Time-Sensitive_Networking

I was personally pretty surprised at the idea that Jane Street found PTP to be too difficult to administer or run. Hardware support for PTP is nearly ubiquitous in NICs and switches (unless you build this gear for yourself, I guess), so it is not that hard to administer. PTP done poorly gets you ~100 ns time sync across your cluster, and if you do everything correctly you can get time within about 10 ns given how small a trading network is.

This time accuracy would need to propagate to all their hosts, not just the ones in a single DC. I presume they have hosts in the EU, London, NYC/NJ, Tokyo, Chicago, etc... I imagine 100ns accuracy with that kind of global installation diversity isn't straightforward.

Yes it is, GPS is 1foot ~ 1ns; thus a few ns are trivial with a vaguely decent receiver.

The gold standard for PTP is to use separate GPS-disciplined atomic clocks in your separate points of presence, and PTP within each one. These are about $10k each, so it is not that expensive even if you have 100 datacenters.

> that EU regulation requires their timestamps to be within 100 microseconds of UTC,

It's the same in the US. It's covered under CAT NMS (Consolidated Audit Trail, National Market System). Probably too much information at: https://www.catnmsplan.com/

Yes, and the fact that this has been a global regulation for years suggests that it isn't all that special. Outfits like Goldman Sachs meet this regulation with fewer blogs.

Nitpick: the C in TCXO stands for "Compensated", not for "Controlled", like in OCXO.

That means that the resonator of an OCXO is held at a constant temperature, while that of a TCXO is at the ambient temperature, but the temperature is monitored and a compensation circuit adjusts the resonance frequency, maintaining it as constant as possible.

From TFA: "Some commercial PTP implementations use packet delay variation (PDV) filters [27], and compensate for known latencies in the receive and transmit paths."

This is phrased vaguely. All gPTP implementations are required to compensate for link delay (not necessarily rx/tx asymmetry).

Reply on news.ycombinator.com