How Precision Time Protocol is being deployed at Meta

I would be really curious to drill down into one of these problems. Is super precise really the solution?

I would also love to see an explanation of “why do we need this much accuracy?” that actually goes through the derivation of how much accuracy you need.

Some of the justification for Google’s TrueTime is found in the Spanner docs:

https://cloud.google.com/spanner/docs/true-time-external-con...

Basically, you want to be able to do a “snapshot read” of the database rather than acquiring a lock (for reasons which should be apparent). The snapshot read is based on a monotonic clock. You can get much better performance out of your monotonic clock if all of your machines have very accurate clocks. When you write to the database, you can add a timestamp to the operation, but you may have to introduce a delay to account for the worst-case error in the clock you used to generate the timestamp.

More accurate timestamps -> less delay. From my understanding, less delay -> servers have more capacity -> buy fewer servers -> save millions of dollars -> use savings to pay for salaries of people who figured out how to make super precise timestamps and still come out ahead.

This kind of engineering effort makes sense at companies like Google and Meta because these companies spend such a large amount of money on computer resources to begin with.

It seems a little like it might mean that software errors caused by race conditions can be reduced by making the timing windows smaller. As this is a complex area (it might not appear so, but it is) the pragmatic solution could be to reduce the windows rather than fix the issue (maybe an FB engineer can speak OTR).

> How many things need to worry the latency of signal travelling ~300mm?

Arguably every program? The slowest part of modern programs is memory access. L1 cache memory access is ~1 nanosecond and RAM is ~50 nanoseconds.

Is 49 nanoseconds a lot? No, if you do it once. Yes, if every line of code pays the price.

The way I read it: a larger "Window Of Uncertainty" impacts performance. Having a smaller WOU by using PTP gave them a 100x perf increase in their experiments with the commit-wait solution for "ensuring consistency guarantee."

I'm completely ignorant on the topic in general though... so I'm probably missing something. =)

I have never seen NTP in a datacenter of reasonable size get below an error of 1e-4. PTP without custom hardware can easily get 3 orders of magnitude better.

First, 300mm is not the real measure in practice for the common use case. PTP is often used to distribute GPS time for things that need it but don't have direct satellite access, and also so you don't have to have direct satellite access everywhere.

For that use case, 1ns of inaccuracy is about 10ft all told (IE accounting for all the inaccuracy it generates).

It can be less these days, especially if not just literally using GPS (IE a phone with other forms of reckoning, using more than just GPS satellites, etc). You can get closer to the 1ns = 1ft type inaccuracy.

But if you are a cell tower trying to beamform or something, you really want to be within a few ns, and without PTP that requires direct satellite access or somet other sync mechanism.

Second, I'm not sure what you mean by special. Expense is dictated mostly by holdover and not protocol. It is true some folks gouge heavily on PTP add-ons (orolia, i'm looking at you), but you can ignore them if you want. Linux can do fine PTP over most commodity 10G cards because they have HW support for it. 1G cards are more hit or miss.

For dedicated devices: Here's a reasonable grandmaster that will keep time to GPS(/etc) with a disciplined OCXO, and easily gets within 40ns of GPS and a much higher end reference clock i have. https://timemachinescorp.com/product/gps-ntpptp-network-time...

It's usually within 10ns. 40ns is just the max error ever in the past 3 years.

Doing PTP, the machines stay within a few NS of this master.

If you need better, yes it can get a bit expensive, but honestly, there are really good OCXO out there now with very low phase noise that can more accurately stay disciplined against GPS.

Now, if you need real holdover for PTP (IE , yes you will probably have to go with rubidium, but even that is not as expensive as it was.

Also, higher end DOCXO have nearly the same performance these days, and are better in the presence of any temperature variation.

As for me, i was playing with synchronizing real-time motion of fast moving machines that are a few hundred feet apart for various reasons. For this sort of application, 100us is a lot of lag.

I would agree this is a pretty uncommon use case, and I could have achieved it through other means, this was more playing around.

AFAIK, the main use of accurate time at this level is cell towers/etc, which have good reasons to want it.

I believe there are also some synchronization applications that have need of severe accuracy (synchronous sound wave generation/etc) but no direct access to satellite signal (IE underwater arrays).

Probably high frequency hedge funds because being first matters a lot. That 300mm is the difference between winning and losing, it’s pretty binary up there.

Most professional network cards have PTP support and a grandmaster is cheap enough or your colo provider will provide PTP as a service.

Disclaimer: I work in finance, and have for 15+ years.

> Given the special hardware needed for the grand master clock to get down to nanosecond time scales, I'm doubtful this will be used in most data centres of most corporate networks.

The "special hardware" is often just a gps antenna and a PCI card though. In fact, many tier 1 datacenters actually provide a "service" where they'll either cross connect you directly to a PPS feed from a tier 0 grandmaster time service or plug your server into a gps antenna up on the roof. It isn't really that exotic. For financial application, especially trading ones, syncing a LAN timesync to a handful of nanoseconds is doable and optimal.

It is just a matter of time before non-finance sees reasons that better timesync is useful. Precision Time Protocol aka IEEE 1588 was released in 2002 and IEEE 1588 version 2 was released in 2008. This isn't exactly a new thing.

With the right hardware and a tier 0 timesource, modern ntp on modern hardware with modern networks can keep a LAN in sync subsecond. However, as a protocol, NTP only guarantees 1 second accuracy.

> But given that NTP (either ntpd or chrony) tends to give me an estimated error of around (tens of) 1e-6 seconds

Is that a hard bound or an average? If it's an average, then what are the limits of the bounds, both in magnitude and duration?

> and PTP can get down to 1e-9 seconds

We use it for audio, and the reason it works well there is because there is no exponential backoff with your peers, allowing for even small timing slips to become enough to notice. 1ms of latency is far too much for our application, we typically aim for 0.25ms.. and we're only running 96kHz. If we lose PTP sync, we notice within a few minutes.

Another advantage of PTP is it can operate as a broadcast and, as the article notes, switches can be PTP aware and help update the timing as the broadcast flows through the network. Conveniently, PTP also allows for multiple timing domains and masters to co-exist on the same network.

It's also an absurdly simple protocol, you can build a receiver for it in about 200 lines of C code. I've actually become quite taken with it since it was forced into our space about 10 years ago.

> But given that NTP (either ntpd or chrony) tends to give me an estimated error of around (tens of) 1e-6 seconds, and PTP can get down to 1e-9 seconds, I'm not sure how many data centre applications need that level of accuracy.

I know that in trading, auditing trades / order books requires extremely accurate timing, and they typically deploy GPS hardware to get the required level of accuracy. As GPS is accurate to a level of 30ns, 1e-6 to 1e-9 (1ns) is exactly the kind of improvement needed to not need GPS hardware anymore.

Thanks for sharing! It's really relevant and a nice overview of the problem.

Just because high precision tasks don't care about time of day as much, PTP still uses TAI timestamps. The only time of day information missing is leap seconds, and those were a mistake that is being corrected. You don't need NTP for time of day when PTP is used.

I've also seen seismologists use it to sync geophones and other instruments. Synchronization is very important for doing time sensitive measurements for things like localization of earthquakes.

There's other places that are working on this although they probably don't pay as much as Meta. CERN and a few other european physics projects having been using this to create the white rabbit project over the last decade or so ( https://en.wikipedia.org/wiki/White_Rabbit_Project ).

If this is the only thing stopping you, you should apply and tell the recruiter you would like to work in infra. People that don't want to work on product stuff don't get "roped into it", I know people who have worked at Meta/FB for years and have never touched user-facing code or ads stuff.

At Meta you choose what team you want to work on during a 3 month “bootcamp” period, very different from Google, Amazon, etc. There’s enough teams solving hard distributed system problems that you’re guaranteed to never need to think about button copy

This is the same as joining the Sinaloa cartel as a transportation and logistics expert specializing in detection evasion research and development. So many exciting and hard problems to focus on!

We're doing some interesting things with time sync at AWS, and we are doing so at scale for millions of customers around the world, unlike the small scale stuff that Meta has announced here. Most recently, our team recently announced the Amazon NTP Pools service at time.aws.com.

We've got more cool stuff we are working on.

If we can get past this hiring freeze problem, we still have a couple of slots we need to fill. If you know a lot about time sync and the NTP and PTP protocols, and you have a passion for building some of the biggest impact systems on the planet, let me know.

Facebook in particular is very open about its data center, see https://www.opencompute.org/

See “Commoditize Your Complement” - https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/ , https://www.gwern.net/Complement

> Why do so many tech companies seem to be releasing “secret sauce” for free lately?

Is it really their "special sauce" though? Do these types of releases actually give away how these companies make money?

In this particular case, telling the world how to get to nanosecond levels of timekeeping doesn't really help any competitors take away Metabook's revenues or profits.

Maybe it's the usual reasons that we see companies releasing stuff, not only secret sauce?

I think usually it's for company PR for various purposes (counteract bad press, attract new hires, etc.).

Sometimes to generate a bigger hiring pool that knows the stuff you're releasing. (And the open source story about crowdsourcing contributions, which sometimes might be worth the costs.)

I've also seen it around partnerships and customer collaborations and competition. Including to "commoditize your complement", or to kill one thing with what they'd rather use. (And, in industry/tech standards, corporate representatives often have motivation to try to bias the standard to their employer.)

In some cases, it's for individual employees' careers. Think how academic and some R&D jobs want research publications, or how some companies want people who do "talks".

Sometimes also for getting code/docs public, so employees can still use it when they leave.

The question for 2023 is "How many companies will be investing in this without a clear revenue stream?"

It's quite likely we're entering a period where the current baseline performance of core infrastructure will be considered "good enough" and companies won't employ people to work on these general improvements.

PTP isn’t secret sauce. Routers I worked on were doing PTP in like 2009. 1588 was standardized in 2002.

1) it makes the engineers happy to be able to speak about the interesting projects they are working on and give back to the wider community.

2) it serves as a venue for attracting other talented candidates who are likewise minded on working on technical problems.

3) when I was employed of FB it was a relatively flat hierarchy, which is to say there weren’t that many higher ups to convince that this should be done.

It's an effective marketing campaign for "you should work here" - VERY effective.

And lots of this stuff is NOT secret sauce, it's basic business building-blocks that they need. It's not the advertising formulas.

I'm sure FAANG is very VERY happy that they can just run Linux everywhere and don't have to pay Sun or Microsoft a massive per-CPU fee for everything they do.

There is no 'secret sauce' here. The real 'secret sauce' stays secret. The goal of articles such as this one is mainly recruiting, of the 'look at what we're doing here, you should totally want to work here' variety, and it is very effective.

Resume building for people to get their next job?

Lots of these companies opened up the ability for internal engineers to write tech blogs for the purposes of recruiting since the FAANGs were all in competition with one another. Presumably, the VPs haven't gotten around to closing the pipeline yet.

I suspect performance metrics

Many higher-level engineering positions have performance reviews with metrics that include some sort of industry expertise or external collaboration.

Agreed. I wonder if the motivation is competing for resources with Meta paying more attention to costs now.

Yes! I had no idea Meta actually invested in innovation/engineering like this. I always assumed most of their work was just in ad tech & "metaverse".

I don't see the code rearming it at every send nor it matches my experience. The code sets the sock option lazily, and only once . The option is needed to configure the driver and possibly the card. Then the timestamp option need to be passed to the send/recvmsg call to phisically retieve the send/recv timestamp.

Far for being absurd it seems a very straightforward extension of BSD socket API.

Having rooftop mounted GNSS receive antennas for GPS+GLONASS is extremely common in telecom and ISP infrastructure applications. It's sort of a belt and suspenders approach to obtaining time from low stratum NTP sources and also having a local GNSS timing source to reference from.

Or for use in a case where the network has a total absence of connectivity to any internet-based NTP sources (maybe because your management network doesn't talk to the internet at all, for many good reasons), and in the event of loss of transport connectivity to your own somewhere-in-the-region NTP servers in your management system, you want to be absolutely sure the system clocks on your local radio transport equipment, DWDM equipment, metro ethernet stuff are all precise.

Using receive-only GPS data as a master time reference source is effectively saying "we think that getting the time as it's set by the USAF/Space Force people at Schriever AFB, who run the GPS system, should be treated as our master point of reference and all other clocks should derive from that". It's not a bad policy, as such things go, because GPS maintaining extremely accurate time is of great interest to the US federal government as a whole.

Even a fairly small LTE cellular rooftop site, monopole or tower often has a similar receiver. It doesn't add a lot of cost.

Accurate delay compensation is necessary to enable redundancy. IF you need multiple GrandMasters at different locations in the facility, using independent RFoF systems and antennas, they will have different GNSS delays, and that difference will propagate down into uncertainty on the hosts. There are other ways to eliminate this uncertainty than to characterize the full delay, but if the racks are on opposite ends of a giant data centre, that might be as or more difficult than just going through those motions.

If you just have one GM, then sure, the delay means you will have a larger fixed offset from TAI/UTC, but that won't be consequential, and you'll still get the benefits of a tightly synchronized monotonic clock. Until that GM fails, and it all goes haywire.

All recent data center switches I've seen advertise PTP; maybe it doesn't work when you turn it on though. Meta makes their own switches so they can enable and debug the software.

That’s a very interesting question.

1ns of error equates to ~30cm of error (from speed of light), so:

You might get a couple of ns of error from the initial s and p waves. (Assuming on the order of 50cm displacement)

If you’re on the fault and there is a major displacement (say order of 1-2 meters) there might be several more ns of error.

Practically, this seems to suggest that by the time that displacement from an earthquake would cause a significant loss of clock accuracy (say > 10ns), you most likely have bigger problems to worry about..

Yeah, Spanner and the Truetime API. Would be interesting to know whether there are many differences between the implementations (or indeed the abstractions offered). I always wonder whether there are rare occasions where such systems fail (e.g. the bounds reported don't actually capture the true time due to some bug), and what the resulting implications are for systems built on top of them.