Simple Precision Time Protocol at Meta
engineering.fb.com
engineering.fb.com
A big problem with the PTP unicast mode is an almost infinite traffic amplification (useful for DDoS attacks). The server is basically a programmable packet generator. Never expose unicast PTP to internet. In SPTP that seems to be no longer the case (the server is stateless), but there is still the follow up message causing a 2:1 amplification. I think something like the NTP interleaved mode would be better.
It seems they didn't replace the PTP offset calculation assuming a constant delay (broadcast model). That doesn't work well when the distribution of the delay is not symmetric, e.g. errors in hardware timestamping on the NIC are sensitive to network load. They would need to measure the actual error of the clock to see that (the graphs in the article seem to show only the offset measured by SPTP itself, a common issue when improvements in time synchronization are demonstrated).
I think a better solution taking advantage of existing PTP support in hardware is to encapsulate NTP messages in PTP packets. NICs and switches/routers see PTP packets, so they provide highly accurate timestamps and corrections, but the measurements and their processing can be full-featured NTP, keeping all its advantages like resiliency and security. There is an IETF draft specifying that:
https://datatracker.ietf.org/doc/draft-ietf-ntp-over-ptp/
An experimental support for NTP-over-PTP is included in the latest chrony release. In my tests with switches that work as one-step transparent clocks the accuracy is same as with PTP (linuxptp).
https://signalsandthreads.com/clock-synchronization/ --quote "So, we’re trying to build a proof of concept. At the end of the day, we sort of figured, “All right. We have these GPS appliances.” We talked about hardware timestamping before on the GPS appliances, and how they can’t hardware timestamp the NTP packets, so that’s problematic. We thought, “How can we move time from the GPS appliances off into the rest of the network?” And so we decided that we could use PTP to move time from the GPS appliances to a set of Linux machines, and then on those Linux machines we could leverage things, like hardware timestamping, and the NTP interleaved mode to move the time from those machines onto machines further downstream.
The NTP interleaved mode, just to give a short overview of what that means… when you send a packet if you get it hardware timestamps on transmission the way you use that hardware timestamp is you get it kind of looped back to you as an application. So I transmit a packet, I get that hardware timestamp after the packet’s already gone out the door. That’s not super useful from an NTP point of view, because really you wanted the other side to receive that hardware timestamp, and so the interleaved mode is sort of a special way in which you can run NTP and that when you transmit your next NTP packet you send that hardware timestamp that you got for the previous transmission, and then the other side, each side can use those. I don’t want to get into too much of the details of how that works, but it allows you to get more accuracy and to leverage those hardware timestamps on transmission." -- end quote
WhiteRabbit and the like allow you to transmit such a signal with great accuracy.
There are timestamping layer 1 switches and NICs that support taking it as input.
Facebook has little concern for traffic amplification that doesn't affect them. I can't find a source article for it now, but there was a time when you could take down a website hosting an image by simply posting <URL>/?<RANDOM>. I believe Facebook's (many) cache servers would individually make requests to the server until they inevitably saturated the image host's connection. I remember people complaining and it falling on deaf ears.
given how this industry protocols work, is likely that other big corporations that run data centers are also part of the real protocol discussion, some of those will be corncerned about traffic amplification
* https://datatracker.ietf.org/doc/draft-mlichvar-ntp-over-ptp...
* https://datatracker.ietf.org/doc/draft-mlichvar-ntp-over-ptp...
FWIW, from "50 years later, is two-phase locking the best we can do?" https://news.ycombinator.com/item?id=37712506 :
> TIL there's a regular heartbeat in the quantum foam; there's a regular monotonic heartbeat in the quantum Rydberg wave packet interference; and that should be useful for distributed applications with and without vector clocks and an initial time synchronization service
Reminds me of a project idea where you list out all the big companies that have GitHub projects like Comcast, Walmart, Verizon, Target and even https://github.com/mcdcorp
Afaik https://en.wikipedia.org/wiki/Apache_ZooKeeper started at Yahoo
https://vespa.ai/ was Yahoo's search engine for news and other content product, now spinned off (https://techcrunch.com/2023/10/04/yahoo-spins-out-vespa-its-...)
Yeah I looked it up. Quest costs Meta $5.7bn for $1bn in revenue.
https://www.fool.com/investing/2023/02/15/metas-next-profit-...
There was a time where I'd see Coca cola advertising on Facebook. That time is gone.
Last post from coca cola is August 2022 https://m.facebook.com/profile.php/?id=100064359142431
The number of top tier brands engaging is going down
The fact that you don't see Coca Cola ads means that Meta is able to find advertisers willing to pay more than them to reach you that you are more likely to convert immediately on.
People will always complain how something doesn’t work, how they pulled out their ad money and etc. But then you see ad spend growth on every big platform.
The rule for selling ads is: Know your users.
The rule for buying ads is: Know their users.
Summer child.
No one who has add space to sell wants an efficient, accurate market. Those inefficiencies create competition, create volume create profit for those with ad space to sell.
The ad market had more effective targeting and better ROI 10 years ago than it does today.
Go and try to run a CPA campaign at any cost... you can't. It's all display add's CPM garbage, carpet bomb for pennies.
when Facebook actually did do this, Cambridge Analytica happened which is how they were forced to let 'higher order' (i.e. shady) players have the tweak-able ad-targetting marketplace they know and love
The rapid innovation is a survival tactic. They know the ads boat is sinking, and unlike other tech companies, they don't have much diversity in their revenue. Hence the Metaverse, AI, etc. which although neat, are not exactly making the same level of money for the organization (at least not yet). In Q4 2023, ads had a revenue of ~$38B, while the Quest revenue was a loss of $4B. AI hasn't been directly monetized directly, so it's harder to say how that's doing.
Given how critical good data is to a model, I'm not optimistic this will work for them.
Their dead-set focus on data collection and advertising is sabotaging their ability to make (potentially billions in) revenue from traditional models. I know Zuckerberg and many other CEOs want their "core business" to be just one thing with all the other businesses being offshoots of that one thing but the reality is that they've become too big for that. Zuck needs to give up on the idea of, "our business is data collection and targeted ads for consumers" and realize the truth: Their business is technology.
alas, microsoft owns that space
Microsoft's operating overhead with Xbox is also vastly greater than Steam. Supposedly they make ~$28 every time they sell an Xbox One. That's based on just the manufacturing/parts cost of the hardware and doesn't include the costs associated with developing the hardware itself where they don't just take off-the-shelf chips and throw in an existing OS (like the Steam Deck) but instead custom-engineer a processor/architecture and make their own custom operating system.
what I was trying to say is that microsoft owns the developer space, Valve has been a tough contender but also, microsoft has never gone straight against valve probably because the business wigs consider videogames less important than microsoft's other businesses.
so I should have said that while Valve may own the marketplace (the "app store") microsoft still owns what it takes to make a game in the first place. which is why facebook doesn't really stand a chance against MS. this also explainss how it came to pass that nobody cared about zuckerberg's metaverse... the metaverse didn't get access to the really cool graphic engines
I think the whole point of XBOX, GAME PASS and all that is to convince people who don’t play games (stock market analysts) that Microsoft is relevant. It’s vice signalling.
On the other hand, those LBEs have an antagonistic relationship with headset adoption. If everybody had a headset than there would be nothing special about MR experiences. For LBEs to be viable you want headsets to be capable and inexpensive but not widely adopted. (I almost wish it could be Winter 2024 forever) I'd imagine a headset vendor would like to charge me more for using a headset for an LBE than they would want to charge a ordinary user but on the other hand people who are blown away by an LBE (very possible) might go home and buy their own headset.
As for their vision, Meta seems to be doing really well running an app store for single-player games. I haven't seen a real multiplayer hit yet but I guess Demo Battles comes close. Meta knows what they'd like to do if they could create something like OASIS from Ready Player One but a close analysis of how Horizon Worlds falls short of that reveals how difficult that is. I guess anybody who can afford a seat of Dassault 3DExperience can also afford an AVP, maybe many Blender users can afford an MQ3. It's not clear to me at all what, past games and entertainment, is going to be a mass market in XR.
I bet Microsoft would hate this
GE and Boeing also had amazing YoY growth and great quarterly reports, until the underlying dumpster fire they were nurturing for years exploded in their faces and now they don't have growth nor profits anymore.
Are you saying people like Facebook's new features like the suggested posts and what they've done to Instagram? Was their VR product line secretly a success?
These things eventually catch up.
There's also a number of fairly bad bets they've made, such as the $1 bln kustomer acquisition they spun out at a 75% loss after a year https://techcrunch.com/2023/05/16/kustomer-meta-spin-out/
They have enough capital and trust to stumble around for a while but they sure look like they're yahooing
Sure, the company may not exist 10 years from now. But there's no downfall indicator yet for this trillion dollar giant. All companies that size have headwinds and tailwinds. The self-assurance you see on this platform for Meta's sure shot upcoming decline is just absurd.
This brand looks extremely vulnerable
Meta too sells the lie that its products are paid for by ads. Its products are paid for by surveilling users, building behavioral profiles from the data that is collected, and then giving other companies access to that behavioral data in order to manipulate users to specific ends. In this quest to build better behavioral profiles, the products are made to be as addictive as possible, eating away people's time which could have been utilized in objectively better ways.
I've never convinced someone of flaws in ethics by framing the perpetrator as a big bad boogeyman to the nth degree. It's unproductive self-satisfaction.
[1] They sell visibility to people queried against proprietary behavioral data.
https://www.afterbabel.com/p/the-teen-mental-illness-epidemi...
The 2022 article on the Precision Time Protocol says (https://engineering.fb.com/2022/11/21/production-engineering...):
- Adding precise and reliable timestamps on a back end and replicas allows us to simply wait until the replica catches up with the read timestamp...
- As you may see, the API doesn’t return the current time (aka time.Now()). Instead, it returns a window of time which contains the actual time with a very high degree of probability...
Which sounds similar to TrueTime (https://static.googleusercontent.com/media/research.google.c...):
- A read-only transaction executes in two phases: assign a timestamp sread [8], and then execute the transaction’s reads as snapshot reads at sread. The snapshot reads can execute at any replicas that are sufficiently up-to-date...
- TT.now() returns TTinterval: [earliest, latest]
I tried Googling "Precision Time Protocol TrueTime" but the only reference I could find is a HN comment by someone else from 2022 :) https://news.ycombinator.com/item?id=33696752
PTP do wallclock time.
Spanner, rather famously, uses this range approach, but a good number of other systems are based on similar approaches. The important thing for reads is getting an upper error bound from the clock, having storage that can perform reads at that time (eg using MVCC), and having a way for storage to know when it's seen all writes before a timestamp.
This can be implemented using PTP, and that's the approach we use at AWS inside some database services.
Now, independently of replica strategies, it's important to understand TrueTime is an API, to be clear, as you noted. It lets you represent some continuous interval of time based on the system clock error. You can then use this API to do things like ask "Did timestamp A occur before B?" And you can get an API equivalent to TrueTime on your own random Linux machine, using the Clock-Bound tools from AWS, combined with the chrony NTP daemon: https://github.com/aws/clock-bound
The API and all that is pretty basic, actually. Rather, the secret behind TrueTime and the like is just a huge amount of reliability engineering to ensure that the upper bound target (7ms IIRC from the Spanner paper) is actually maintained reliably and accurately, at global scale. That reliability means engineers can build on it with specific guarantees. You can slap chrony, ClockBound-D, and a PTP card into your rack and program away. But it's a matter of engineering guarantees more than like, theoretical computer science. Theoretically speaking, TrueTime can only help you definitively establish that some event A has actually "happened before" B in a distributed system. That's extremely powerful but it needs muscle backing it up to be true and useful in practice.
AWS has publicly advertised that EC2 has access to their 'AWS Time Sync' service, which is a globally consistent clock synchronization service designed to provide the backbone needed for services like TrueTime, and is freely available. Assuming you are willing to trust the EC2 network and AWS engineers, you can slap chrony and ClockBound-D on your AWS instances and get a TrueTime-like API with very tight global error tolerance, which would allow you to do consistent read replicas like Spanner, among other tricks
It is worth keeping in mind that very, very, very few people/companies are required to solve the same problems a company like Meta is.
I love it. These sound grand.