Hacker News Reader

Comment by Gigachad | Hacker News Reader

What is the consequence of being a fraction of a second out of sync?

I’m really struggling to understand the problem - what does out of sync here mean - especially when you are talking about a max 500ms. ntp clock drift can already be as bad as 500ms and if you need something tighter, then you may have already gone the gps route. Either way I can’t imagine anyone building a system that has less than a 500ms tolerance for any arbitrary computer on the internet. At the very least special relativity is going to be one the first road blocks you face in trying to get perfect universal time sync

> I’m really struggling to understand the problem - what does out of sync here mean - especially when you are talking about a max 500ms.

From a purely academic perspective, it means from noon preceding a leap second to noon after the leap second there's a difference between UTC and what google (and anyone else who does smears) reports as the time. If you really hated yourself and were trying to use timestamps from conflicting sources to order transactions in a database, for example, this could mess up the order of your transactions which could have all sorts of fun consequences.

Algorithms that involve 'i did this thing at this time, everyone else should wait for that time to have passed before continuing to ensure correct ordering' (ie. Many database systems) will dramatically slow down.

Systems that use atomic clocks might have been able to do 10,000 transactions per second will suddenly only be able to do 2 transactions per second if they contain a node that doesn't use the same smearing.

Seems a weird way to write an algorithm. Easier to rely on database locking and/or the obvious fact that if you have received some data telling you about an event having happened, then it has happened… Curious if there are any real reports where this caused issues?

If you have not heard of problems with time in distributed systems everywhere, maybe read up instead of guessing? There are literally thousands of papers on the topic, more every year. It is not considered solved.

But where are the reports/blog posts from the thousands of companies using these algorithms that ran into performance problems the last few times cloud providers used time smearing? I’ve done some searching and can’t find any, which suggests this is mostly a theoretical issue.

Yes I could come up with an algorithm where this is a problem, but in most cases I’d favor a solution that doesn’t depend on separate systems having clocks perfectly in sync.

Okay, how many suggest relying on a perfectly synchronized clock versus how many suggest designs which account for slightly desynchronized clocks?

If you have a distributed system that requires strict time synchronization why do you care if Google’s time is wrong? Surely if you need that level of synchronization you have your own implementation with gps clocks.

If you don’t, then you already have to deal with standard drift with ntp anyways so Googles time smear is the least of your problems

One does not always have the luxury of systems operated entirely within one organization.

If you don't have the luxury of managing all your systems within one organization, then time smear is the least of your problems. Day-to-day ntp drift alone will be a bigger problem.

Refer to “Practical uses of synchronized clocks in distributed systems” for some info on algorithms like this.

https://dl.acm.org/doi/10.1145/112600.112601

Google spanner uses stuff like this with a clock called TrueTime. I don’t know how leap seconds / smearing is handled in that system.

But _all_ of Google's servers will be off by the same offset, which means that they can continue to use their distributed time-based locking mechanism. Generally, if you require that high resolution of time, you don't leave your infra.

Those algorithms depend on their reference time being common across all processing nodes - in fact the only system I know that uses wall clock time for inter node ordering is.... Google Spanner, which runs just fine with their smearing approach.

Reply on news.ycombinator.com