Hacker News Reader

Comment by evol262 | Hacker News Reader

> “Currently, nobody uses time except for maybe Spanner at Google, CockroachDB or someone doing database things,” Rosenblum said. “We believe that there’s a lot more places, especially as more and more time-critical things came up. We can do time sync, since we figured out how to do that pretty well. And so we asked: is this part of a trend where we’re going to start programming these systems differently? And [researchers] got kind of excited about that possibility of us being able to pull this off.”

Who gave this guy who's apparently never heard of kerberos, ceph, SAML, or any other technology the rights to interview? Sure, their windows are larger, but "nobody uses time" is so clueless it blows the mind, and it's hard to imagine who decided that "NTP, but machine learning" is a $21M idea

View on HN

None of the services you mentioned need even close to nanosecond precision between servers to operate. Applications that do (other than the ones in the article like Spanner) are concentrated in supercomputing/scientific computing and spend a LOT of money on the problem. So yeah if you are letting your clocks drift by multiple seconds you aren't "using time" in any real sense.

If these folks can achieve what they want at a couple orders of magnitude cheaper than current prices then you are absolutely going to see a lot more regular use cases show up to take advantage of it.

Telco workloads and 5g require precise time using technologies like ptp. The cheaper, more commodity those solutions get the better.

If only these devices had a clock capable of measuring time to within 1 nanosecond.

If one has such an accurate clock they might be able to accurately plot their position on earth to within 1 ft merely by checking the timed pulses of orbiting satellites.

I guess you're downvoted for being sarcastic or maybe people don't know 20 dollar GPS chips provide a ~10 nanosecond resolution clock

And GPS-disciplined oven controlled crystal oscillators (OCXOs) provide some of the best frequency accuracy and stability around. They range from about a hundred dollars up through a few thousand, depending on phase noise.

Haha I have no idea. But yeah just call the GPS algo a Time AI and voila, $21 m in funding to go get some kids in shenzen to put a GPS chip on a USB, PCIe, NVME slot.

Hey look we got a GPU to learn the Kalman filter algo.

Many are actually better than 10 ns basically a 1 foot CEP is 1 ns. If they are fixed and always on the accuracy it can build over a day is incredible.

granted it does need an unobstructed view of the sky, so getting it on a PCI-e board would be an innovation (:

Makes me curious how accurate a GPS clock is, they must be pretty good given the timing precision needed for accurate positioning.

statum-1 time server

For GPS precision is much more important than accuracy.

Precision vs. accuracy of what measurement?

For GPS to work at all, you need very accurate knowledge of the offset between each satellite's signal.

The only way I can think of to have precision but not accuracy is if you design a system where there's a buffer between the antenna and the signal processing and you don't know how long the buffer is. Is a design like that a practical concern?

Of UTC. Theoretically GPS can work out of sync with respect to UTC as long as all satellites are in sync among themselves.

Oh, you're talking about a theoretical situation where the GPS satellites are reprogrammed to use a different synchronization source? Personally I'd still call that highly accurate, but to a different time standard. But I see where you're coming from.

They're saying that as a receiver of GPS signals, it doesn't matter if your clock is out of sync with everyone else's clocks, as long as it measures the length of a nanosecond accurately.

But GPS gives you a clock signal.

With specialized GPS receivers, about +/-2ns. Closer to 40ns with commercial receivers.

They can't do better than a microsecond or so without hardware timestsmps. There aren't a lot of applications that seem to need anything between milliseconds and nanoseconds. MiFID II did open a nice market in finance, but many of those companies use PTP anyway.

I considered starting a competitor in 2018 when I first saw this company, but I don't have the same connections to customers that these guys do.

My conclusion was that PTP precision with software timestamps would be a good company, but not something between. I hope they can prove me wrong!

My understanding is that PTP is neither necessary nor sufficient. The main advantage of it is that some hardware will only support adding hardware timestamps to PTP messages (ie you want to use the time packets left/arrived at your network card rather than the time you processed them in the kernel/userspace). But I think the specialness of PTP can lead to dealing with a lot of bugs from bad implementations.

> None of the services you mentioned need even close to nanosecond precision between servers to operate. Applications that do (other than the ones in the article like Spanner) are concentrated in supercomputing/scientific computing and spend a LOT of money on the problem

No, not really. I work in finance (HFT, systematic market making) and this industry heavily relies on high precision clock sync. It's even regulated by law (MiFID II); in practice everyone on the street uses PTP because if you don't know how fast you're really going or where your latency spikes you already lost the race, just don't know about it.

There's a couple of other domains where micro- and nanosecond level time sync is of paramount importance. PTP has become much cheaper over the years especially when you are in a data centre and can get it as a service instead of setting up grand-masters, slaves, GPS antennas, etc.

This statement just reads like badly researched topic on their part.

especially the Huygens-implementation delivering accuracy in nanoseconds could be some part of its secret sauce (https://en.wikipedia.org/wiki/Clock_synchronization#Huygens)

> So yeah if you are letting your clocks drift by multiple seconds you aren't "using time" in any real sense.

If you arrive to the railway station 10 minutes +-2 minutes before your train leaves you are using time. In a very real sense. In robotics we sync multiple computers on the robot to about miliseconds, that is using time. In a real sense.

Maybe more accurate, cheaper sync will enable more applications. Maybe it is a good business to specialise on it. But saying that nobody excep for Google, CockroachDB or someone doing database things uses time is bulshit and can be called out for what it is.

> If you arrive to the railway station 10 minutes +-2 minutes before your train leaves you are using time. In a very real sense. In robotics we sync multiple computers on the robot to about miliseconds, that is using time. In a real sense.

Why are you in a very specific technical discussion correcting what is obvious? Yes, you're using time. You're not using time at the precision this company is targeting. At that precision, there are few current use cases, but if they make it cheaper, there will be more. What's there to argue here? You're offended you're not being included as users of time?

"We're lunching more accurate satellite imaged maps. The current users are largely nation states and niche industries due to its cost. This will enable more map use cases"

I use Google Maps all the time. I used it for my last trip! How dare you.

> You're offended you're not being included as users of time?

Please refrain from personal attacks. It is unecessary and doesn’t add to the conversation. Thank you.

> I use Google Maps all the time. I used it for my last trip! How dare you.

Do note that your example didn’t say “nobody uses maps”. If an imaginary salesperson would say “nobody uses maps” I would call bulshit on that too.

> Yes, you're using time.

Great, so the company representative should simply not say “nobody is using time”. And the commenter I was responding to shouldn’t say that applications which are fine with a lower accuracy “aren't ‘using time’ in any real sense”.

Heck, all I know maybe nanosecond accuracy is where the bees knee is, and everyone is going to be amazed by the awesome new applications it is going to enable. But you don’t need to disregard all the history of timekeeping and all the current applications to make that point.

On 28 April 1789 a mutiny broke out on HMS Bounty. The ship’s former captain, departing on a rowboat, demanded that the mutineers give him K2 the ship’s chronometer. The mutineers refused, because the chronometer was worth about as much as the whole ship, and they needed it for their onward navigation. Somebody should have told them that they are not using time in any real sense. Probably they would have laughed at that idea.

What are those applications? They've passed me by unless that's meant to cover large-scale physics measurements of some sort. I don't understand how it would even make sense in mainstream supercomputers. It should be good to synchronize OS scheduling in HPC nodes to minimize jitter in tightly-coupled applications, but that's not ns stuff and I haven't come across it actually being done (though Bull had some support as free software).

I don't see how something like Kerberos or TOTP, that fails without some sort of synchonization, can be defined as not "using time" in a real sense. (If cluster nodes were drifting by seconds, I'd be checking them, even if that was only going to affect something like make.)

Google uses precision time (SunDial) for observability, telemetry, congestion control too: https://www.youtube.com/watch?v=GEXP68yNPPM

Yup. Going back to the principles of distributed systems, I’m really wondering how “NTP but machine learning” can be used to minimize the clock skew to hundreds of nanoseconds with software only. The RTT between two servers can vary a lot — and way past the calculated minimum RTT.

The figures (eg $21M) and names dropped (eg Stanford) are an appeal to authority, which does make me curious.

I’d love to see some papers, I went through all of Balaji Prabhakar‘s publications (titles only), and didn’t see a single paper that sounded like “NTP but machine learning.

If anyone else knows more about this, I’d love to hear from you. Surely $21M doesn’t get dropped with at least someone doing due dill on the tech?

I also haven't seen any work of his using ML, that may just be a buzzword thrown in the PR release. But if anyone's interested, I believe this is the paper alluded to in the article:

Clock synchronization: https://www.usenix.org/conference/nsdi18/presentation/geng

I think the TechCrunch article doesn't really explain their application of clock synchronization well. Here are the other relevant papers and my attempt at explaining the general idea below.

Network edge-based measurement (one-way delay?): https://www.usenix.org/system/files/nsdi19-geng.pdf

Congestion Control: https://www.usenix.org/system/files/nsdi21-liu.pdf

Each paper I've listed builds on the last. Their method to synchronize clocks made accurate and efficient measurement of one-way delay possible with commodity hardware. This measurement of one-way delay occurs at the edge, allowing them to "hold" incoming packets at the edge for extremely small periods to reduce congestion (latency) while maintaining throughput. From my understanding, traditional congestion control algorithms require rich telemetry from the entire network, which is likely not accessible in a public cloud environment. Balaji and clockwork's algorithms only need to make these measurements from the edge (which customers in public cloud have access to).

I'm curious to see how all this will scale for multi-region deployments. If the latency between VMs from region 1 and region 2 is significant, I wonder if the measurement will actually be useful in deciding to "hold" the packets.

This is exactly what I was looking for. Thank you. It’s a shame fundamental publications aren’t part of PR articles. I’d also love to see better details on their website. Oh well, I cant expect everything to work like academia — c’est la vie.

> I also haven't seen any work of his using ML, that may just be a buzzword thrown in the PR release.

The first article you reference includes something about Support Vector Machines (SVMs).

I'm from the Clockwork team, thanks for listing the relevant papers.

Accurate clocks sync enables true one-way delay measurements (instead of RTT/2), this allows for edge-based network visibility. We launched Latency Sensei beta – a sensor, monitor and auditor that provides visibility into cloud deployments. The gallery has cloud fitness reports on GCP, AWS and Azure. Some interesting reports include: 1)how VM colocation impairs network bandwidth, and 2) tale of 2 cloud regions, London vs Singapore. Take a look and we'd love to get some feedback https://sensei.clockwork.io/user/gallery/

On Congestion control, an edge-based solution is coming soon. If you're interested in a private beta, email us at hello@clockwork.io

> Surely $21M doesn’t get dropped with at least someone doing due dill on the tech?

You’d be surprised, I was.

> The figures (eg $21M) and names dropped (eg Stanford) are an appeal to authority, which does make me curious.

They have the 'names' because they are a reputable group of people. They have been at this problem for a while, first with extensive research that required clock sync as a pre-requisite, and then to evolving into clock sync as a formidable problem unto itself.

Clockwork is a rename of the company as far as I can tell. Their original name (Tick Tock Networks) [1] was probably too close to what has become a very popular homophone.

[1]: https://opencorporates.com/companies/us_de/6947372

Sure, but ‘names’ don’t explain how they do what they claim to do. Another reply kindly provided references to publications (which I couldn’t find myself) — they made for great reading.

You seem to know their work, so if you have further publications, I’d love to get them please.

Here's a talk Balaji Prabhakar gave: https://www.youtube.com/watch?v=YQSA99nrFEU. Looks like the clock sync part starts about 30 minutes in.

Haven't watched this one, but might be complementary: https://www.youtube.com/watch?v=Opf9CBwP5R8.

> Surely $21M doesn’t get dropped with at least someone doing due dill on the tech?

Juicero raised $120M.

Juicero only raised $18.5M in their series A.

And?

From the article, time synchronization is only a small part of what they do. Their Big Thing is traffic shaping and latency management within and between datacenters.

Those products are built on time synchronization. E.g., a latency measurement needs a _true_ difference between the timestamps of two servers.

> Surely $21M doesn’t get dropped with at least someone doing due dill on the tech?

Just some words:

Theranos.

Ubeam.

Moller.

And many, many others. Investors do this all the time. FOMO + 'wouldn't it be nice' are powerful ways to part fools from their money.

To be fair, their pitch deck did say both Machine Learning and Stanford

Reminds me of early in my career where I encountered a home made time sync system because the creator had never heard of NTP.

How so? Clearly these guys have heard of NTP. They are talking about ns precision systems, not ms.

20 years ago I went on a ridealong with a bunch of men who were changing the times on the clocks installed in a bunch of church towers.

It was fascinating to get to watch the process. I'd always taken it for granted the big hotels, railway stations, and churches would often have big clocks you could see from a distance and that they'd be self-correcting. But of course a lot of old clocks are purely mechanical so when the time changed an hour forwards/backwards somebody would need to physically change them.

Even the clock in my car (2016) doesn't self correct. It goes ahead of about 1 minute every month.

Exactly! what would Leslie lamport say? ;)

https://lamport.azurewebsites.net/pubs/time-clocks.pdf

> it's hard to imagine who decided that "NTP, but machine learning" is a $21M idea

Bless you. Are you younger than 40? :-)

Tech people who are over about 40 have, alas, no trouble imagining this at all!

Epitome of garbage HN drive-by dismissal: pick a seemingly stupid sentence from the article, quote it out of context, conclude that author/subject expert is clueless about <insert basic things>. Usually just a case study of the Dunning-Kruger effect.

Given how much complete garbage has been VC-funded (or rather, overfunded, since the correct amount of funding should've been $0) it's good to remain skeptical.

If your way of remaining skeptical is opening the HN discussion and jumping into the first crappy hot take concluding that all people involved are clueless idiots without even skimming the article, your skepticism doesn’t mean much.

Example of a garbage HN drive-by comment: this.

I've spent a _long time_ working in distributed systems with components which are "close to the metal". There are use cases for this, many of which are better solved by "hardware timestamping NICs" and "run a stratum 1/2 NTP server with high locality (and if you're geo-distributed, run multiple on systems dedicated to that purpose, because ensuring you're within nanoseconds of _someone else's_ infrastructure is generally an order or magnitude less important than internal coherence).

That was the best possible quote from the author/subject trying to explain the use cases. Did you even read TFA? It's completely unclear from their article what the intended uses cases are, what the problem space is, and how much working knowledge they have of existing solutions.

Naming people who invested in other companies who are investing in this one and showing what looks like a dashboard for a timekeeping solution is normal TechCrunch garbage, but that quote was a step above.

Comments like yours (going all the way back to /.) days really just tell me that you didn't read the article, and it makes it look like your primary goal is sophistry.

Reply on news.ycombinator.com