One second per second is harder than it sounds

GPS NTP server can be obtained for as low as $600 or so. So those are becoming not as exotic now.

NTP is good for doing time sync across the WAN, but to get the most out of it for certain applications you need to hook it up to reference clocks.

So, NTP can use PTP as a reference clock, and Bob's your uncle.

I need to have an accurate clock to within a few seconds when I sign Amazon S3 files. I have code that gives access to restricted files on S3. To give access, the customer makes a request from my server, I authenticate, sign a new URL and redirect them to the signed S3 URL.

For security reasons, the URL gets a signature that expires after a few seconds. The expiration date is based on X seconds from my servers clock. If my servers time differs from S3 server time then, according to S3, the link could be expired before it's even created.

In theory, a redirect takes a second or less to complete so you'd think an expiration that's 5 seconds in the future would be enough. The reality is, over time, the server clock can differ as much as 1 minute or so from S3's clock. So, I just set the expiration to be 120 seconds into the future and call it a day.

But to give a practical example. On something like facebook, if you receive two comments on a post within a millisecond, how do you determine the order that they appear in the list? Each request may hit a different datacentre in the world, yet somehow the machines determine the ordering, and every request sees that ordering from then on. You can't just have a single server, because that couldn't handle the load for the entire world. These kinds of things require very strict temporal ordering, which is why time is such an important thing. This leads to the CAP theorem, and distributed systems in general.

I work in broadcast automation. It's a distributed system and the clocks need to be within a certain tolerance of each other. Historically that's done by having a dedicated piece of timekeeping hardware that dictates the station clock, but recently we've been experimenting with using NTP.

We have one node (A) where you author time-sensitive events, and another node (B) that polls for pages of those events, loops through them and takes some action when they should occur. Without both nodes agreeing somewhat on the time you'll end up missing events.

Clock minutes out of sync: "Here's a temporary security token, I don't think it's expired" "That credential has expired" "This never came up in testing, I think I'll crash"

Clock seconds out of sync: Log files on different servers can't be merged reliably. Different requests went to different load-balanced servers and now you can't tell what order the logged events happened in.

Or at least that's how I remember my experience managing an unstable (as in ddosed, every irc service in active development, a few in-house bots, etc) IRC network and having read the IRC and IRCv3 specs and unreal's and TS6 server message specs. If I'm wrong I'm sure someone here could correct me.

Well they don't strictly require them, but having that capability increases efficiency, robustness and safety.

They need to be accurate because electricity is fast - so a lot of protection schemes need to have a decision made across very long distances (which breaker, at which point on a 100 mile long transmission line should be fliped to isolate faults or stop "bad things" propagation) with minimal impact to the system as a whole.

They also need to be accurate because a good measure of frequency and phasor at geographically disparate points allows system tuning. A tiny change - e.g. speed up one of the 100s of generators could result in a much smoother, overall more-efficient transmission system. Similarly, adjustments to capacitor banks and transformers can provide slight phase adjustments for optimal power flow. But these resources are hundreds of miles apart - e.g the western half of the US and Canada (and parts of Mexico) (except for texas) is one big system.

Being a large, physical system, with feedback, and also being electricity, you get some very interesting effects to try and reason about. There can be very slow (relatively) oscillations in the system, but at the same time you have many, many high frequency noise signals mixed in to the elecrical signal. To find trends and patterns for the slow oscillations you need a very high sample rate to determine what noise to ignore. And these samples need to be time aligned, because even a perfect wave will have different phases at different points along it's propagation.

Anyway - that's an example I'm familiar with. I hope it made sense, seeing as I may not have had quite enough coffee yet.

Here's an interesting paper describing the software implementation of certain features that are usually handled by the hardware (well, by software running on the baseband processor): http://people.freebsd.org/~sam/FreeBSD_TDMA-20090921.pdf.

Eventually I had ntpdate running every minute via cron. Pretty annoying.

At this point I'm thinking I need to run ntp on the physical hardware with the least jitter, then just run ntpdate in a cron job on all the VMs. This would work better than ntp clients in the VMs or the VM tools.

This, by the way, horrified all our NTPD theoreticians, who made it clear that running ntpdate was going to cause catastrophic things to occur in our Operations environment - but, given that our logs were all getting progressively more and more useless as we were unable to correlate times for events between servers - the worst case scenario (in my mind) had already occurred.

I don't recall any particularly negative side effects as a result of our ntpdate sledgehammer.

I presume things have gotten better in the last 10 years with VMware.

I'm told the w32time codebase is not something MS employees like to look at.

Performing hard resets of the clock continually is a catastrophic infrastructure failure. A high performance 24/7 application that relies on sub millisecond let alone sub microsecond accuracy across multiple physical systems will be utterly destroyed by this approach.

The problem is when your system clock is so far out of whack (and/or unpredictable) that even ntpd won't help you. Get enough supposedly-identical machines running and this will probably bite you too.

Genuinely curious, is it actually true that you can get sub-100ns time accuracy on a mobile phone? Is this because you have a direct connection with the satellite, which is calculating where you are? I would have assumed that the uncertainty on the latency of any such connection would be higher than 100 ns.

Ironically, often a misconfigured NTP server screws up GPS fixes (it somehow seems to be necessary for assisted GPS, which relies on the cooperation of the mobile network or Internet to faster acquire a GPS fix).

http://stackoverflow.com/questions/8308412/gps-how-ntp-time-...

Half of the GPS PCI devices require you to hack ntpd directly to get anything working, and the Ethernet sources have a strange tendency to go belly up after a year or 2 of running on non-120V sources. Not to mention that if you run the antenna cable too long, you might not notice that your GPS amplitudes are a little low until the next cloudy day. Or if someone adds a new hunk of metal on your roof, or blocks line of sight to some portion of the sky...

Long story short, time is a nightmare and I am glad other people think so too.

You just have to configure an adequate number of upstream time servers, add judicious use of iburst, and you should be good to go.

On my laptop, I usually get good timesync within ten to fifteen seconds of startup.

Second, you're trying to run them in a virtualization environment, and that plays holy hell with whatever good timesync you might otherwise have been able to get. And AWS is probably about as bad as you could reasonably get in this regard -- VMWare has had a long time to get to where they are with regards to timesync for client OSes, and that's not particularly good.

So, your best bet may be to run NTP clients on the Windows machines, but to set them up to run at an artificially high rate of updates. Or maybe even run ntpdate periodically against good real-hardware time servers that are available to you.

You're starting off with both hands tied behind your back, and that's a hard place to come back from.