Look Before You Leap – The Coming Leap Second and AWS

I don't think the leap second bug had any effect on EC2 itself; the only reason EC2 was ever blamed is that many of the sites which had issues happened to be hosted on EC2.

http://blog.fastmail.com/2012/07/03/a-story-of-leaping-secon...

And lets not forget:

3. http://www.quirksmode.org/blog/archives/2009/04/making_time_...

Case in point: the Linux leap second kernel bug that made problems in 2012. Read the commit of the fix:

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux....

"This patch tries to avoid the problem by reverting back to not using an hrtimer to inject leapseconds, and instead we handle the leapsecond processing in the second_overflow() function. The downside to this change is that on systems that support highres timers, the leap second processing will occur on a HZ tick boundary, (ie: ~1-10ms, depending on HZ) after the leap second instead of possibly sooner (~34us in my tests w/ x86_64 lapic)."

So the bug was the effect of the programmers more worrying that the leap second adjustment happens as fast as possible, in 34 microseconds, ignoring that the call from that particular point in code made the livelock.

And instead of pushing the change "as fast as possible" we see that both Google and AWS solve the problem by spreading the changes over the long periods of time. Which is the right approach generally for all automatic adjustments to the system clock -- avoiding discontinuities.

There was an interesting HN thread along these lines back when the current impending leap second was announced: https://news.ycombinator.com/item?id=8840440

I personally implemented it 12 years ago, to support (a) a kind of geo-balancing based on time coordinated spoofed DNS responses, and (b) time stamped URL token expiration validation, sensitive to seconds, across distributed datacenters. I got the idea from Microsoft Windows' ability to drift the clock back into sync.

Google has adopted or reinvented a lot, and gets credit because they have extra time and resources to publish.

// TBC, they also invented a lot. But not this one. Sibling comments point out other implementations predating Google as well.

Here's some examples I thought of that may help clarify:

1. "Each second is 1/86400 longer and AWS clocks fall behind UTC. The gap gradually increases to up to 1/2 second."

- AWK clocks do indeed fall-behind when their "seconds" tick longer. Because their seconds tick longer, over a fixed period of time, AWS will count less ticks than UTC. (Think of it this way: A mile is longer than a kilometer. After you have traveled 800 kilometers, you've only traveled ~500 miles.)

2. "AWS clocks gain 1/2 second ahead of UTC." - Before the addition of the leap second, AWS clocks are behind by 1/2 second as per (1). The addition of a leap second to UTC is another second that the UTC clock must tick - the AWS clocks don't have to tick this amount - so the AWS clocks are now ahead by 1/2 second.

Basically, this is what AWS is doing, using our distance analogy again:

1. Normally, we have to cover 1000 km every day. This includes a Civil group of travelers and an AWS group of travelers.

2. Today, we decide we're going to cover 1001 km. Everyone in Civil decides this is OK to count to 1001 instead of 1000, just for today. But the AWS group only wants to count to 1000 because counting to 1001 is a Very Bad Thing. However, the AWS group still has to cover 1001 km.

3. The AWS group comes up with the ingenious idea of just making each "kilometer" 1.001 real-world kilometers, just for today. Thus, they will only count to 1000, but each time travel 1.001 km. The end result is the same - they will have covered 1001 km.

  Leaving this message in place as an example of how  
  thinking about time and calendar-related programming is 
  easy to mess up.
		

However, I do wonder how they implement this. Do you change timekeeping in the kernel so that a fraction of extra time must pass for each second to be "counted"?

http://lxr.free-electrons.com/source/kernel/time/ntp.c?v=2.6...

    Clock: inserting leap second 23:59:60 UTC

http://man7.org/linux/man-pages/man2/adjtimex.2.html

The stock linux kernel works such that if ntpd will have set the STA_INS flag via adjtimex some time before, the kernel will do the leap-second insertion at the end of the UTC day.

If you disable ntpd and it doesn't reset this flag (which I doubt it does, but you'd have to check), the kernel will insert the leap second on its own, even if ntpd is not running.

If you disable ntpd, and either ntpd on termination (which I doubt), or you via the adjtimex syscall, clear the STA_INS flag, then the kernel will not insert the leap second. After UTC midnight, the clock will then be one second off, and a restart of ntpd will slowly steer the clock back to correct time.

For playing with all of this, there's a adjtimex tool which can display and even change the timex values:

    ➜  sbin  ./adjtimex -pV | sed 's/^/    /'
             mode: 0
           offset: 0
        frequency: -344608
         maxerror: 16000000
         esterror: 16000000
           status: 64
    time_constant: 2
        precision: 1
        tolerance: 32768000
             tick: 10000
         raw time:  1432018768s 308111us = 1432018768.308111
     return value = 5

In the most basic case, code to fetch a date n days ahead that naively assumes fixed length days will return a date n-1 days ahead instead if it runs at exactly midnight eg date("Y-m-d", mktime()+($days*86400)) will return a date $days-1 in the future if it runs on leap second day. An edge case certainly, but if you're adding millions of records a day something you ought to consider.

It affected pretty much everything running (a recent version of) Linux.