How a leap second caused havoc with the Internet (2021) | Hacker News Reader

How a leap second caused havoc with the Internet (2021)

talesfromtheopsside.com

40 pointsfrantic282112 comments

A couple years ago we had issues caused by a 0 (zero) leap second, which honestly I couldn’t find documented anywhere but hacker news! Something about the system expects a leap second every 17 years and if it doesn’t get one it’s more troublesome than the 0 leap second.

https://news.ycombinator.com/item?id=27944776

One of the more unseen side-effects of this particular bug back in 2012 was datacenter internal temperatures spiking by ~5℃ due to the increased CPU usage iirc.

We've already discussed the Leap Smear, where clocks run 0.0012% slower for a day.

https://news.ycombinator.com/item?id=28047714

https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...

And inevitably there is also Leap Unsmear: https://news.ycombinator.com/item?id=22223591

By Google. Who started this whole trend of leap smearing in the first place.

Google didn't start this... It's standard practice to speed up or slow down the clock during transition in ERP systems.

VAX/VMS are way older than Google.

Actually, before 1972 they would run the radio time signals several parts per million fast or slow to keep in step with the earth's rotation.

I recall this hitting our startup’s various MySQL Linux servers in 2012.

They were slow/maxed out for a while till we figured it out. The DBA rebooted a few but didn’t know why and it kept cropping up on others; I drilled into root cause with some google-fu.

Not mentioned in the article is that I think this issue occurred again in 2015. I don’t recall if it hit us because we hadn’t patched/upgraded some servers since 2012 (!) and had just relied on resetting the clock/rebooting, or there were more Linux bugs (see LWN below).

Good technical writeup from Linux weekly news: https://lwn.net/Articles/504744/

And more on Linux patching in 2015: https://lwn.net/Articles/648313/

IANA canonical list of leap seconds: https://data.iana.org/time-zones/tzdb-2018a/leap-seconds.lis...

> Hany explains: “Due to physics, at certain times of the year, the Earth moves faster along its orbit than others. How do we base time off of this, then?”

Sorry, but no. This is so wrong I don't know where to start.

It's not wrong really, Earth's orbital velocity does indeed vary. It just doesn't have anything to do with leap seconds

TLDR: a leap second overloaded computer systems due to issues in how Linux handles time.

Time to plug my old explanation of leap seconds:

https://robotsinplainenglish.com/e/2022-11-20-stopwatch-time...

> As you can see, UTC is not a proper, continuous time stream. It has breaks in it wherever leap seconds have been added

I can not see that. 23:59:58 - 23:59:59 - 23:59:60 - 00:00:00 is perfectly continuous, proper, and even monotonic, or at least as much as 23:59:58 - 23:59:59 - 00:00:00 is.

> When you receive a package with a computer-generated time stamp, it is showing this fudged UTC number, not the actual number of seconds according to the atomic stream.

I think you are mixing up UNIX timestamps (which are their own thing) and UTC. UTC and TAI both indicate exactly same thing, there is perfect 1:1 mapping between UTC timestamps and TAI timestamps (ignoring pre-1972 era).

Reply on news.ycombinator.com