Hacker News Reader

Comment by jedberg | Hacker News Reader

I don't think a negative leap second would cause much of an issue at all. The reason a positive leap second was such a problem was because it add an "invalid entry", ie. 23:59:60.

But a negative leap second would just be skipping a valid entry, which happens all the time if you're using NTP to set your clock.

Sure, date math libraries would need to updated, but they already got updated for the positive leap second to have tables they can look up to find out which seconds to add, so adding an entry for a skip isn't much different.

Most of the havoc of the positive leap second was in time series data having that invalid entry. That won't happen with a negative leap second. There just won't be a value there.

View on HN

> I don't think a negative leap second would cause much of an issue at all.

But nobody has tested it in the wild. It's entirely possible to trigger an existing bug in the code base. I expect some disruption from the first negative leap second event. After that everything would probably be gradually stabilized.

It's constantly tested in the wild. Every time your computer does an NTP update and has to jump ahead one second or more, you've just tested it.

> Every time your computer does an NTP update and has to jump ahead one second

PCs don't run anything crucial, usually you can set your clock to 2001 without visible problems, the same is not true for a server. A server uses ntpd, not a bruce-force ntpdate. Under ordinary conditions, ntpd adjusts the clock in small steps so that the timescale is effectively continuous and without discontinuities. But I think a negative leap second is quite violent.

I was talking about ntpd. It would do its update after the negative leap second and then think to itself "oh, I'm more than one second behind now, I better start fuzzing and catch up".

It's rare for ntpd to step time forward, unless ntpd lost synchronization and the local clock has drifted beyond +/- 128 ms by default. On a 7x24 production server, it's very unlikely, time is always slewed. But a standard leap second implementation requires the addition or deletion of time via adjtimex()'s TIME_INS and TIME_DEL, it's not adjusted gradually.

> "oh, I'm more than one second behind now, I better start fuzzing and catch up".

No, ntpd would say, oh, leap second, now it's the time to tell kernel to delete a second! And if the kernel's negative leap second code path has a bug, kernel panics and everything crashes.

...Yes, you probably can force ntpd to ignore the leap second deletion and also force NTP to slew time only. This approach is similar to leap second smearing (which is considered a violation of standard but nevertheless an useful hack). But it requires manual configuration, by default the standard approach is used.

So the bottom-line is: Time will be forcefully stepped ahead by 1 second (which is an uncommon scenario on production server), and, it would not just be an ordinary step - it uses the operating systems normally untested negative leap second code path. I'm not sure what the consequences would be for various services, perhaps not much, but my worry is the interactions in a complex application can produce unexpected outcomes. Also, how many systems have implemented the negative leap second in the code path? And are they all tested?

Well, not really. Clock skew is tested, but the code paths in adjtimex() for handling a leap second are different.

This is why the 2005 leap second crashed every Linux 2.4 kernel worldwide as I recall.

Thankfully there's been a push toward leap smear (certainly on every system I run) which should obviate this type of problem entirely.

Does NTP do that, though? I was under the impression it made time slightly faster/slower until the point is met, if the difference is small (which it will always be on production servers, in principle)

ntpd does indeed skip time, contrary to popular belief. I didn't believe it myself until one day I studied the documentation and code.

Check out the "-x" flag in the manpage.

I recommend dumping ntpd and using chrony instead, and also configuring chrony for a time smear rather than the leap second as required by the standard.

> ntpd does indeed skip time

The documentation also indicates it's rare for a 7x24 running server to suddenly step time (it only occurs when there's a 128 ms difference), unless you get serious network congestion that prevents sychroninization for an extended period, so a time step in a production server is uncommon. But a standard-compliant negative leap second requires time stepping, it's why I said there might be disruption.

Yes, but the documentation is wrong in its assumption. There are many reasons time may skew >128ms beyond "network congestion."

I've dealt with datacenters getting a bad ntp config push, for example, or a bad firewall ruleset which blocks NTP. Fixing the config may inadvertantly step time.

There's a whole host of systems issues, sometimes involving VMs and sometimes bare metal which can cause this type of problem. The most simple of which is ntpd not running properly for a while, but this can also happen when systems get overloaded - extreme swapping and the like.

ntpd's behavior of non-monotonic time adjustments was brought to my attention after a developer uncovered database corruption, caused by a database which (correctly) assumed that time should be monotonically adjusted while the database server was running. I believe it mis-ordered a transaction log or something along those lines.

There are other, better ntpd implementations like chrony which can be configured to never step time. If time needs to be stepped, applications should be stopped and the machine should be removed from service first.

Thanks for sharing the experience.

> There are other, better ntpd implementations like chrony which can be configured to never step time.

I totally agree that ntpd is worse, it has an aging code base with many potential security issues, chrony is a better and cleaner implementation. Nevertheless, I refuse to refer to "ntpd" as a worse implementation because it implemented the leap-second as the standard specified. Also, ntpd can be manually configured to never slew time (excluding leap second) if -x and -g are used.

> ntpd's behavior of non-monotonic time adjustments was brought to my attention after a developer uncovered database corruption.

BTW, my original topic here is how a negative leap second may cause service disruption, ntpd's behavior is only a footnote to that discussion. I think your experience only strengthened my original argument: if the standard leap second implementation (instead of smearing) is used, which is the case in a default configuration, something might happen. But I'm not sure, a negative leap second doesn't break the monotonic assumption. So it's all speculations...

a skip ahead is different than the officially being at 23:59:60, albeit quite many NTP nowaday use leap smearing to avoid that https://developers.google.com/time/smear

> But nobody has tested it in the wild.

The FreeBSD (kernel) folks test their code for it:

> I run the freebsd kernel and ntpd through "negative" leap second events usually a couple times a year as part of testing our timekeeping products at $work. I'm skeptical that we'll ever see a negative leap, but we have some customers who insist on having it demonstrated to them as working correctly.

> […]

* https://lists.freebsd.org/pipermail/freebsd-stable/2020-Nove...

Of course then there's userland…

Great, at least we know the kernel is probably good (assuming Linux gets tested too and at least the latest version doesn't have bugs)... But I still worry about the interactions in a complex userspace application can produce unexpected outcomes. And I wonder whether Linux previously had bugs (too lazy to trace down the git history today), and if there was, how many are still in deployment...

What I don’t understand is what the world gains by this level of global software time accuracy that couldn’t be localized to the very specialized cases that actually need it. Why can’t the world generally drift a second here and there over a few years and super time sensitive programs keep a tally of leap seconds to adjust? Sure it’s a pain in the ass for those applications, but isn’t “we don’t know for sure if this time adjustment is going to disrupt major world affairs” more of one?

> The reason a positive leap second was such a problem was because it add an "invalid entry", ie. 23:59:60.

It's only a problem if you have a hardcoded assumption that all minutes have exactly 60 seconds. You'll encounter the same problem if 23:59:58 is followed by 00:00:00

Computers don't usually count time in this manner and the display is converted via mathematical formula. It'd just have to be adjusted to account for leap seconds, positive or negative.

> You'll encounter the same problem if 23:59:58 is followed by 00:00:00

On the contrary, I would expect skipping a second to happen all the time from the perspective of userland programs, due to common things such as blocking system calls and program scheduling under load. Also features such as sleep and hibernate can easily skip entire days.

That's not really the type of "skipping" that is the issue.

Software tends to ask the system the current time (via a library), and the system responds with a "here is a number, it is the number of [units] since [epoch]". The library then converts that number using a formula to work out a date/time. If it is decided a leap second is added or subtracted, and the library's formula doesn't account for it, then the time will be calculated incorrectly.

For most software it is unlikely to matter too much, but this sort of thing can lead to unpredictable emergent behavior in complex networked systems.

A negative leap second, unaccounted for, would cause anything scheduled during that second to be skipped. Probably not Y2K catastrophic, but certainly could be a problem.

Reply on news.ycombinator.com