Storing UTC is not a silver bullet (2019) | Hacker News Reader

Storing UTC is not a silver bullet (2019)

codeblog.jonskeet.uk

46 pointsingve64 comments

This article always bugged me, because it needlessly makes things seem complicated. There are times when you are trying to represent a moment in time, and then there are times when you are trying to represent a moment in a calendar (which is not the same as a moment in time). It's all about capturing intent and using the right tool for the job.

The moment in time case is pretty straightforward. The moment in a calendar case can get horribly complex because calendars are... very human constructs, and there are challenges capturing the entirety of human intent (did you mean for the position on the calendar to remain fixed even if the calendar changes? what about leap seconds? etc.), because often humans aren't even sure about their intent.

Yeah I was about to say the same, why make such a contrived example. A much simpler example is the case of iPhone alarms going off at the wrong time.

A different real-world example that I like to give is when I was working at a photo sharing site that had global reach. We had a debate about how to store, convert, and/or display datetimes and I opted for local date/time. Ideally want to store both the local time and place of the photo which allows calculating every other time format. Even without the place, it still makes sense to show night time photos with their local time, and new years celebration photos at midnight.

> So from November 1st 2021 onwards, the UTC offset of each country will be fixed – but there may be countries which currently always have the same offset as each other, and will have different offsets from some point in 2021. (For example, France could use winter time and Germany could use summer time.)

It’s worth noting that this did not end up happening. Maybe a 2019 tag could be added to the title?

That doesn’t affect the point of the article at all though. Timezones change, so if you’re storing a datetime of something that happens at a particular physical place, store it with that place’s timezone (not offset)

(EDIT And even eg scheduled video calls take place in physical places, so in practice future events of any kind are best stored in localtime with the relevant user’s timezone attached)

> scheduled video calls take place in physical places

have you ever travelled and had to join the call from a different country, different time zone? the calendars usually help you to translate the call's timezone to your local timezone, but they only can do that if they can determine the offsets.

Just convert using most up-to-date source time zone into UTC, then proceed as usual. The point is that the event is not actually fixed in time, but affected by local law that can arbitrarily move times of all scheduled meetings by an hour or so.

It would be nice if get get rid of DST so that we wouldn't have to special case situations that only happen once every half century, though.

Yeah so you determine the offset at display time. Store “Europe/Amsterdam”, not “UTC+1” so that if Amsterdam (or likelier, the Netherlands) decides to shift timezone or move the DST change dates or something insane like that, before the event occurs, you can trivially take that into account by updating to the newest tzdata file.

Converting from timezone to offset is a lossy conversion.

The most annoying case is when an event is already scheduled, and the timezone then changes. E.g. it is decided that from now on, DST will end a week later.

What to do with the events that were scheduled in that week before this change was known? One assumption is that the intent was to schedule it at the same time during the business day (the 9 o'clock meeting, lunch is at 12 etc.); but then you would've been better off storing local time with a location rather than even a timezone, as the (lack of) S in the timezone might not have been intentional.

I'm not saying you shouldn't use UTC though, I'm saying daylight savings is ludicrous and should be gotten rid off.

My personal taste is to store a day, a time, and a regional timezone (like Europe/Berlin) for calendar-like uses.

But even then, you'd need to notify those users who now have that in their own local calendars on the other side of the world... assuming they aren't using an electric attachment with the original TZ.

For example, working in a secure environment and putting certain events on a calendar manually on one side or the other.

> even eg scheduled video calls take place in physical places, so in practice future events of any kind are best stored in localtime with the relevant user’s timezone attached

I was in London when I scheduled a video call with someone in New York. When we actually took the call, I was in Spain and he was in Chicago.

I can’t think of any timezone that could have been attached to that meeting to make it useful at the time of creation. I absolutely expect my machine to be able to tell me what the local time of the meeting is when its timezone changes.

I move between time zones a few times a month. At some point time zone changes get annoying. I keep my work laptop and work phone on home time zone, and second phone autosyncs to local time zone.

I track meetings by notifications (2 hour prior etc)

This is possibly a better approach than I’m currently taking, thanks! I’ll try this on my next trip!

That only works if a timezone doesn't split in the future. Like a city in germany changes its timezone. Then the Europe/Berlin you've stored is wrong.

I admit, it unlikely, but it's been known to happen.

But you're storing a timestamp, so you can at least theoretically look up historical time zone rules based on place and time.

The only real problem would be geographic boundaries changing, e.g. two states currently sharing a time zone and DST rules diverging.

That's why the TZ database uses cities, not states or countries, to designate time zone information. It's still not bulletproof (not all cities exist in the TZ database obviously, and collecting that information can be very tricky too), but it's pretty good compared to the alternatives.

The short of it is that using UTC for events in the past is usually OK (you can at any moment recalculate the wall clock time in the past with UTC plus location data), but not for future events (e.g. scheduling, opening hours). Because the intent of stored future datetimes is almost always the wall clock time in the location where the event takes place. If TZ rules change for that location the intent is to pin the event to the local time rather than to UTC. For disambiguation, future events should ideally include location as well as datetime, though this is often omitted / implicit.

The remaining edge cases are future events that span different time zones (e.g. scheduled calls with participants from different TZs). If one of the timezone rules chances, this can only really be resolved by rescheduling and updating the info for the participants.

Jon Skeet is one of the two greatest of my software engineering heroes. The other one is Andrew Hejlsberg.

Just by reading some of his answers on SO, or by reading his C# in depth book, or his talks and blogs, I learned how you can think of scenarios where a design falls short, and to think of pros and cons rather than right and wrong. He has a clear mind on things, and a very direct style of writing.

Did you mean Anders?

My all time favourite programmer - everything he's worked on back to Turbo Pascal has shaped my career.

Darn yes, thanks, too late to edit. Agree about him. He has style. When I was learning c#, every convention, every name made perfect sense.

Same feeling I had back in the 80's as a kid learning Turbo Pascal - he just views the world the way a much smarter version of me would :).

I think the whole article can be generalized as following: one cannot make assumptions about relative values.

You can replace UTC with US dollars and Amsterdam time with euros and talk about conference prices. If you convert 100 euro to US dollars at the time of record creation, the price at the day may not be 100 euros anymore.

Ideally you record a tuple that allows to define value unambiguously (e.g. time 2022-02-21 9:00 timezone Europe/Amsterdam) because in this case you can always resolve the absolute time value later.

This is a typical hiccup for any business expanding into multiple locations with different parameters (time zones, currencies, units etc).

Once case that I find particularly interesting is an event that happens in different locations at the same logical (but different absolute time). Let's say you want to send an email to all your users across the global on Monday 9:00. If you want to add up to complexity, you could think about sending an email to all your users across the globe on the first working day next week.

Storing UTC for future times has never made much sense. But people still think UTC is the silver bullet for storing timestamps, ie. for records of past events. But even this is prone to failure. At some point someone will ask the question "how many events happened before lunch time?" This is a very clear and reasonable question to ask, but when is lunch time?

Unfortunately many implementations simply store the UTC timestamp and throw away any time zone information (e.g. postgres TIMESTAMP WITH TIME ZONE, despite the misleading name). There are two ways around it: store the user's current time zone in another field, or record it in something like ISO8601 as local time with offset (although this only records the offset, not the IANA time zone). Both options have pros and cons.

I wish implementations supported the basic tuple of (timestamp, timezone) where timestamp is a Unix-style timestamp (a number). The timezone can be ignored when it isn't needed, but is always there when it is. The difficulty is how to store the timezone and I'm sure there are numerous other issues with this.

There was some interesting discussion around what data structures you should store at https://news.ycombinator.com/item?id=40933110 (Engineering principles for building financial systems).

Almost always you store some kind of log with timestamps when decisions were made or events generated or whatever.

Is there any library that can translate utc to local time where it addresses time offset changes based on the knowing the utc when the future time was decided upon?

We should just drop all other time zones and use UTC

We should just actually understand the concepts we're working with.

Because making programmers happy is so much more important than making users happy?

Let's go back to each city having its own time. _Because making train companies happy is so much more important than making users happy?_

All jokes and "snarkiness" aside, it's just a matter of habit. If the whole world used UTC and working hours/social time were set differently per-country, I bet we'd get used to it eventually.

I don't know, you'd still need to coordinate based on local time, let's say I want to set up a meeting with colleagues from different time zones at 10. I have no idea what time of day that is for them, are they sleeping, having lunch, etc. With time zones, I can look up the time offset, with universal time I'd have to check their location and what time of day it is for them, which could be confusing, how do you map it exactly, bed time, lunch time, one hour before lunch time, two hours after bed time?

The other problem is that I don't think habits would be easy to change. Right now people in some european countries want to stay permanently on summer time to have more daylight after work. We could just as easily stay on winter time and have everyone go to work earlier, shops could open at 7 instead of 8, etc. But for some reason this isn't considered, so I assume it's easier to enact all year DST than to change working hours.

Another thing is that having the change of calendar date in the middle of your waking hours is rather annoying and potentially confusing, and everybody in the world not living near wherever the meridian ends up will be subjected to that. (Because in that case a single calendar date can refer either to solar date A in the afternoon/evening, or solar date B in the morning).

Like should public holidays e.g. start and end right in the middle of the solar day because that's when midnight UTC happens locally? Do you reintroduce time zones by the back door by defining a local start/end time for day-based events to align them back to the solar day?

You are joking, but I think City-based timezones would work well on today's internet. We have better than ever computers to do time math and scheduling/coordination for us, let's leverage that. It's easier than ever to go to more timezones rather than fewer, because we don't have to make train companies happy anymore and because some people would get better sleep if their local time was closer to their solar time and because DST is easier to get rid of with smaller timezones. (DST is, among other things, a hack around our timezones are too wide.) The IANA timezones are even already primarily city-based, so a lot of software wouldn't even need to update anything but their IANA tzdb.

Now might be a great time to do something like that. It doesn't sound like a serious proposal, but there would be interesting benefits to city-based timezones again in the modern world. We actually have the technology to scale smaller timezones now. We aren't relying on mechanical clocks like the train companies were and paper calendars like early multi-timezone companies like GE were. We could make local time resemble solar time a lot more again, and solve it as a software problem.

Not just developers but everyone.

The world has changed and we often collaborate globally.

I remember we had todo this zoom meeting.

- 1 guy from USA - 1 guy from Europe - 1 guy from East Africa - 1 guy from Singapore

It was extremely difficult to schedule a meeting off head.

When the it was okay for the guy from East Africa to meet it was like 5 am in Singapore.

Even if you successfully organise the meeting.

The organizer having to translate all times to the respective time zones of the audience is just so painful.

Okay now we have Calendly a tool we have to sign up for. But conversations of scheduling would be much easier if everyone in the world used UTC

1. People who never collaborate globally/cross TZ vastly outnumber those who do. And the people who do can already use UTC right now.

2. Making everyone in the world to use UTC is going to make everything less convenient for everyone who lives a few hours away from the UTC TZ (i.e., most of the world). Because in UTC many of their daily events are going to span 2 dates/weekdays.

3. For the handful of people who need to schedule cross TZ meetings there's https://www.timeanddate.com/worldclock/meeting.html, no sign up required.

That was some of the arguments behind the interesting but failed experiment that was Swatch Internet Time: https://en.wikipedia.org/wiki/Swatch_Internet_Time

I always use TIA internally nowadays. Continuous, no leap seconds, no problems.

How do you synchronise to TAI? Does NTP support it?

NTP provides UTC. UTC deviates from TAI by a number of whole seconds, the original offset plus all the leap seconds added since.

So you subtract the number of leap seconds from the UTC clock to get TAI? How do you update the number of leap seconds to subtract?

You never do this sort of conversion manually.

On linux you can use clock_gettime() with CLOCK_TAI:

       CLOCK_TAI (since Linux 3.10; Linux-specific)
              A nonsettable system-wide clock derived from wall-clock time
              but ignoring leap seconds.  This clock does not experience
              discontinuities and backwards jumps caused by NTP inserting
              leap seconds as CLOCK_REALTIME does.

              The acronym TAI refers to International Atomic Time.

Now, leap seconds are advertised in advance and your system know when they occur through a leap seconds file (leap-seconds.list) that needs to be kept up to date.

Ah, thanks. I didn't think it sounded like something to do in application code.

Honestly, the moment your time handling library tries to address all the things a calendaring application could quite reasonably want to do, you're going to end up with an unsatisfying solution.

After all, if I schedule a weekly recurring video call at 9:00 in City A, aka 14:00 in City B, then daylight savings starts in City A but not City B then either the meeting shifts to 13:00 in City B, or to 10:00 in City A.

No amount of clever date representation can sidestep the fact that the meeting has moved by an hour for some attendees but not others.

If you want to represent a point in time, use UTC. If you want to represent a point in a calendar, use local timezone.

But there is no point in time without a location and I think this is the point of the article. The quirk is that the offset to UTC might not be static because of timezone changes, and saving UTC doesn't capture that.

You're confusing time with calendar.

If I want to meet an alien species in Jupiter we're not going to coordinate using Earth's calendar. Doesn't mean it's impossible to meet just because they don't know Earth's calendar, we just have to specify a point in TIME without resorting to Earth's calendars. And points in TIME don't depend on timezone changes. Points in the calendar do.

A "timezone" is a map from point in time to point in the calendar. And a "timezone change" is an operation that takes a point in time from one point in the calendar to a different point in the calendar.

> The quirk is that the offset to UTC might not be static because of timezone changes,

An offset to UTC by definition does not depend on timezone, the same way that integers don't depend on timezones. 5 is 5 regardless of the timezone. If I say offset=3711, that has nothing to do with timezones.

You're confusing time with calendar.

I don't think aliens would care about UTC. And it seems a bit pedantic to separate it, humans tend to store times related to events on Earth.

> If you want to represent a point in a calendar, use local timezone.

And actually, going back to the original comment, this part was covered by the article. Storing timezone doesn't cover the case of a location switching between timezones.

So it seems the article can't be summed up in a simple rule like the original comment contends.

I didn't say aliens would care about UTC, did I? I said that we would have to specify a point in time without resorting to calendars. I didn't say we would use offset to UTC, because aliens don't know UTC either. But presumably they can measure time, so we could e.g. measure a point in time as an offset to "now". We would also have to use units that they understand. Not seconds, but perhaps Cesium revolutions.

> And it seems a bit pedantic to separate it

I would say that the fact that you can't see a distinction between time and a calendar is precisely the reason you're confused. So I wouldn't consider it pedantic.

No, you are absolutely right about the distinction, I didn't see it as useful since the article is dealing with calendars here, eg. time of event at some location. I don't think I've used time without a calendar so it seems a bit abstract to do so.

Every time you say "wait a second" you're using time without a calendar or a timezone. You're referring to a point in time as an offset (1 second) to "now" which is the point in time when you say it, and its the same point regardless of which timezone you're in, or whether there's a timezone change in between.

Timekeeping is hard enough already. Let's leave the aliens aspect out of it.

You might be right. I was trying to disentangle "time" from "calendar" by showing that just because someone doesn't know anything about calendars doesn't mean you can't coordinate with them to meet at a future point in time, and THEREFORE you should indeed use an offset to represent points in time and local timezone to represent points in a calendar as I originally claimed.

In retrospective I wish I'd chosen a less fantastical example, because of course people are gonna get distracted. "Aliens" was not the point. But now I can't be bothered anymore, losing interest fast.

You wanted to make a point that time does not depend on physical location, while you chose an example that proves otherwise.

There is no absolute time in spacetime, so your calendar invite from an alien friend would include not only the coordinates on Jupiter but also a time value relative to something. Possibly Earth. Maybe even UTC, as observed on Earth.

Please don't bring relativity into this except as a fun side note. It doesn't make a meaningful difference for actual objects in space, because they all share basically the same reference frame.

Exactly.

What I've found is that if you save the UTC timestamp + an IANA time zone (e.g. Europe/Paris) you are good. How you convert time for a given IANA time zone at a given UTC time should be consistent. For example in 2024-07-23T10:00:00Z, Europe/Paris should use the offset for the CEST timezone (UTC +02:00 hours).

If at a later point EU gets rid of daylight savings time and France decides to stay on CET year around, date libraries can account for this when you parse the UTC date string and convert it to the provided IANA time zone.

So UTC + IANA time zone gives you an accurate point in local time as long as your date libraries handles IANA time zones correctly.

That's all fine, but the problem case as described in TFA is that if the meeting was scheduled for 9AM (perhaps say 06:00 UTC) before the change, but 9AM now means 07:00 UTC, then you can convert that previously saved timestamp back to local time, but it will still be 8AM, which is not when the meeting was scheduled. The humans all arrive at 9AM (an hour later than originally scheduled with respect to UTC, but correct by the clocks on the wall), the software thinks it started an hour ago.

There are lot of points in time that don't have a location, such as a historical event. Then it's most accurate to store in UTC, then convert to the timezone you want only at display time.

The author actually made the point in the beginning of the article.

> When I read Stack Overflow questions involving time zones, there’s almost always someone giving the advice to only ever store UTC. Convert to UTC as soon as you can, and convert back to a target time zone as late as you can, for display purposes, and you’ll never have a time zone issue again, they say.

But what if you're not actually storing "a point in time" but actually storing a "time at a location"? The trick is to follow the Stack Overflow advice, but just in the opposite direction. You store the time as a "wall time", and don't store a timezone or an offset, and convert it to UTC (an actual point in time) at the last minute possible.

For a given location, what timezone or daylight savings rules it's going to follow, are always up for change, so the only thing you know for sure is the "wall time" and the "location". Waiting till the last minute maximizes the chance that you've got the latest time library with the latest rules, so it follows the same principle as the Stack Overflow advice.

Now, my opinion might also be colored by experience working in the rental car industry, because the primary goal there is what the customer experiences. Because in their mind, they set up their rental in Phoenix, AZ to start at 9am, so when they're at the counter, and it says 9am on the "wall", the car better be there. They don't want to hear that time zone rules for Phoenix changed in the 6 months since they placed the rental, so "technically" their rental doesn't start until 10am. So in our database, we actually just store "07/23/2024;0900;Phoenix". It's actually incorrect to even store a timezone or an offset, because there's no guarantee those won't change, only the location won't change, so you have to do the lookup for the timezone and the rules for the given location at the very last minute, maximizing your chances of having all the latest time library updates.

Ultimately, the issue is separating and capturing user intent between two major categories:

1. Events that are socially bound to a context, and capturing that contract or triggering rules, e.g. "Whenever the nation of Ruritania declares March 3rd end of workday".

2. Events that occur a fixed and knowable internal from "now."

It is complicated further by the fact that many users won't really be aware of their own varying different intentions.

Is anyone else sick of the "European Parliament" deciding the rules for the rest of the Internet? I know I sound like a dumbass American here, but I feel like there's gotta be a way for the EU to get what they want without bothering everyone else in the world constantly...

The issue is that they keep enacting laws that have good intentions, but are written poorly and without any sort of experience in the technical realm. This results in the laws' implementation being filled with weird loopholes that you can use to get around the extra work while still maintaining the legality, but completely missing the intention of the law in the first place. It just seems like a massive waste of time to me.

How do European timezone rules bother you?

That has nothing to do with the European Parliament. Or not in the general case. If you're interested, dig yourself into the changelog of tzdata. Timezones are inherently political.

Is anyone else sick of the US deciding the rules for the rest of the Internet? I'm fed up of American advertiser-friendly doctrine deciding what is acceptable content anywhere in the world.

(joking, but this is kind of an unsolveable problem of having a global network without a global governance system, you're going to run into mismatches)

EU pondering daylight saving in its territory bothers you? Well, too bad.

I'm not really sure what you mean, changing the timezone of a country is certainly their prerogative and "technical experience" doesn't really have anything to do with it.

The EU is one example here but many other countries have timezone/DST changes, the EU is one of the places that don't change timezones often. The US on the other hand has such variety in timezones and DST observance, not even following state borders, that it's a real pain to support.

Basically, you are ok with only the US influencing the rest of the world?

Because the US has control of the DNS root zones, is powerful enough to create a chip ban on China and has back doors in half the planet's machines through programs like PRISM or systems like IME. I think from an influence point of view, they are quite powerful enough.

Given the US also went to war with Irak despite the entire UN voting against it and while lying about WMD, have a mass spy program on their population, and will soon reelect a convicted felon, it's a good balance to have other parts of the world trying to have a say as well.

The human world is full of imperfections, but we usually average it at scale.

Do you know who last arbitrarily changed how summer time works, and dragged other countr(ies) along with it?

It was the US, in 2005.

Reply on news.ycombinator.com