Hacker News Reader

Comment by alphazard | Hacker News Reader

Most developers are basically powerless here, they use the standard libraries for time provided by their language, and those have to deal with what they get from the kernel. There's nothing they can do about timestamps in the filesystem.

For anyone designing new systems that store or transmit time, you should be aware of TAI64. https://cr.yp.to/libtai/tai64.html

Unless you are dealing with scheduling, or generating events with a particular cadence (e.g. daily, weekly, monthly) there is no reason to include UTC or time zones in your data model. They are view only concepts, with conversion to the local display format happening from TAI64 on the way out.

View on HN

> There is no reason to include UTC or time zones in your data model.

That is a bold claim. Not everything is about the entity viewing the data, sometimes it is about the entity producing the data. Not storing the timezone is losing some information.

A few examples:

- The "Date" header in emails include the timezone (or UTC offset) of the sender. It allows the recipient to know whether the email was written in the morning or the evening.

- A SQL query that builds a report of restaurant orders per hour needs to normalize for time of the day in local time otherwise it is not possible to know whether people order lunch at 11 am, midday or 1 pm.

This is not only a bold claim, but utterly wrong. Every single datetime you’re storing should absolutely contain a timezone, preferably UTC.

I‘ve been bitten by dates lacking timezones in various situations during my career and it has always been a pain in the ass to debug and resolve all those random problems they‘re causing.

Do. Save. Dates. Containing. Timezones.

My understanding is that you don't even want to store them as UTC and always store them in the timezone of the user/system that generated the data since you lose some information in the translation from local timezone to UTC - given that the timezone database changes, you may actually want to know that the record created in Fiji before 2022-10-28[1] was stored in DST so that it can be properly converted to the new Fiji time.

[1]: https://data.iana.org/time-zones/tzdb/NEWS

This! People mistake the offset for the timezone, which is incorrect. The offset isn't very useful other than a guestimate of the originating timezone.

Ideally store both, UTC and original local time, or at least the original TZ information. Time and timezone are a huge mess and one of the worst things to try deal with in programming.

In the airline industry they sling huge numbers of dates and times around, and they never include the timezone. What do they include? The airport code. Boarding, departure, arrival, and all other times are always local to the airport. In practice, that means that every piece of code that wants to handle datetimes has to also have access to a way to lookup the timezone for the provided airport code. Have a time at LAX? look up LAX, find the timezone, and compute the offset.

In practice, almost nothing does TZ conversions or datetime math. In effect, airline timestamps are a tuple <datetime, airport_code>, and nothing much messes with that.

Doesn’t sound great. It‘s also an industry plagued by legacy systems.

Oh, it's not great at all. The only upside to it is that passengers looking at schedules and boarding passes don't have to do any timezone conversions for long flights. The departure time is always the time at the origin, and the arrival time is always the time at the destination.

Until you add a flight to your calendar and have to look up the timezone.

I am that donkey banging it's head on the same post; when I see dates stored without TZ, I assume it's UTC and have done so for decades. It's turned out to almost never be true.

> preferably UTC

Yes! I have spent so much time detangling timestamps in data which used non-UTC times and non-standard formats. Like 06-10-01 WTF is that? Use UTC and then let the reader translate to local time on display.

> [...] sometimes it is about the entity producing the data

so should we not store that data, then?

> The "Date" header in emails include the timezone (or UTC offset) of the sender [...]

i don't think that's strictly true - at best it may contain the timezone configured on the local machine when the e-mail was sent (what happens when you fly?), at worst it always contains UTC anyway for privacy concerns. in either case i don't think most people know this is a field that is transmitted, and it's presumably something the sending/recieving party likely already know each other (for personal communication)

> A SQL query that build a report of restaurant orders per hour needs to normalize for time of the day in local time [...]

i'd posit that that data should be stored principally then, surely?

fundamentally, why should the timestamp contain two data points? i wouldn't expect the database to have a single column for, say, my user id and registering user-agent string, so why would i expect it to combine a point in time with a vague location?

Ultimately my comment was about entropy: a date with a timezone contains more information than a date alone. The technicalities about database columns, inaccuracies or privacy implications do not change this fact.

I argued that keeping this information instead of discarding it is often useful and gave a few examples. I understand that sometimes systems can get by with only storing local dates or UTC dates but sometimes a system needs to be able to deal with both and keeping these extra bits of information is the only way.

I've had pushback on items like this sometimes with the standard "YAGNI" excuse. A few months ago I came across some blog post describing the reverse: "YAGRI" - You Ain't Gonna Regret It. Just... store the extra data.

of course, if you need it then you need it.

i think the original commenters point only talks about how bad TSTZ is as a storage format, and that TAI is not only better than timestamp with time zone, but also UTC in almost every measure - something i agree with - not that time zones are useless. if you need the time zone to be recorded, i think the most sensible thing would be to record it independently.

recently did a project where some items were "future dates" - scheduled appointments for days/weeks/months in future, and were for entities in different time zones, and people making the appointments were occasionally in different time zones. The latter part was rare, but needed to accomodate that.

Storing the appointment date/time as ... a date/time without timezone (timestamp without tz in postgres IIRC), then storing the timezone as a separate string - that gave enough flexibility to accomodate any situation that came up.

"Wall time" - the date/time and tz separated - for future dates worked. Someone else wanted to store the date/time as 'utc timestamp' after the date had passed. "Wall time for future, UTC for past" was the motto, but... there wasn't a clear way of looking at the data to know which one you should be using immediately. I argued for consistency in the original data - if you want another view of it... make a db view that has the 'utc timestamp' conversion in it.

This guy gets it. Thank you for explaining better than I did.

If someone wants to reveal their location, why wouldn't they include latitude as well? Location is a separate piece of information. If the idea is to give the reader an idea of where the sun was in the sky when the writer crafted the timestamp, isn't latitude also important?

> it may contain the timezone configured on the local machine when the e-mail was sent (what happens when you fly?), at

Most people have their system clock update automatically

indeed, but i don’t think they are able to retroactively update the time stamp of an already sent email :)

Why would it?

"It allows the recipient to know whether the email was written in the morning or the evening."

I guess sure, you could write it in the morning then send it in the evening. Most people write and send at the same time, and thus most headers have the stamp of the time it was written.

On your second example, you should be able to do that without storing the timezone of each event. In fact, odds are the timezone of each event will mislead you and break your analysis.

On the email example, the timezone is there for debugging purposes. It's really not for knowing if the email was sent in the morning, and if you use it for that, you are almost guaranteed to get broken data.

Of course, there are reasons to include timezones on your data. That's what you get right. But if you want the timezone of the data producer, you should store it for the producer, not for the data.

(And then there is the issue that the SQL standard dictates that dates without a timezone are broken. So you shouldn't ever store them on a SQL database. But that one timezone is useless, if you need that information, you should store it elsewhere.)

Your sql query example doesn’t seem to actually need timezones. If all orders are stored in the correct UTC regardless where they come from. An order is always made at a specific point in time that is the same globally.

> Under many cosmological theories, the integers under 2^63 are adequate to cover the entire expected lifetime of the universe; in this case no extensions will be necessary.

Pessimists!

Indeed. As they say, eternity is very long, especially towards the end. For proof, check out this video that explains how eventually even protons will give out and evaporate.

The thing that blew my mind: when things started getting very weird, and I was thinking "ok but we're roughly at the end now", I checked the indicator. It was only half-way. It is a 30 minute video. And the time speeds up logarithmically.

https://www.youtube.com/watch?v=uD4izuDMUQA

AFAIK, the most accepted theory is that the Universe will allow for useful computation forever. Always get slower, but adding up to an unbounded amount of it.

Only if you use seconds, unlike the JVM which uses milliseconds.

This bites you in the ass in embedded devices, which then have strange behaviours every 49 days.

Experienced this first hand on an embedded device with an RTOS with a 5ms tick. The thing was locking up every 124 days (2^31 ticks). That’s long enough where it had to happen 2-3 times before someone put 2 and 2 together.

In that case, use RFC2550. https://www.rfc-editor.org/rfc/rfc2550 .

> As discussed in 2.4.1, the end of the universe is predicted to occur well before the year 10 * 30. However, if there is one single lesson to be learned from the current Y2K problems, it is that specifications and conventions have a way of out living their expected environment. Therefore we feel it is imperative to completely solve the date representation problem once and for all.

The moment your data is produced in more than one time zone, you're lost without the time zone. This may happen if you have customers in more than one city, or your customers use a mobile device to access your services.

Maybe your CLI utility can be timezone-oblivious, but those who develop and operate systems mentioned above, or just happen to travel, would be more happy if you handled timezones.

Naive timestamps are fine as long as they do not leave a single machine, and are used to calculate durations, timeouts, etc. These do not depend on the absolute value a clock shows, but only on differences between such values.

You just have to store the user’s time zone as a preference and then convert to that time zone from UTC.

That is usually fine. It is generally good for system generated times representing the past. You can store in UTC and display in in any zone.

It is not good for user entered historical times that they expect will remain exactly as they entered. Even if the user specifies a specific timezone, for long enough ago timestamps the TZDB does get occasional updates to past timezone data to better reflect what actually happened.

If the user entered 10 AM America/New_York for a long ago date, you convert it to UTC, and then the TZDB updates historical timestamp info, based on newly uncovered evidence, now it might appear as 9 AM or 11 AM or whatever.

It is also insufficient for user entered future times. If the user enters a time for an event 5 years from now in their local timezone, and the local DST rules chance, they would typically still want the event to occur at the local time, which means converting to UTC and then back won't work.

Yes, people should use a TAI-correlated time for timestamps and events corresponding to the present or past (the future is a different story). However, they should not use TAI64 since, even though this is my first time seeing it, it is clearly a antiquated format. It is literally using a big-endian(!), 63-bit, unsigned (but biased) integer to encode time. Every single modern computer is little-endian by default and the usage of a 63-bit unsigned integer instead of a [64-bit unsigned integer] or [64-bit signed integer with reserved upper and lower values] is bizarre.

It's not helpful to criticize aspects of a system which have good reasoning behind them, without understanding and presenting that reasoning in good faith, and providing a better alternative.

Endianness of formats is not a concern on modern CPUs since we have efficient mechanisms for adjusting endianness on load and store. The choice of biased 63 bit integer ensures the the same arithmetic can be performed using both a signed and unsigned 64 bit integer. The JVM comes to mind as a system without support for unsigned 64 bit integers (maybe they've fixed that by now). As mentioned the upper values are reserved for extensions, meaning the most significant bit could be set to switch to another interpretation, potentially when migrating to something more modern. So far it's unused, just evidence of good forward thinking protocol design. Additionally, big endianess allows the format to be sorted using string comparison.

Even if I agree with the thrust of your point, I think it’s too much to claim GP is not criticizing “in good faith” just because their critique wasn’t as defended as strongly as you’d prefer.

Seems like a misuse of the term, unless you think GP had ulterior motives behind their critique of a datetime standard format.

Network formats (at least the popular ones) typically transmit in "network byte order", i.e most significant byte first. There are advantages in being able to read a hex dump without manually reversing the bytes in your head for every field that is two bytes or larger.

Far from antiquated I would consider the selection of big-endian formats as an indication they actually gave some thought about what they were doing, ranking debugging convenience higher than a marginal efficiency gain on some cpus. The same consideration generally applies to all binary formats intended to be portable across systems.

Ah hackernews -- where someone confidently decries a timestamp format designed by the world's most accomplished applied cryptographer as "bizarre" and "big-endian(!)".

Did sorting ever occur to you?

Do the CPUs you use typically not have instructions for comparing little endian 64-bit integers?

It can be convenient to be able to just sort as an opaque byte string.

On embedded systems? No.

Are they big endian or little endian? If they are little endian, is it cheaper to perform 8 1-byte compares, 4 2-byte compares, or 2 4-byte compares?

> Most developers are basically powerless here

In fact, there are so many corner cases in date/time, trying to calculate anything like that yourself is probably as ill-advised as rolling your own crypto.

Obligatory xkcd: https://xkcd.com/1883/

> For anyone designing new systems that store or transmit time, you should be aware of TAI64

I wouldn't hold your breath when it comes to TAI64 adoption.

We live in a world where people still debate whether leap smearing is a good thing or not (and then barely implement it properly). And we live in a world where IPv6 adoption is, well, "still happening" to put it politely.

Most people have not even heard of TAI64, I had heard of it, but I had forgotten about it until it was re-mentioned here about two decades since I first read up on it.

The truth is in your first paragraph. Developers will use what they are given, their role its to get the coding done in the shortest reasonable timeframe. Dictating the use of a barley used form such as TAI64 is "above the pay grade" of most developers.

Finally, in this increasingly cloud-first world in which we live, unless one of the big-three suddenly embrace TAI64, the whole idea of TAI64 is effectively dead in the water.

I like the idea of wider use of TAI but there seem to be plenty of obstacles. For example:

* Is it possible to set up a standards-compliant network time server that serves the time in TAI? (The older protocols, at least, seem to specify UTC.)

* Is it possible to put the time in TAI in e-mail headers? (It would be a nice visible way of showing one's support for TAI, but it doesn't seem to be possible according to RFC 2822 -- https://www.rfc-editor.org/rfc/rfc2822#section-3.3 -- though you could just put "TAI" as an "obs-zone": according to section 3.4 it would be "considered equivalent to "-0000" unless there is out-of-band information confirming their meaning")

It never came to my mind that TAI64 could be something useable in “every day” programming. Thanks !

> Unless you are dealing with scheduling

It felt like a niche use at first read (“I’m not building a calendar afterall”), but it’s more than half of what I’d use time manipulation methods for actually.

Anything with a validity start or an expiration date for instance would fall into “scheduling”. But then timeouts and end of process estimation also fall into “scheduling” but need to not be affected by timezones etc. and be TAI64 in this case. And there must be a ton of weird cases that don’t come to my mind at the moment but would screw me when time comes.

To expand on calendar scheduling, there is still a place for TAI64. The calendar application is typically asking for "all of the events between this timestamp and that timestamp". You will have to use the Gregorian calendar, and maybe the timezone of the meeting location to figure out what those events are, and when their start and stop timestamps are.

"every Monday", or "the first Thursday of the month" are all rules for generating events. It doesn't make sense to store them as UTC either. The application is going to take in some rules, and some timestamps (TAI64) and it's going to produce a list of events within those timestamps. UTC, leap seconds, the whole mess will come into play, in memory, but not in storage.

> Unless you are dealing with scheduling, or generating events with a particular cadence

Or are concerned with spacetime :-)

Aren't we all concerned with spacetime, technically?

Meh, I’m not concerned.

Next minute, FPGAhacker's GPS navigator directs him to drive into the local zoo's lion cage...

(I don't have a reference handy, but I recall reading that if you don't allow for time dilation effects of the GPS satellite's clocks moving faster in orbit than the ground below, you lose just over 10km of position accuracy per day.)

It carries a lot of weight.

First time I hear about this, will have to read more. Is this used widely?

I’ve always run my servers on UTC time and stored all dates the same.

https://dyscour.se/post/12679668746/using-tai64-for-logging

> This means that the date is stored as a 64-bit number, the bottom 63 bits being the number of seconds since 1970-01-01 in TAI, offset by 2^62. In essence, this makes the value quite similar to time_t (which uses UTC), except for two things:

> There are 63 bits to play with, not just 32 (no year-2038 problems!).

> The timestamp is always monotonic, counting all leap seconds.

TAI time seems to be just UTC unix timestamps but accounting for leap seconds. Since leap seconds are slated to be retired after 2035, I’m guessing TAI time and Unix time won’t be so different after that.

TAI is not a timezone. For actual clock, you still want to use UTC most likely. TAI is more of a format for describing that time. You can convert from UTC to TAI by removing the 37 leap seconds we got so far. Check out https://en.wikipedia.org/wiki/International_Atomic_Time

The pedant in me is forcing me to say: The general conversion is a little more subtle than just removing 37 leap seconds. This is especially if you want to convert a date in the past since we haven't always had 37 leap seconds to account for. Also, things can get tricky right around those leap seconds when it matters.

If you're just talking about running a typical Linux server with your server applications on top, then it's 99% of the time all 64-bit and nothing to worry about. 64-bit time is going to be used on 32-bit systems as well, so if you have those systems make sure to upgrade them in a decade.

Though saying this, I think you still have to manually remember to upgrade XFS filesystems so maybe there will be some things to worry about next decade.

It may sounds convenient to just use TAI, but that does open a whole new category of issues - you get UTC and TAI time mixed, we humans are good at making mistakes, god knows how many horrible bugs, damages or even deaths could be caused by mixing those two in systems.

You can just change the system time :) Would be better to mock the time provider, but it's not the end of the world.

Reply on news.ycombinator.com