Some Notes About Time
unix4lyfe.org
unix4lyfe.org
50 year old numerical / geophysical / real time data acquisition/processing/interpretation programmer here.
Unix Time isn't much chop for "real time" continuous data from the "real world" - it's those pesky leap seconds. If you bother to read the first paragraph of the wikipedia article on Unix Time you'll see :
> Unix time, or POSIX time, is a system for describing instances in time, defined as the number of seconds that have elapsed since midnight Coordinated Universal Time (UTC), 1 January 1970,[note 1] not counting leap seconds.[note 2] It is used widely in Unix-like and many other operating systems and file formats. It is neither a linear representation of time nor a true representation of UTC.
It follows on with a definition of Unix Time and points out various examples when it is ambiguous. These are real issues and can occur when missiles fly, when planes navigate, and when stocks are traded.
Time is tricky.
And all it takes to store a time like that is a 64-bit integer and it is very convenient. And a lot of software do precisely that. Most timestamps are just that: they don't care about "real world" details like leap-seconds, {23,24,25}hours per day, etc.
Because, in many case, you really don't give a flying shit about the "real-world" time.
What is your point? That a server that needs to open trade precisely at 9am has to take leap seconds into account? Sure. The article ain't disputing that.
But a lot of application basically only need timestamps and everything becomes much simpler when one starts to reason in milliseconds since the epoch instead of "real world" time.
Btw... Many of your "real issues" are due to programmers not realizing that they could have realized the exact same system but way more resilient had they knew to use timestamps instead of "real world" time.
There was an amazing paper by Google on the subject by the way, where they explained how they were reconciliating their hundreds of thousands of servers' time by allowing it to be "not correct" over x minutes / hours to dodge many issues.
And they clearly did emphasize that: a) these problems shouldn't have occured in the first place and were due to poorly designed programs (poorly designed in that they relied on real-world time instead of simply internally using milliseconds since the epoch) and b) most programs on earth do not give a flying ^^^^ if a time displayed to the user is one third of a second off...
Yes and no: it might just exist theoretically, but we have no way to get at it. The closest we have is TAI, which is only an approximation of the time elapsed at mean sea level on the earthly geoid, because clocks fall victim to gravitational time dilatation and compression.
To accurately measure time you'd need a clock sitting perfectly still in space, and all other clocks in the universe would slowly drift behind it.
After the humans will wander thru space ( at some time - very probable), there will no time reference, but only time intervals ( like, day on a spaceship has 24h , etc ). So in this case, you would measure 86400 seconds and call it a new day. No more leap seconds, etc.
Now the UNIX makes sense: count seconds since a certain event in time and meajure from there on, internally. Want to display it? Then use special computation to render it in the format ( read timezone, add relativistic skew, etc ).
You only encounter issues with TAI once you get down to femtosecond or smaller levels.
I may have worded it badly, let's try again: TAI is "completely perfect" for the approximation it is: time elapsed at the geoid, which is a theoretical construct. That's an approximation for both "experienced time" and for anything which could be called "absolute time"
Alas, I'm afraid you've no other option but to admit a minor error, as traumatic as that may be.
Yeah. Hmm. Clearly an opinion expressed without either reading, comprehension, or experience.
Why would I say that? Well, consider your statement about "actual seconds elapsed since epoch". Leaving aside the fact that Unix Time enumerates no such thing (and it was Unix Time I was discussing), there are such things as time frames and relativistic effects which come into play when doing fine scientific measurement, astronomical work, and ground satellite communications.
There are levels to dealing with time and the linked article makes a decent first order stab at climbing the ladder, even ground based programmers need to be wary of the leap seconds issue and the plethora of time standards with minute but significant (in some context or another) issues.
When you've cooled down a little consider the title "What every programmer should know ...". Had it been "What most programmers can skate by on" I probably wouldn't have bothered to comment.
I agree there are many good papers on the subject, going back decades, I'm betting the google paper you speak of references a classic one on time stamps and relativity as some times local event order trumps universal order (and in fact the paper pivots on the observation that universal order doesn't really exist, just effective order).
I don't doubt that you have valuable experience and information to lend to the discussion, but your frame seems aggressively negative towards the OP with no discernible reason.
No, he's declaring that not every programmer can afford to ignore these.
The OP article suggests that UnixTime is the answer for every programmer. My take home message to every programmer that uses UnixTime is to be aware that it's non linear and has hiccups that will bite them every few years if not taken into account. This is not so much aggressively negative as it is a simple statement of fact garnered from years of experience.
My message to every programmer that works in a distributed system is that they should read "Time, Clocks, and the Ordering of Events in a Distributed System" (Lamport 1978) which uses observations and arguments from relativity to comment on the manner in which events propagate outwards from sources.
As for limiting awareness to "controlled scientific experiments", no, I'm not advocating that at all as time slip (something that has many causes outside of dilation) pops up all over the place these days, for example many things rely on GPS time which is something else that is non linear and periodically updated. I'd suggest that anyone writing software that relies on second / sub second granularity should be aware of where their fiducial time marks come from and what hiccups there are in that system.
I find your opinion and experience on this matter very interesting. Could you possibly describe your experience or projects you've worked on that could shed some insight on the complicated issues time causes?
Obviously you don't need to go trough all that trouble if you simply want to display the date you published a post on your blog.
Your claim that using Unix Time saves you from worrying about leap seconds is incorrect. Unix Time goes backwards when a leap second occurs, which can screw up a lot of software. Check out Google's solution to the problem, which is to "smear" the leap second over a period of time before it actually occurs: http://googleblog.blogspot.in/2011/09/time-technology-and-le...
Practically no software uses true seconds since the epoch; if it did then simple operations like turning an epoch time into a calendar date would require consulting a table of leap seconds, and would give up the invariant that every day is exactly 86,400 seconds. Whether this was the right decision or not is debatable, but it is a mistake to think that using Unix Time saves you from all weirdness surrounding civil time.
But there are loads of cases where your computer/programme needs to talk in "real-world" time. So you can't avoid the problem.
Thanks for clarifying!
Humans represent time vaguely rather than precisely. Computers tend not to.
A while back, the British Library catalogue was put online as linked data. The library cataloguers had added details of author birth and death years but not always birth dates. But the data model normalised "1941" to "1941-01-01". (They fixed it after I moaned on Twitter.)
You should be able to represent in your data model, date-time library etc. vague dates and relative dates. When I say someone was born in 1941, I don't mean they are born at 1941-01-01T00:00:00Z. When I say that something happened on Thursday, I don't mean it happened at 2013-01-17T17:04:00Z. It may have happened at any time on Thursday.
If you store everything as Unix epoch seconds, good luck representing vagueness.
ISO 8601 gets around the problem rather neatly by being big-endian and allowing omission at any point. Wanna say 1941? "1941". Want to say January 1941? "1941-01". Want to say January 14, 1941? "1941-01-14". Want to get very detailed? You can add points of a second: "1941-01-14T12:14:03.0482Z". You can do timezones.
Eventually, we'll have an even better way of representing date-times that are more like bitmasks. We can then represent in a sane way the concept of opening hours. If you say the shop is open from 9am to 4:30pm Monday to Saturday, how do you do that?
If I am digitising an old text of unknown origin, and on a page it says "January 12", and on the page after that it says "January 15", can I represent those with a reference to an implicit but unknown year? I don't know the year, but I do know that the January 12 instance comes before the January 15 instance on the next page. And at some future point, we might deduce from circumstantial evidence what year it was and the sequential ordering will still make sense. Relative dates would be useful here.
ISO 8601 allows you to represent ordinal dates (the number of days since the beginning of the calendar year).
Can your date-time representation deal with moveable feasts like Easter?
Have a look at the WHAT WG Wiki page on Time to see more obscure use cases: http://wiki.whatwg.org/wiki/Time
Your date-time representation, storage format and calculation libraries are probably inadequate.
Of course, if the year is unknown, and could be before ~1918, and your dates are, say, the dates of two letters, you can't even be certain about the ordering of "January 12" and "January 15".
For the record, the Open Street Map project has come up with a human readable way to encode opening hours. http://wiki.openstreetmap.org/wiki/Key:opening_hours
Computers can be made to represent vagueness rather precisely.
> You should be able to represent in your data model, date-time library etc. vague dates and relative dates.
If it is needed. Often it is not needed. Or rather for most applications there is an implicit hard coded resolution, or error interval.
The library case is special and then why not just have an additional byte specifying the confidence interval. You only need a small enumerated type of 20 distinct values to encode confidence intervals from a nanosecond up to a millennium. Heck you can do it with bits if you want to.
Hi. A colleague of mine is working on a large medieval dataset than contains many dates and recently explained this concept of vagueness to me. What's the standard way for dealing with this in MySQL for instance, how do comparisons and orderings work for a collection of vague dates?
You're saying ISO 8601 can do this, and MySQL can talk ISO 8601?
True story: when I was at ITA Software, Orbitz ran hundreds of instances of QPX, our low fare search software, on their own servers. We had an ops fire drill one weekend because customers were complaining that the site was showing incorrect prices. The root cause? A single machine in their server farm had the time wrong, so advance purchase was computed incorrectly for any query sent to that particular machine. That was fun to debug.
Even the meaning of, say, a minium stay requirement is hard to precisely define. If you fly across the international date line and back on a Friday night, did you have a Saturday stay or not? There are so many flights going so many places, questions like this actually come up in practice.
Coordinated Universal Time (UTC) is based on International Atomic Time (TAI), but offset by an integer number of seconds so that it remains in approximate agreement with more traditional time scales based on the Earth’s rotation (i.e. Universal Time or, equivalently, Greenwich Mean Time). There is no natural gearing between TAI and GMT and so additional ‘leap seconds’ are applied when required to keep UTC in agreement with GMT to the nearest 0.9 seconds (see notes on time scales in Annex A for background information). UTC is the global standard for civil-time keeping today. It provides the most stable time base available because it is based on TAI, but also acts as a good approximation to its antecedent, GMT, for everyday purposes.
Don't do this in your database. Use your datetime types. Please. You might save some work with your timezone but you're not going to be able to use intervals etc in a smart way.
This would be even dumber with PostgreSQL, which has much more robust date/time functionality (being able to use the - operator with datetimes and intervals, for example) and completely sane timezone support.
The only true problem with storing unix time is leap seconds and relativistic effects, which can both be safely ignored for most time keeping usages.
Even just saying "24 hours from now" -- as your user would see it -- you'll sometimes get the wrong answer if you just add on 24 hours worth of seconds... suppose they have a DST shift tonight?
What if you need to know how many days remain from today to March 1st of this year? ...well, are we in a leap year?
Postgres datetime doesn't solve all of the problems that crop up, but it's certainly better than using unix time.
Same for working out how many days remain - you convert March 1st to a unix date, get an interval in seconds, and then convert that into a human readable format on display.
This isn't some revelation - unix time has been used successfully for decades now.
EDIT: Also, Postgresql uses a number very similar to unix time in the actual storage of the date - it just handles the pretty display for you. So you're arguing for using the same thing whether you say use a postgre datetime or a unix date number.
It's rather more than that, though I don't have the time to kill digging into it now.
And personally I mostly seem to end up doing more complicated date processing in code, not queries; but there everything needs to be a date immediately (not unix time or similar) for most purposes -- then I can use complicated libraries written by others to let me do "simple" things with dates, like rolling months.
Ah, also: > you convert March 1st to a unix date Then that's where the complicated logic goes. Basically, you need that somewhere, and it's non-trivial (leap year calc is the least of it).
My point isn't that unix dates aren't useful for storage, but that they aren't useful by themselves for calculation.
Postgresql and every other database that I'm aware of stores dates internally as a 4 or 8 bit number anyway, so you aren't saving anything by using an (hopefully) bigint instead of a datetime column.
Unsurprising, given how cavalier MySQL is with all the other data it "stores".
Also, does the problem that the leap second solves actually warrant all of the problems that it causes?
Sun rising 30 seconds later every century is not really a problem in my opinion.
Definitely not. The only benefit of leap seconds is keeping UTC in sync with solar time: the sun is highest in the sky at 12:00 noon at Greenwich.
But in the modern world, synchronizing solar time and clocks to <1 minute is of no value to anyone but nostalgic astronomers.
It will take 3000 years before leap seconds add up to an hour, but most countries adjust the clocks that much every year for DST. It will take 40000 years before the usual daytime hours turn into night, and by then I hope a single planet's rotation is a historical oddity.
For example: I had a game that physics behaved "woobly" in multi core systems. The issue is that as the OS juggle your process around, you get the delta time since last physics step inaccurate, sometimes even with backwards jump in time ( and according backward movement in the game ).
Later I saw someone showing the effects of this on the file system, with a multithreaded file copy resulting in very strange timestamps.
For my game, the solution was force time-dependant threads to request affinity with one core.
If your software has business logic that cares about, say, what day a specific person perceived an event as happening, it needs to think about the timezone they have set.
So, in such cases, you'd better keep a history of all the timezones they've ever chosen, and the time (UTC) they changed them.
(Hopefully this lets someone else avoid my past mistakes.)
This post advises to use Unix timestamps. That's already what I'm doing mostly everywhere. And it seems to me like most people do and like this is somewhat accepted as the best available option - but still not perfect.
Then I am wondering again: Isn't there really a perfect solution? Esp., the problems with Unix timestamps are the non-monotony and that e.g. two equal timestamps can actually represent two different times (seconds). This could be bad in cases where seconds matter (e.g. some log-files which need to be very precise about time).
Maybe TAI is and we all should store TAI timestamps instead of Unix timestamps? But is there an easy way to get TAI timestamps? Also, I haven't really seen other projects doing this - why? Maybe it is just too little gain over Unix timestamps and too less tools available to work with them... Btw., I just checked, there is http://cr.yp.to/libtai.html and http://pypi.python.org/pypi/tai64n, maybe I should just start using that.
This of course still doesn't solve anything about synchronization or inaccurate system clocks but it would be better than Unix times.
Edit: Getting the TAI timestamp is probably not easy or maybe even not possible... I just saw this: https://github.com/stoni/libtai/blob/master/tai_now.c ...
The Unix timestamp 915148800 (from the table) represents two times (separated by one second from each other).
One of the solutions used in various systems from cash machines to databases is to use GPS time. All GPS satellites carry atomic clocks, and when positioning the receiver also determines local time at accuracy up to 10 nanoseconds.
GPS time does not have leap seconds, and is always at a 19 second offset from TAI. It's available without a network, and without synchronization between computers.
The link in the paper's headers is still live: http://naggum.no/lugm-time.html
The home page refers to his ill health: http://naggum.no/
And he in fact died in 2009 at 44: https://en.wikipedia.org/wiki/Erik_Naggum
Interesting guy, involved in specifying the Internet from link level to mid-email level, worked on Emacs, and among the first Usenet users to be known for flaming. I vaguely remember the name in that context.
On balance, a contributor.
Advice to display an offset when displaying a time is wrong or at least incomplete; you should display a symbolic timezone (i.e. "London" or "Eastern US" etc.) as that's what's going to be meaningful to someone reading it.
This won't work for any system that allows users to schedule events in the future. Let's say that you agree for a meeting in Moscow 1 dec 2013 3pm. The actual (astronomical) time of meeting might change significantly if they change daylight saving rules. So you need to store something like "2013-12-01 15:00 Moscow", there's no way around it.
When information is to be transmitted between timezones, you want to be able to determine the locale time of the sender or the information relays. This is why you'll find a time zone information in dates and time stamps in mail headers.
This is also why ISO 8601 bothered to standardize specification of the time zone.
At the International Meridian Conference in 1884 41 delegates from 25 nations met in Washington, D.C., USA and selected the meridian passing through Greenwich as the official prime meridian due to its popularity - not an arbitrary decision ;)
However, the French abstained from the vote and French maps continued to use the Paris meridian for several decades.
UTC: The time at zero degrees longitude (the Prime Meridian) is called Universal Coordinated Time (UTC).
If you stand at the Prime Meridian in London during the summer and ask someone what time it is, the correct answer will be one hour away from UTC.
Among many other problems, converting between calendar systems requires that you define when a day starts; there are many ways to define that among the systems.
I think I have the latest, the third edition, https://en.wikipedia.org/wiki/Special:BookSources/9780521885...
Bonus points for anyone who knows what I mean by TVM and can describe why every programmer should know about it..
Love to know what this TVM is.
Not necessarily. The time might be "floating time", i.e., depends on some local context.
For example, the show starts at 10 pm 19 Jan 2012. You don't need the time zone because people going to the show know what 10 pm means locally.
1. Google Plus conversation remains https://plus.google.com/106413090159067280619/posts/Wtkhk9jU... Good luck finding it on Twitter
mmahemoff, are you using actual milliseconds, which means you have to keep track of leap seconds?
Doesn't the time value tell you? Or have there been temporally overlapping definitions of UTC that disagreed?