Hacker News Reader

Comment by nly | Hacker News Reader

Am I the only one who would expect the behavior in the top example?

I don't want my equality operator doing timezone conversions. If one type has no timezone attached and one does, then you probably shouldn't be able to compare them at all. Likewise, for a type that has a timezone attached, if the timezone differs between a and b, then I want a == b to return false, even if it represents the same instant in time.

Equality can be a tricky thing. Named functions are the way to go when two different developers might intuit different behaviors.

View on HN

The problem is not that the equality behavior is wrong, it’s that I would expect “utcnow()” to return a timestamp with a UTC timezone. The problem is that it does not, it has no timezone at all, and therefor is generally less compatible than creating a timestamp with timezone from a utc timezone object.

I work on the Python API of a timeseries database and it’s very frustrating to have to accept timestamp objects without time zone because you don’t know what to do with them. People aren’t aware of this constraint, which leads to unexpected behavior.

Django solves this by having the default timezone in the framework settings. Any datetime without a timezone is assumed to be using the default one. If you don't set a default timezone, the server timezone is used, and you are responsible for making sure your data is congruent.

Yes I know, but that doesn’t help pretty much any of our Python users that are all heavily reliant on Pandas and/or definitely not writing web services.

As of now, we just reject any timestamp without an explicit timezone and try to explain to our users why utcnow() is bad. People are generally very surprised that this issue exists at all.

Yeah naive datetime should not even exist in high level API IMO. They're like bytes string with no encoding or geospatial data without a projection name field.

It's going to bite you at some point.

Timestamps do not have timezones. That's the whole point of a timestamp.

Timestamps must have an implicit timezone, otherwise they are meaningless, because they cannot identify a certain moment in time.

Only time differences do not have a timezone.

When Python continues to use the word "timestamp" for a time value with an unknown time zone, then that is the fault of Python, because the name is completely inappropriate.

I assume that whoever enabled a value "None" to be possible for the timezone did not really expect that anyone will ever use a timestamp with unknown timezone, but that in such cases there is a timezone known to the programmer, which is not stored with the timestamps, and that it is the responsibility of the programmer to restore the timezone whenever necessary.

In this case we're talking about Unix timestamps though, which count the number of seconds since 1970 UTC. Adding a timezone to it is nonsensical. The number is always the same, no matter which timezone you're in. It's in the conversion to year-month-day, etc. that timezone becomes relevant.

GP’s argument is that that means it has an implicit timezone of UTC.

I would argue that "seconds since epoch in UTC" has an explicit timezone of UTC.

A point on the globe has a timezone. "Distance from London in feet" does not.

That is true, but a timestamp is intended to specify a particular time, and a simple temporal interval fails to do that just as a simple spatial interval fails to specify a location; both are incomplete (the former on account of a missing starting point, and your counter-example on account of a missing azimuth.)

On the other hand Unix (POSIX) time does not have leap-seconds, while UTC does not, so it is no longer "in" UTC.

POSIX time isn’t “in UTC” regardless of that. UTC is just one of many timezones you can use when specifying which point in time timestamp 0 refers to. You could just as well use EST or anything else.

You are right - through this discussion, I have come to see that unless a value expresses a specific time (as opposed to interval) in a form that includes hours and minutes, then timezone is not an applicable concept. POSIX time does not do this.

This should be files under "falsehoods programmers believe about timezones".

So the analogy is "Distance from London in feet" to "Time since epoch in seconds"? Ok, but "epoch" has a timezone, which is UTC, so measurements relative to that have the same timezone.

I agree with you and I think the above poster needs to adjust their thinking.

What they're saying is correct in the most technical sense, but our industry has landed on timestamps not needing timezones. If they want to work in this industry, they need to accept that.

It's ok to be surprised by something like this, it's less ok to argue about it.

That's not exactly right. I don't live in UTC, when I use timestamps they don't automatically (whether implicitly or explicitly) refer to UTC.

A Unix timestamp is a difference in time, not a time point. It's a number of seconds. It's a duration, not an instant, which is why it doesn't in and of itself need a time zone attached.

That being said, people do use timestamps to represent time points sometimes, and that does requires picking a particular timezone. That's fine too, but it's not the only use or meaning of a timestamp. It doesn't make timestamps without time zones meaningless, and doesn't mean they implicitly refer to the UTC epoch.

Durations and time differences are timezoneless numbers. When you add a duration to a time point (like the Unix epoch in your preferred timezone), you get another time point, in the same timezone.

> A Unix timestamp is a difference in time, not a time point. It's a number of seconds. It's a duration, not an instant, which is why it doesn't in and of itself need a time zone attached.

The number of seconds since what, exactly?

It appears to be the number of seconds since Jan 1, 1970, at midnight... in some timezone.

If you refer to that date, then UTC. But you may just as well say that it's the number of seconds since 1969-12-31 7:00 pm EST. There is nothing special about UTC wrt Unix timestamps.

The point is that the question 'since what' is a misleading intuition, it's not at all essential or implied.

You want to know how long something takes. You take a timestamp before and one after.

A timestamp since what? Well, by definition, the number of seconds since the unix epoch.

Which unix epoch, exactly? In what timezone?

Unspecified. You could have picked any timezone, and the result would not change, as it is not essential. Even if there is no timezone, not even one that you picked in secret, it still works.

Asking the question for durations is a subtle error. You only need a timezone for a time point. Timestamps are often immediately converted to time points, but that is neither necessary nor inherent.

This seems just like kicking the can down the road, pretending that time zones don't matter because you don't want them to.

If I'm comparing system time one year ago to system time now, that's a duration, and the timezones don't matter. But if I'm not comparing, just recording, then the point of comparison is the unix epoch, which is a point time.

Time zones don't matter to durations, only to points in time, okay. The unix epoch is a point in time. Saying it isn't a point in time, or that its timezone doesn't matter doesn't actually make it so.

"It's the very precise number of seconds since an event that took place 52 years ago, and it has a margin of error of 86,400 seconds," sounds crazy to me.

To be more clear, I hope:

Comparing seconds-since-epoch "now" to the same from a year go: timezone doesn't matter so long as I'm consistent. Pick UTC, pick CDT, pick nothing, just do the same thing both times. This is the vast majority of usages of seconds-since-epoch, I think.

Comparing seconds-since-epoch "now" to epoch: in the abstract, timezone still doesn't matter, except, well, how do use the same timezone as epoch unless you know which timezone epoch was in? You could be off by 82,800 seconds! In practice, assuming that the epoch was Jan 1, 1970, midnight UTC seems to work for those rare cases, leading to the widespread (but technically incorrect) belief that epoch is in UTC.

It doesn't. Equivalent definition of Unix timestamps would be "seconds since 1969-12-31 7:00 pm EST". It doesn't mean Unix timestamps have an implicit timezone of EST.

This does exist as a concept, though I agree that "timestamp" is not a good name for that concept. The Java standard library calls it LocalTime. It is necessary to attach time zone information in order to determine an Instant.

The big problem with the python API, in my opinion, is that they conflate these two very different concepts in a single type.

utcnow() does not return an integer timestamp, it returns a tuple including hours and minutes values, in both Python 2 and 3. Regardless of whether you regard Unix (POSIX) time as implicitly timezoned, the return value of utcnow() unambiguously is.

I'd say the main issue here is that in the article, we have

> ts = 1571595618.0

This a timestamp, there is no TZ data required.

> x = datetime.utcfromtimestamp(ts)

Now for some reason the developer cares about UTC all of a sudden?

If the developer doesn't care about the TZ, they should use:

datetime.fromtimestamp(ts)

A timestamp doesn't require TZ data because the timezone of the unix epoch is defined in UTC. To convert a timestamp into a meaningful datetime value, we need a timezone, otherwise there is loss of information.

The main issue is that .utcfromtimestamp and .fromtimestamp both return naive datetimes, instead of aware ones with the timezone property set to UTC or the local timezone respectively.

> I don't want my equality operator doing timezone conversions

Nobody argues for that. Especially since we’re comparing timestamps (floats), not datetimes (objects). Let’s go line by line :

> ts = 1571595618.0

This is 2019-10-20 18:20:18.000 UTC

> x = datetime.utcfromtimestamp(ts)

I would therefore expect x to be 2019-10-20 18:20:18.000 UTC

> x_ts = x.timestamp()

I would therefore expect x_ts to be 1571595618.0. Which is surprisingly not.

Nowhere the equality operator was involved in the surprise.

The comparison is being done between x_ts, which is a float, and x, which is a datetime object. Call be crazy but I think allowing this comparison to begin with is the problem.

    >>> from datetime import datetime
    >>> ts = 1571595618.0
    >>> x = datetime.utcfromtimestamp(ts)
    >>> x_ts = x.timestamp()
    >>> type(x_ts)
    <class 'float'>
    >>> type(x)
    <class 'datetime.datetime'>

The code in the first example says `ts == x_ts`, not `ts == x`.

Argh.

That's not the code from the article. The code in the article is comparing 2 floats. Though, equality operator would return `False` for comparison `x == x_ts` as well, instead of throwing an exception.

If I’m calling .utcfromtimestamp then by the name I expect it to be doing _more_ than just attaching a time zone (which it also doesn’t do). The only other behaviour that seems sensible is that it’s doing… some sort… of conversion. Exactly what I’d need to look up at the time I wanted to use it.

I have, however, been using python so long that I’ve gotten used to the way things sort-of-work and developed such a healthy caution around it’s unintuitive built-in time zone handling.

I wasn't aware about the functions until this thread, and my expectations match the grandparent. Why is ts not equal to x_ts?

It's actually worse than I had initially suspected. Pythons' dateutil timezone handling is, depending on your opinion, either hard to reason about or just outright broken - certainly it tries to be clever in unintuitive ways. This is why there is a couple of external libraries, like dateutil to try and make timezone things easier - which is even mentioned at the beginning of pythons' datetime documentation!

In this case, I believe that it's actually the inverse from what I thought - utcfromtimestamp is doing... nothing; but is different by the fact that nothing is different from everything else. It looks like .fromtimestamp builds a "naive" datetime in local time. So, it assumes the timestamp is UTC and "helpfully" converts it to the local date and time. When you do .timestamp it inverses this process and gives you back the timestamp value in UTC by converting implicitly from your local time zone. Thus, if you do .utcfromtimestamp it doesn't do this conversion, and gives you a naive datetime "in UTC".

This is horrible and I fully support the recommendation never to use it.

Turns out the answer is what the article is all about.

> Likewise, for a type that has a timezone attached, if the timezone differs between a and b, then I want a == b to return false, even if it represents the same instant in time.

Really? So if you want to compare that two times are really the same time for two different timezones, you'd want to convert them to UTC first? What's the use case for this?

Seems like the less-used way to me.

You don't need to convert them to UTC, just one to the timezone of the other, but yes. "a.is_same_instant(b)" is a lot more clear to me than "a == b", especially in a dynamic language where it might not be obvious what the types involved are.

Hm, I don't know, I guess == on datetimes means "is the same instant" to me.

Noone else has to guess (or go and inspect what an overloaded comparison operator actually does) if there is a meaningful name attached to the comparison, like in the parent comment.

Sure, but there's also a reason we don't have `x.is_numerically_equal_to(3)`.

And that reason is that there is a canonical definition of what it means for two numbers to be equal.

The fact that your intuition of "these two timestamps are equal" is "these two timestamps denotes the same instant" seems problematic when we know that the notions of timestamps and time are not in fact aligned (because, for example, of leap seconds).

That's what it means to you

But an API should never get into a place where your personal choices should matter and various incompatible interpretations potentially make sense to different people.

This sounds like turning a good API into an unreadable mess. And no, it is not pythonic

No, most people do want to compare times in different TZs without the extra cruft.

It would be turning a really bad API that is a constant source of errors into a slightly better and less error prone API. It's totally possible that "pythonic" means "error prone", but that would not be something to tout proudly if so.

I think a better API would be to have a different types for timezone- naive vs. aware datetimes, and a canonical way to convert either of those types into an instant in time (requiring a timezone be attached for the naive type). Then the instants in time could be compared with the equals operator without confusion.

> It's totally possible that "pythonic" means "error prone"

Those who say that don't know what it means. And given how much I have to deal with crap APIs that disregard those principles it's safe to ignore discussions leading elsewhere

Ok, let me take this a bit more seriously. What "pythonic" principles does the existing API maintain that the proposed "crap API" disregards?

Reply on news.ycombinator.com