Pytz: The Fastest Footgun in the West (2018)
blog.ganssle.io
blog.ganssle.io
The project has also been backported to 3.6+, so you can still use it if you're stuck on a legacy/unsupported version.
See https://www.python.org/downloads/ and https://pypi.org/project/backports.zoneinfo/
An unofficial Python installer for Windows 7 exists: https://github.com/adang1345/PythonWin7 (it uses an unofficial implementation of api-ms-win-core-path-l1-1-0.dll).
Have you tried building 3.9 from source to see if there's a particular API that's unsupported?
No, I haven't tried to build from source. That's a rabbit hole I'm not particularly interested in jumping down.
Keeping everything in Zulu ms or ns and rendering into timezones as late as possible is a useful guideline as far as it carries one, but sometimes it’s not enough, this is one of a zillion reasons why e.g. calendaring is just really hard.
I’ve long since mentally tagged “timezone handling”, in any language, as a slow zone: let’s stop and really think this through.
As in, if I set my alarm for 08:00, I'd expect it to wake me up 08:00 local regardless if I'm in Oslo or I've traveled to say New York.
However if I make a calendar event things are less clear cut. For meetings and such it makes sense that the time is global time (hence stored in UTC), but for tasks it can be be either or, depending.
In many cases reasonable assumptions can be made, e.g. a calendar event for two people in different time zones to meet on a video call almost certainly needs to be scheduled for a proper time (or likely UTC, which is close enough since if you can ignore leap seconds). Although even then I wonder what the two attendees would expect if, between the scheduling of the event and the event itself, both of their local time zones changed their DST rules such that their UTC offset changed by +1 hours. If only one time zone changed its UTC offset we might assume that the previously agreed upon UTC time would hold, but it’s hard to say what the humans would expect if both of their time zones shifted the same way.
Then again, you don't want all your meetings changing by an hour half the year to avoid a one-week discrepancy.
Another bad example is using Windows bootcamp on a mac. Windows writes local time to the RTC and MacOS writes universal time, so when switching between MacOS and Windows, it's always wrong at first.
If you're in Pacific time and you buy the ticket for a date in the future (let's say June 1st) before 3pm your local time everything works fine. But if you buy the same ticket for June 1st after 3pm Pacific local time (which is already the next day in Vienna) the ticket that you're buying isn't for June 1st but actually for June 2nd (only visible once you receive the confirmation).
So apparently they're somehow translating between timezones for the displayed dates, although that doesn't make sense: If I buy a ticket for June 1st on Vienna public transport I don't care what my local time is. Even if I'm in California while buying that ticket I still want it to be valid on June 1st instead of June 2nd.
I seem to recall there's a registry key to control that and have it store UTC to the RTC
edit: found that one https://superuser.com/questions/975717/does-windows-10-suppo...
I'm with you as well. Sometimes you don't have a choice when dual-booting, MS-DOS only wants to work in local time while on the opposite end of the spectrum MacOS only wants to work in UTC, but if I have the choice I'm going to change Windows to use UTC.
lusers who don’t know what those things are will learn soon enough.
I really mean it. The ambiguity is a problem.
Google did 100% the correct thing, but it goes to show that this is just an inherently tricky thing not limited to whether you are writing code.
Let's say you have have a repeating 7am alarm on your phone. What should happen if you want to set it to 8am for one day? I mean, without walking you through a dialog that looks like: do you mean today 8, or tomorrow 8?
I think this is tricky too - if you have a 9AM meeting, if your time zone switches, should the time of your meeting change too? Most of the time, no, but if you have a meeting with other people in far away time zones, maybe you don't want it to change.
some of my perennial favorites:
- in samoa, there was never a december 30th 2011.
- in indiana, time zone observance is determined per county
- in arizona, the reservation land observes dst, and the state doesn’t, and reservation lands can enclose non-reservation lands.
- lebanon recently had a legislative dispute about dst such that some entities in the nation did or did not observe dst for about a week.
That was interesting, so I had to read about it. Apparently they skipped a day to be closer to Asian time zones. They did the same thing in the opposite direction in 1892 by having two 4th of Julys to be closer to American timezones. Now they kinda just walked back on that.
Interestingly I think the Python datetime approach can handle both of these with no problem. The rest of your list sounds... a lot tougher.
Is that timestamps as in unix timestamps? That won't work with birthdates for people more than 53 years old for example; a colleague mentioned a weird issue for someone born in 1938, but that was because the timezone in that location at that time was something like GMT+1:23, then the Germans came around and standardized it to +1:20, then later it was normalized to +1:00.
tl;dr, time and date can only be accurate if time, date and location are known. Actually, some places have moved to different timezones over time as well, so you can't just say "this time has a timezone of GMT", you have to ask "what is the timezone of this location at this time/date"
If I say the event happened 60 years ago exactly, and it’s 1pm, what do I mean by “exactly?” Do I mean the day 60 years ago, at 1pm? Or 60 years ago, taking into account timezone shifts? Do I, the person entering this information, even know or care about this distinction? If you enter an event as 60 years ago, would your average joe expect that the time would shift by 23 minutes, or would they think there’s something wrong with the software?
What about timezone differences that vary depending not only on geography, but also on the ethnicity of the people associated with the event?
I'm sorry but if you don't even know yourself what you mean, the computer cannot know it either. The topic here is how to store a datetime, not how to guess meanings.
> What about timezone differences that vary depending not only on geography, but also on the ethnicity of the people associated with the event?
Again, display conversion. Have these natives put their date system into the timezone database and make the UI have a selector for this timezone. Under the hood it still happened at a certain actual moment in time which is what should be stored in the database.
No significant difference from any other timezone that you have to apply to a unix timestamp to get a display time, or conversely, apply to input data to get the timestamp to store.
Typically, recently, this problem would occur in disputed territories where the “legal time” at a certain point differs because it is claimed by different polities with different timezone rules, so its more of political allegiance than ethnicity (though those correlate).
[1] https://www.timeanddate.com/worldclock/china/urumqi
[more background here] https://www.theatlantic.com/china/archive/2013/11/china-only...
So Zulu for the past, datetime+location for the future.
I will second you ok that: date & Time handling across timezones was one of the 2 hardest things I've ever worked on that are deceptively simple on the surface. The other was adding localization/making an app translatable to any language after shipping the first version with hard-coded text.
And after thinking about it a lot more, I think this attitude is actually more sensible than mine used to be. Why do we as programmers keep trying to fit a square peg into a round hole? Timezones are a political construct, not a mathematical one.
We should address edge cases by deferring to people as much as possible. Oh, you find it annoying to enter an offset? Don't worry about it, it's optional! Your dentist will expect you for your cleaning at 8:00am +/- 5 hours.
If we keep smoothing over the rough edges for our users, the problem will only get worse. Better to make timezones as painful as possible for as many users as possible. They'll eventually vote for something less obnoxious.
> If we keep smoothing over the rough edges for our users, the problem will only get worse. Better to make timezones as painful as possible for as many users as possible. They'll eventually vote for something less obnoxious.
This is gonna go over great with the people making purchasing decisions.
For server-side and storage layer, it's saner for everyone to simply store in UTC or epoch seconds/nanoseconds.
We don't change the months in the southern hemisphere. Australia is cold in July, Europe is cold in January, this doesn't seem to bother us.
There may be no technical reason for it, but having a common reference point for times of the day in terms of human experience and behavior is an actual reason. Saying you were walking down the street at 2am or 3am is immediately distinct from saying you were walking down the street at 10pm or 11pm in ways many people understand. Having to qualify it with the date and geographic location and/or sunrise or sunset patterns would be a much bigger PITA than your average person will ever experience because of time zone inconsistencies. It's just not that important for most people to be able to pinpoint a timestamp to one exact time throughout the world.
We've got GMT. Nothing is stopping anybody from using that to schedule their lives. There's a reason people don't.
Whereas it's (relatively) easy to guess, "Los Angeles is in California, that's three hours behind me in New York, so it would be 6 am there at 9 am here, so it's too early to call them yet." Not perfect, but not too bad for simple cases, and easy to do in your head most of the time.
Timezones are about standardizing _human_ time, not actual time.
That can be expanded. Time and dates is just fundamentally hard.
To me it's very often easily worth the extra 50-150MB of install size that just works. I've used pandas for around 7 years now and it was inconsistent and buggy at the beginning. But these days, it's practically flawless. Also I'm aware that pd.Timestamp is built on top of many awesome libraries.
When I left Rails, this is one area I really missed.
Time.use_zone("Singapore") { (Time.zone.now - 3.days).beginning_of_week }
This readable line in datetime utils or even pendulum, is such a pain.
Generally every other library works in terms of standard python types where possible. Very few other places will you see things like a re-implementation of a time library.
...and scipy and all the scikit libraries and whole bunch of others. 'Accidentally' pulling in hundreds of MBs of libraries you never use for a 100 line data processing script is par for the course in python land.
... pytz for example ;-)
But you are right that pandas' use of pytz is benign and as far as I know doesn't suffer the same usability issues.
from django.utils import timezone
now = timezone.now()
later = timezone.now() + timezone.timedelta(minutes=1)
If you are looking to solve a timezone problem right now, then you care about what the standard interface is right now... not "when the library was created".
from zoneinfo import ZoneInfo
from datetime import datetime, timedelta, timezone
localtz = ZoneInfo('Europe/Brussels')
utctz = timezone.utc
a_local = datetime(2023, 3, 26, 1, 59, 59, tzinfo=localtz) # right before change to DST
b_local = a_local + timedelta(seconds=2) # 2 seconds later; should be right after change to DST
c_local = a_local + timedelta(hours=2) # 2 hours later
a_utc = a_local.astimezone(utctz)
b_utc = b_local.astimezone(utctz)
c_utc = c_local.astimezone(utctz)
print('datetimes:')
print(f'(A) {a_local} (utc: {a_local.astimezone(utctz)})')
print(f'(B) {b_local} (utc: {b_local.astimezone(utctz)})')
print(f'(C) {c_local} (utc: {c_local.astimezone(utctz)})')
print('deltas:')
print(f'(B-A) local: {b_local-a_local} utc: {b_utc-a_utc}')
print(f'(C-A) local: {c_local-a_local} utc: {c_utc-a_utc}')
datetimes:
(A) 2023-03-26 01:59:59+01:00 (utc: 2023-03-26 00:59:59+00:00)
(B) 2023-03-26 02:00:01+01:00 (utc: 2023-03-26 01:00:01+00:00)
(C) 2023-03-26 03:59:59+02:00 (utc: 2023-03-26 01:59:59+00:00)
deltas:
(B-A) local: 0:00:02 utc: 0:00:02
(C-A) local: 2:00:00 utc: 1:00:00
For (C), I would expect 2023-03-26 04:59:59+02:00: that is the point in time 2 hours after (A). As you can see by the times in UTC, (C) is only 1 hour later then (A) instead of 2.
Both feel really wrong.
The article compares pointer arithmetic using Python's time zone model vs pytz's time zone model, and concludes that pytz is worse because you have to normalize the result for the time zone to be adjusted. But the pytz result is correct (according to my expectations, at least), while Python's standard approach does adjust the time zone but without properly adjusting the time (and hence is incorrect, according to my expectations). It seems the author only looked at the time zone, without checking the time itself.
I can work around it by converting to UTC and back, fortunately. I had a quick look at Pendulum and it seems it has functions to do arithmetic properly.
Unfortunately, some people dislike the former because it's 3 lines of code (convert to UTC, add delta, convert to local). I've had the same data scientist reject 3 MRs fixing this sort of issue in his code. At least it only causes a once per year error or a blip that no one notices.
Always using UTC doesn't work always depending on requirements. The worst was interacting with an API that required precisely every hour of the following day to be sent without good documentation.
The PEP provides instructions for detecting invalid times (that work as long as you use the standard library zoneinfo database.) In short - for invalid times, the UTC offset will be different for different fold versions (1/0) but with converse order than for times that exist doubly:
# Non-existent wall time (std -> DST):
>>> print(datetime(2023, 3, 26, 1, 30, tzinfo=ZoneInfo('Europe/London'), fold=0).utcoffset())
0:00:00
>>> print(datetime(2023, 3, 26, 1, 30, tzinfo=ZoneInfo('Europe/London'), fold=1).utcoffset())
1:00:00
# Ambigious wall time (DST -> std):
>>> print(datetime(2023, 10, 29, 1, 30, tzinfo=ZoneInfo('Europe/London'), fold=0).utcoffset())
1:00:00
>>> print(datetime(2023, 10, 29, 1, 30, tzinfo=ZoneInfo('Europe/London'), fold=1).utcoffset())
0:00:00It's not an ideal situation, but given path dependence I don't think it can be improved much from here.
I like dateparser for certain use cases:
Can stop and step time predictably. Any code using datetime functions will get the time controlled by the test, with no additional abstractions or other boilerplate needed.
https://arrow.readthedocs.io/en/latest/ (looks good!)
This needs a (2018) (or (2020)) @dang
Wat. The author has no idea what a foot gun is.
Pytz is a must because using the standard library and timezone manipulations often leads to bugs. There are nuances around “naive” date times where one can easily just add a fake timezone value, not realizing that the time didn’t actually get localized to the timezone.
There are other libraries such as Arrow and Pendulum worth checking out but pytz is probably enough.
NYC = pytz.timezone('America/New_York')
dt = datetime(2018, 2, 14, 12, tzinfo=NYC)
print(dt)
# 2018-02-14 12:00:00-04:56
In the article, "fastest" is italicized. The sentence is about whether the footgun is fast, not whether the footgun is a footgun.