Debian 13, Postgres, and the US/* time zones
rachelbythebay.com
rachelbythebay.com
* https://data.iana.org/time-zones/tzdb/backward
If one traces references, one finds this connected bug on Launchpad. Amusingly to anyone who has ever seen these sweeping timezone database changes over the years, Launchpad marked it as "This bug affects 1 person.".
* https://bugs.launchpad.net/ubuntu/+source/tzdata/+bug/200807...
The rules for the "backward" file are here:
* https://data.iana.org/time-zones/theory.html#naming
All of the US/* timezone names, such as US/Pacific here, have been backwards compatibility measures in place for the whole of the 21st century and some of the late 20th. The Olson database in the 1980s (mod.sources v08i085, comp.sources.unix v14i030) used these names. But the naming scheme changed somewhen in the 1990s to a continent/city and ocean/city form and backwards compatibility with the old names has been preserved ever since by the "backward" file.
* https://groups.google.com/g/comp.unix.solaris/c/ON_MPZxVdv0/...
Debian has moved into a non-depended-upon package a backwards compatibility measure that is as old as Debian itself.
That just means no one has clicked "affects me too" button yet (after logging in).
US/* was moved to 'backward' (the file for backward compatibility) in the tz database in 1993(!) and as such was essentially marked as deprecated long enough. https://data.iana.org/time-zones/tzdb/backward
You're telling me you didn't notice ? It was on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard.
In a large fraction of cases in the FOSS world, it comes across that the developers really do want to communicate this sort of thing, but there's no clarity on where or how they should do so. See for example various deprecations in Python packaging tools (and standards).
* https://data.iana.org/time-zones/tz-link.html#tzdb
* https://web.cs.ucla.edu/~eggert/tz/tz-link.htm#tzdb
Paul Eggert explained the continent/ocean plus largest city naming convention on a WWW page almost a quarter of a century ago. The WWW page was so well publicized that you can find its URL baked into at least four of the O'Reilly animal-cover books from the early 2000s.
* https://web.archive.org/web/20011023074744/http://www.twinsu...
It was explained on Usenet and on mailing lists prior to that.
An example of this falsehood? Well, in the 70s my father convinced most of my hometown, at least the portions between Main St and Wharf, that DST was absurd.
For almost an entire year, this was observed.
Do you think they kept record in tzdata? I tried to convince them, but no! I still have some dateplanners my father had printed up, and even a picture of the sign that was out on the road (to alert visitors!).
But no!
Do not trust the tzdata people. As you can see, they are not so accurate.
I frequently kick myself for losing track of that book.
> "The tz database is not authoritative, and it surely has errors. [...] Errors in the tz database arise from many sources: Sometimes, different people in the same city maintain clocks that differ significantly. [...] Even if the time is specified by law, locations sometimes deliberately flout the law. [...] Any attempt to pass the tz database off as the definition of time should be unacceptable to anybody who cares about the facts."
Continents are conventional are there are multiple competing conventions. The same place absolutely can and will move continents if you decide to swap out one of those conventions for another.
Or consider Cyprus-traditional geography assigns it to Asia-it may be an island, but that’s the nearest continental landmass-and so IANA tzdata lists it as Asia/Nicosia-but since it (or at least the internationally recognised part of it) is in the EU, a lot of people view it as part of Europe. And the standard name for its time zone is “Eastern European Time”
Nobody can really find fault with Asia/Jerusalem, whereas either Israel/Jerusalem or Palestine/Jerusalem would be controversial.
I had another issue during my upgrade to Debian 13 that also wasn't mentioned in the release notes. I filed a bug report, but I was told that the issue was not important enough to put in the release notes, and I should have instead more closely read NEWS.Debian of the package. So I don't really know anymore what the "Issues to be aware of" chapter in the release notes is for, because apparently it's only a small selection of issues you need to be aware of.
(There is a tool to convert the old configuration syntax to the new, but it requires a working installation. There is a command line option to enable support for the old format, which if nothing else helps to be able to run the conversion tool, but there's no information how to enable that command line option in the way Debian starts pdns-recursor.)
The operational upgrade failure is if you have any existing drop-in config that sets `server_tokens off` already a hard fail Nginx error about duplicated keywords will occur, causing the apt process to exit with failure code(s) during the standard upgrade process.
[1] https://github.com/nginx/nginx/blob/master/conf/nginx.conf
* https://packages.debian.org/source/sid/nginx
* https://salsa.debian.org/nginx-team/nginx/-/commit/3e7838a6b...
* https://salsa.debian.org/nginx-team/nginx/-/merge_requests/8...
timedatectl list-timezones
For your convenience, here is a list for a Debian 13 box with 628 entries:
https://pastes.io/output-of-timedatectl-list-timezones-on-de...
Some of the other Australian entries are bizarre though - 'North' and 'South'? There is no standard timezone across the whole north or south of the country. And Broken Hill is in NSW, so why are they different time zones?
- Debian 13
- Interactive Brokers' Trader Workstation
- Racket's `gregor` date and time library
IBKR still sends the old, deprecated US/* timezones. As noted in the article, the solution is to `apt install tzdata-legacy`.
https://gitlab.com/gitlab-org/gitlab/-/issues/556779#note_26...
The fix is simply to change them to the non-backwards linked names but it caused some confusing API errors since data which would no longer pass validation was already in the database and didn’t look wrong.
(That's what I heard anyway)
But they explicitly or implicitly have also decided not to store the timezone in the strings, so every single value is technically ambiguous. Absolutely drives me crazy.
Update: since there have been questions, these are strings, not native datetime values.
Not sensible at all for future date/times.
You have to make certain it is always handled as UTC, and if you export it to another system without making the timezone explicit, or anyone who is unfamiliar with the convention views it, you’ve lost the guarantee.
When you’re shuffling around petabytes of data, adding another few megabytes to include the “Z” at the end of every timestamp is hardly a major expense.
On the other hand, UTCx timezones are not silver bullets if your fleet is a part of a multi-continent federation.
I wouldn't expect to see all the important news of the tens of thousands of packages I don't have installed in the release notes.
I was hit by this using unstable (before they made a NEWS item), but when upgrading my stable machines to trixie I got a proper warning/reminder of this specific thing.
0: Note that different cultures disagree on whether there are two continents called North America and South America, or one continent called America.
> At the time, I went "WTF?" and just commented it out to get it running again. I had bigger fish to fry... and just kind of forgot about it. Everything seemed fine.
I get this, but now you get to laugh and dust yourself off. If it had silently started doing the wrong thing, that would be a worthy complaint.
I mean, the folks who run the tz db definitely know what they're doing, it just never 100% clicked with my thinking.
I always prefer `US/Eastern` over `America/New_York` -- it seems more "canonical" to me. New York is _currently_ the anchor city for ET, but will it always be? The place I live (Boston) is currently on ET, but in the future it might be on Atlantic Time. If there was an `America/Boston`, I would use that to be safe, but since there isn't, it just seems better to be to be specific that I mean "Eastern Time" and not "whatever the time is in NYC"... At least then if Boston switches to a different tz, I could intentionally switch to "Atlantic Time" -- doesn't that make more sense? Versus I guess what I'd have to do, which is switch to `America/Puerto_Rico`? (I had to actually search that one, too bad there's no `US/Atlantic`...)
(I'll note that I agree with the general wisdom to store data in UTC; here when I talk about zones I'm either talking about user local machine time or display)
It's one guy. He demonstrably does NOT know what he's doing.
> I always prefer `US/Eastern`
As you should. It's the actual name of the timezone as published by the entity that defines it. Outside of the goofy definitions in the tz file it's what everyone living _inside_ of that timezone would call it and see it referenced as.
To call this "backwards" is an absolute insult to civil time keeping and drives me insane.
> doesn't that make more sense?
I always thought it should be served through DNS. Then each country can just define it's own TZ record type and embed it at the root of their country code domain and could expand on it however they like.
eastern.timezone.us
Also, since domain names have punycode for internationalization, you could actually call timezones in countries like Mexico what they're actually named for end users.
You could use this to promulgate SRV records that direct you to a country’s authoritative time servers, too.
This hit me in the early 2000s and now everything I do is in UTC. All dates, timestamps, everything, UTC. If you want to look at a local window in time, convert the window to a utc start and end date and search. When viewing, use a js function to translate the utc date to a local one to print. The mental gymnasium of local to utc to local again…
> The two difficult things in software engineering, naming, and timestamps.
For example: “this meeting will take place at 10 AM on July 31st, 2026, US Pacific time” cannot be expressed in UTC. You can guess what time UTC that refers to, and you’ll probably be right, but you’ll be wrong if it turns out for example that the US abolishes DST before that date.
Moreover, most people in real life will understand that to be a time and won’t care how it maps to UTC.
https://lists.iana.org/hyperkitty/list/tz-announce@iana.org/...
You're running a database system and you just casually comment out the configuration setting the timezone?
In what way did everything "seem fine"? SELECT 1 returned something? No further investigation required??
Config defaults somewhere still using them? Man page examples? Tutorials using them? Or just force of habit?
When was the last time you rebuilt your company's postgres config from scratch?
Last year, when we upgrade to version 17.
I looked at the example/template configuration, diffed it with our configuration from PG15, and for every change decided whether to keep our version or the new setting.
I didn't use it, but Debian/APT has had a tool to do this sort of comparison for any software upgrade for as long as I can remember.
Do other people just copy the old config and shout "YOLO!"?
And you might say "well I know how a statement timeout works" and I agree! I would also generally agree that something like a timezone setting would generally be something I'd expect to be fairly stable.
That's what I meant about rebuilding the config from scratch. Rebuilding from scratch would almost involve _not even looking at the existing configuration_ and then doing first principles footwork to figure out what is needed.
I think the diffing flow you went through is the right way to do it, but I believe that flow might lead to some values getting less scrutiny than others. Still perfectly reasonable though
Because of a lack of things compelling people to change them until it causes a breakage. And then when it does cause a breakage, most people would rather move heaven and earth to complain, research workarounds, etc. rather than just change it. (Institutional structures can also make "just" changing it far harder than that should be.)
Given the old names were deprecated in 1993, it's hardly surprising that I never before discovered "GB".
And for most of its life it’s been an explicit policy that timezones are named by continent and representative population center, not by country, to avoid entangling it in territorial disputes and improve naming stability for historical data. The US/* (and Canada/*, etc.) names are deprecated and have been since 1995 (?), but apparently people were still using them because the deprecation wasn’t really apparent unless one was especially into reading release notes.
I’m just joking all the way, by the way :)
(Detroit gets its own zone because it was significantly different to new york before 1970, normally it's only difference post 1970 which merit a new zone)
You did read the “I’m just joking” in my previous post, right? I don’t give a shit as long as I can select a city in my country and time zone :)
American exceptionalism time zones aren't used since the 90s. even the cpus from that time are already dropped from kernel support. heck even the text encoding is gone.
If anything, the city TZ always felt off, like I was opting in to that specific city’s strange legal decisions or something.
The timezone selector in KDE shows "Los Angeles | America/United States of America | Pacific" and "London | Europe/United Kingdom", for example.
that is exactly what time zones are for :) not being snarky (wasn't before either, i really love that blog!). but the whole reason for tz is to join the ever changing oddities of political bodies from one very specific region.
(It doesn’t, but that’s what it implies to me.)
One important thing to understand is that the time zones of the tz database, and hence generally the time zones used in computing, are a slightly different concept than legislative time zones.
Still feels weird, though. What if LA specifically passes a time zone law so that now it’s sometimes wrong for everyone else in California. Do we add an America/Cali_except_LA zone?
That’s probably hypothetical. It seems unlikely. But what a major pain in the ass if it did happen?
Yes, it's silly and inefficient, but so are time zones. It's not an easy problem to wrangle on a computer, for reasons that are exactly as you have described.
It’s an inherently complex, ugly mess.
So the trade-off is timezones being specific to a particular city but remaining unambiguous forward and backward in time.
You can’t avoid pain when there is a change in the geographic area of a legislative time zone. But you can avoid the case of a time zone ID becoming ambiguous in terms of the UTC<—>local time mapping it’s supposed to define. The latter is the aim of the present scheme.
For example, they patch OpenSSH source code in a way that makes defaults behave differently than upstream. In the name of backwards compatibility of course.
I assume this will continue until it doesn't anymore, and the only notification you shall receive from the ivory tower is a cryptic one-liner buried in a changelog somewhere.
I prefer real choice and light patches that try to upstream as much as possible, or workaround upstream obstinacy rather than create incompatible idiosyncrasies. One area that isn't well represented in barely a/no distro is init freedom neither married to nor completely divorced from the sprawling octopus.
As an actual answer, it's not too bad on Debian; we only really use/need: systemd (system and user), -logind, -journald, -udevd. All in all, not too many tentacles but there are a few...
Also, I worked in the same department as Russ once upon a time in a galaxy far, far away.
Isn't it the same thing with the RedHat downstreams ? (Not necessarily OpenSSH but other packages)
IIRC RedHat do all sorts of things to keep their gov / corp customers happy, also usually in the name of backwards compatability, all of which then end up in the downstreams.