Investigating why Steam started picking a random font
blog.pkh.me
blog.pkh.me
I can only imagine that a lot of other legacy software will have similar issues when we reach year 2038.
That's actually very clever. Instead of crashing in unexpected ways or doing odd things, just cleanly exit. If you really want it to run after 2038, you have to emulate the clock, which would then avoid these potential Y2038 bugs.
Although the most clever solution would be to not have any bugs in the first place ofc ;)
Also take a look at struct tm. Its tm_year looked like just a 2 digit year and as such people may format it with printf(“19%02d”,…). It is actually the number of years since 1900. In early 2000 I had to fix a broken ftp server that was sending 19100 as the year.
I think it's likely to be better handled, but at the same time people keep citing the non-disaster of Y2K as a reason not to do disaster preparation, so I don't know.
I use EPOC time in my own code NOW, a lot... :-/
As I understand it, it seems like it’s mostly software using 32-bit integers that will struggle.
So if you’re writing modern code on a modern runtime running on 64-bit platforms you should be fine (easy to verify by changing your dev environment’s clock).
Out of support, aging 32bit SPARC hosts running 4.6c SAP.
That's production!
replace "int64"
I kid, fixing this will require an even greater amount of effort compared to the Y2K bug considering how many more linux devices have been deployed since then.
I do think though there were some bioses that messed it up too, so that's rather low level too.
Total coincidence, but fun to think of the conspiracy theory :D
I know, time_t is 64-bit with pretty much any new Linux distro out there, so why are people seeing Y2038 issues? It’s because the Windows 32-bit POSIX compatibility layer handles Y2038 very poorly. Once Y2038 is reached, the POSIX time() call in a 32-bit app fails with a -1. It doesn’t use a rolling timestamp somewhere in 1901 the way 32-bit Linux applications with 32-bit time_t do. It fails hard, returning -1 for every call to time().
Now, it’s true that Microsoft does have proprietary calls for time and date which are Y2038 compliant, and, yes, native Windows32 apps should use those calls instead of the POSIX ones, but in the real world, it’s sometimes a lot easier to, say, just use stat() to get a file’s timestamp instead of having to use CreateFile() followed by GetFileTime().
This is why a lot of Windows apps are still seeing Y2038 issues.
In terms of Linux apps, the Y2038 stuff is mainly seen in old 32-bit binary only apps. Since that stuff is mainly games, where an inaccurate datestamp isn’t a serious issue, I think we will see emulation libraries which give old games a synthetic time and date so they aren’t outside of the Y2038 window. New apps will use a 64-bit time_t even if compiled as a 32-bit binary.
Before 2000, we had software running our systems, yes. But it was not as distributed, and not as ubiquitous, and not as deeply ingrained into human culture as it is today. This proliferation will obviously continue past today, and while hardware and low-level OS/software mitigations (as well as a herculean effort to clean up the mess) will make up the gap, it's not hard to see that this is likely to be much more impactful upon failure because of the "embeddedness" of these systems.
A box that has just been doing its thing for 40-50-60 years and all of a sudden fails, is likely to be more impactful than one that was 20-30 years old even.
Case in point, this article's pointing a finger at fontconfig. Who considers fontconfig mission critical software? fontconfig has been open source forever, is it just "too boring" that no one has bothered to do a Y2K38 audit on it? It probably doesn't even really care about dates for the most part, so maybe no one even realizes it needs a date math audit? Multiply that across the very long tail of Unix apps and libraries since 1970. That's the weirder risk of Y2K38 than Y2K: the huge amount of "non-mission critical"/"non-date math" code that potentially exists in every Unix-derived tool. (With all non-Windows OSes in common usage today themselves being Unix-derived, that's a lot of surface area.)
Y2K was looking for everything that did date math with varchar(2) or 2-digit BCD. Those were needles in haystacks certainly, but the needles were sharp enough to know when you found one. Y2K38 is looking for subtle differences in (mostly) C macros and C library function calls and making sure that time_t structs are appropriately sized for modern platforms. That almost sounds to me more like looking for particularly colored straws in a haystack.
Virus? Possible, but unlikely. A virus wants to spread, not limit its opportunities to do so.
After investigating, I was able to determine that the system clock somehow had gotten set past 2038 and this was sufficient to destroy network connectivity. As soon as it was corrected, everything was fine again.
Not the first time I have run into a clock issue breaking software.
We are lucky 2038 is still 16 years away and not next month.
Thanks for throwing me into an existential crisis... /s
May make some extra money right before that.
There's parts of me that still think that 2020 will be the great shiny future, even though we all know that it turned out to be an epic bust. 2050? Can't see it from here.
Windows expects the system clock (the one you can set from the BIOS that keeps time when you don't have NTP) to be set to the local time zone by default. Linux and most other operating systems expect the system clock to be set to UTC.
Usually it's easier in a dual boot environment to set your non-Windows operating system to treat the system clock as local time, most Linux distros literally have a checkbox for this, but sometimes this isn't an option (IIRC Mac OS on a Hackintosh is one of these cases) and sometimes you just want to stand on principle that UTC is "correct" and make Windows adapt to what the rest of the computing world agreed on.
In that case, you can open up the registry, navigate to "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\TimeZoneInformation", and create a QWORD "RealTimeIsUniversal" which is set to 1. Reboot and now Windows will treat the system clock as UTC.
I vaguely remember a few different timestamp formats in use in different places, but the 100ns-tick is very common in Microsoft APIs.
FILETIME is a nominally unsigned 64-bt value. 1601-01-01 + 2^64 * 100ns = 60056-05-28. That is, Sunday May 28, 60056.
I messed something up about it, and must have been pulling from the wrong graphics memory, and noticed That the brown wood didn't have a swirl pattern but had what looked like text in it?
After staring a bit closer, I noticed it was the text that was written to the console in visual studio, I had somehow brought that graphics buffer in to use as my "swirl pattern". I had to sit back and think a bit about how data on your computer isn't always as safe as you think sometimes after that...
I've seen many bits of cahced browser viewport textures when writing shaders and making mistakes. I've been wanting to create something procedurally from it somehow ever since I encountered it.
There's no separate system call for the modification time; a single system call (https://man7.org/linux/man-pages/man2/stat.2.html) returns the three times (atime, mtime, ctime) together. The font library probably wanted just the modification time (to check whether the font cache is stale), but it cannot get the mtime without also getting the atime (and ctime).
EOVERFLOW: pathname or fd refers to a file whose size, inode number, or number of blocks cannot be represented in, respectively, the types off_t, ino_t, or blkcnt_t. This error can occur when, for example, an application compiled on a 32-bit platform without D_FILE_OFFSET_BITS=64 calls stat() on a file whose size exceeds (1<<31)-1 bytes.
I'd be surprised if it was actual userspace code in the font library that was generating that errno. After all, if you care enough to spot an overflow in your calculations, you probably care enough to handle that error case better (and know enough about the situation to handle it properly). Something must be making a specific syscall, getting EOVERFLOW, then throwing it back up to the user. But is it really the ubiquitous stat() ?
They cover both, but they focus more on the glibc wrappers. For instance, the manpage for stat(2) we're talking about says "On success, zero is returned. On error, -1 is returned, and errno is set to indicate the error.", which is not the syscall interface return value (the syscall does not know about errno, it returns the negative of what will end up in errno instead of -1). Another example is the manpage for exit(2), about the _exit() function (the exit() function, without the underscore, is at exit(3) since it's not a system call), which says "In glibc up to version 2.3, the _exit() wrapper function invoked the kernel system call of the same name. Since glibc 2.3, the wrapper function invokes exit_group(2), in order to terminate of the threads in a process."
I tend to prefer failing before returning wrong data. Who knows what the program using the function will decide to do based on that data...
There is no way to be sure the date in question is or is not important. In that case, instead of giving someone a lifetime exposure to ionizing radiation, or pausing the respirator because the last time the person took a breath is in the distant future, it's better to crash right away so a person can figure out what to do.
The original post is about a game distribution client. You don't think that the approach for health/safty SW and entertainment SW can have different approaches to error handling?
Also remember that the latter class of software is usually abandoned after a short time so there will be noone around to fix the bug once a user runs into your fail-early check. So yes, pretty please don't fail early in release builds of single-player offline games and other SW with similar characteristics - currupting some state that *might* result in problems is much preferable to intentionally making the game unplayable.
Its also a matter of how you fail and how much. Surely you don't want your computer to BSOD if there is any application error so maybe there are other cases where you want to limit the scope of the error handling too. E.g. zeroing out the atime if it overflows could make sense. Failing the whole stat call likely causes more probblems than it prevents.
I'm not certain I'm looking at the correct file, but that does seem to be the case, at least on latest glibc: stat redirects to fstatat(AT_FDCWD) (https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/uni...), and fstatat calls a 64-bit system call and fails with EOVERFLOW if any of st_ino, st_size, st_blocks, st_atim, st_mtim, st_ctim wouldn't fit in the 32-bit struct (https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/uni...).
Yes, st_ino too, which means it could also break on a filesystem with large enough inode numbers. Using a 32-bit userspace nowadays seems more problematic the more I look.
Yep, been broken for 8 years in some Valve's games: https://github.com/ValveSoftware/Source-1-Games/issues/1685
The downside to having no abi guarantee is that you will not have old binaries to run in the first place, hope you remembered the source. sigh
All things considered, if you have the source to everything, abi is overrated, if you don't, it is vital.
extra thoughts: obenbsd is cool because they don't have or need the *64 file access functions. (fopen64, fseek64...) however this sucks when porting... because they don't have the *64 functions.
The correct way is to create APIs that take a 64-bit time_t and migrate applications over to them. No ABI guarantee means the old APIs can be removed if they are a burden to implement, but obviously for the case of time_t they aren't, so sticking a warning message in there is sufficient for the next 15 years or so.
> All things considered, if you have the source to everything, abi is overrated, if you don't, it is vital.
ABI might be, but API isn't. Even within a single application, the correct way to do internal interface changes that affect a lot of code is generally to create the new one, move callers, then remove the old one. Certainly in a case like this where keeping the old APIs around is trivial.
And OpenBSD does not have the source code to everything, and even in ports, there tends to be an upstream and issues with porting.
typedef time_t int32_t
to
typedef time_t int64_t
does not change your api
openbsd tends to be respectful to the api
however the binary interface changes every couple of weeks. and they have a flag day(breaking incompatible change) every year or so.
As such, actions that are unthinkable on linux, like an abi flag day. The openbsd project has gotten really good at handling, after all, if you break stuff all the time you get good at picking up the pieces. to misquote Raul Julia "For you, linux, the day your abi changed was the most important day of your life. But for me, it was Tuesday."
This means that the openbsd project is exceptionally unfriendly to binary only programs(commercial software), As much as I like openbsd I would not even try.
Not sure I agree, because time_t itself is part of the API, and programs can use it for more than just calling your syscall, like in their own structures.
Linux has found they don't need these flag days, they're an ugly old sledgehammer that used to be quite common in systems programming, but Linux (and presumably Windows though I haven't seen the source code to make a judgement) really pioneered much more disciplined, thoughtful, and structured way to manage API and ABIs such that new versions can be brought in with little disruption and old versions can also be maintained usually with little burden to the code base. It's a better system all around IMO, even if you did decide to remove the old stuff right afterwards, the change process is just the right way to go. And keeping around the old stuff and not having to change the world or break your users is actually a good thing too, the ability to make changes less painful than these big hammer flag days makes things very flexible and adaptable.
Even without the Y2038 issue, lots of things use timestamps for cache invalidation and might mess up when things are updated while you are in the future.
* Or use Big Picture mode, which has other issues.
It used to be VGUI and last I checked there were still some dialogs that weren't updated.
click
"Hm, yes."
It's not out of the realm of possibility, but I'd be a little startled if Windows hasn't already been gone over with a fine tooth comb for Y-2038 issues.
Odds are high I just saw the headlines about how they are stopping 32 bit sales of their OS. Though, I couldn't tell you for sure what I was misremembering.
Spoiler ahead: But I'd use faketime in userspace to avoid messing up the system ;)
Never did understand why the software even looked at dates, and support couldn't explain it either.
Indeed it will be!
Ooh, Friday the 13th too, ominous.
If fontconfig fails to stat() a font file, it presumably aborts trying to record any information about that font file. Notably, it doesn't know what the font file's actual font name is. If all fonts fail due to stat(), no information will be available about any font. fontconfig has multiple levels of fallback (e.g. "similar" fonts first), but in this case since nothing is known about any font you just get whatever font happens to be first in the list of all fonts.
Honestly, everyone should run their filesystems with the `noatime` option set, so that they never record access times at all. We very rarely care about when files are accessed. Usually we care most about when a file was last _modified_, the mtime, so this loses us almost nothing and saves a huge amount of writes to the filesystem at the same time.
atimes aren’t completely useless, just useful less often than mtimes.
It is, and that was the problem. The legacy 32-bit API the library loaded within Steam was using, or more precisely, the legacy 32-bit system call used by that library, could not represent the 64-bit time, so the kernel returned the EOVERFLOW (Value too large for defined data type) error. The root cause of the problem seems to be that Steam is still a 32-bit application (and hasn't been recompiled to used the Latest and Greatest version of glibc, which allows for 64-bit timestamps even on 32-bit processes).
If his Linux installation did not use 64-bit file times, the timestamp stored for the file would have fit in 32 bits, and the error wouldn't have happened (though other things would probably have broken first).
(Well, actually, Linux is using 34-bit file times, at least on ext4 and xfs, but that's a bit of a nitpicking: what matters is that it doesn't fit in 32 bits.)
Edit: it seems that it's actually glibc that's returning the error, not the kernel, see https://news.ycombinator.com/item?id=33675526
genuine question -- no snark intended. Is there some benefit other than completion?
The software can also re-lock achievements you've already gained, like if a family member plays on your account and unlocks some that you were planning on earning during your playthrough. Or if you just want to reset your achievement progress for some reason.
The other day I was playing around with modding a game, and inadvertently unlocked some achievements I didn't truly earn, so I re-locked them with the achievement manager. I'll wind up unlocking them in the future during normal play.
Of course you could also use it to cheat, and instantly gain 100% completion in a title.
When I was still using MacOS, I have not noticed anything special about its font handling. The difference in comparison with Linux or any other free OS was that MacOS included a very good set of high quality fonts, much better than those provided by default in any Linux distribution, not that it had some special font handling. Because of that, even after I have ditched MacOS, I have kept a few of its typefaces. Even today I am still using a couple of them.
On Linux, it should be possible to obtain any complex or weird font matching behavior that might be desired, by editing the fontconfig rules, which should be located in some directory like "/etc/fonts/conf.d/", but the exact place might vary between Linux distributions. However, I have never attempted to do that, beyond establishing nice defaults or replacements for the fonts that might be specified in Web pages, e.g. "serif", "sans-serif", "Arial", "Times New Roman" and so on (i.e. by editing 60-latin.conf, 60-generic.conf, 45-latin.conf, 45-generic.conf and the like).
The main method by which I have ensured that everything is displayed with beautiful typefaces on my Linux computers has been simply by uninstalling all the default fonts and installing other nicer fonts in "/usr/share/fonts/". If you omit to uninstall some ugly font, there would always be some application or Web page that might insist to use that, despite your attempts to suggest better fonts.
Many years ago, I have bought a number of beautiful typefaces from some on-line stores like Linotype, Adobe and others, and I am using mostly those on Linux. Nowadays it is much easier to replace the default fonts with better ones, because, unlike a decade ago, now there are a relatively large number of good fonts that are open-source or at least free of charge.