Hacker News Reader

Comment by dingdingdang | Hacker News Reader

Thank God someone has the capacity to attack this issue in depth. UTF8 situation with regards to systems level programming is scary and annoying at same time

edit: I note that HN filters out utf8 emojis..

View on HN

> UTF8 situation with regards to systems level programming is scary and annoying at same time

These days one of the systems level programming languages (Rust) is natively UTF-8.

This would have been a huge source of drama in say, 1993 (when UTF-8 is basically brand new) and maybe even in 2003 (by which point it's clear Unicode is a success, but still conceivable that UTF-16 "wins" in some sense) but in 2023 people just shrug - obviously it's UTF-8, why not?

On Windows this isn’t obvious at all, as the native encoding is UTF-16, and converting to/from UTF-8 on every Windows API call is not only a pessimization both in runtime and memory usage, but also introduces complications in having to handle possible conversion failures (involving unpaired surrogate characters, in particular).

C in principle allows to abstract over the system encoding.

What about compiling programs with `cl -utf-8` and using -A APIs? I know the docs say “If the ANSI code page is configured for UTF-8, -A APIs typically operate in UTF-8”, to which I say, just “typically”?! But in practice is it not sufficient?

Windows UTF-8 support is relatively recent and I have no experience with it, so I don’t know. However I expect Windows to just do the same conversions internally or in the linked runtime, that the program would otherwise have to do by itself. I’d assume that there will be edge cases that such programs then can’t handle, such as UI input and file paths containing unpaired surrogates.

The -A APIs are converted to calls to the -W APIs, IIRC, so you pay that cost everywhere. Not sure if that's universally true anymore, but it has been that way at least on the NT based kernel since...inception? Probably a good question for Raymond Chen. Sadly, I can't seem to find anything relevant in a quick search of the Old New Thing.

The GP is referencing the new UTF-8 APIs that reuse the -A suffixes.[0]

> Until recently, Windows has emphasized "Unicode" -W variants over -A APIs. However, recent releases have used the ANSI code page and -A APIs as a means to introduce UTF-8 support to apps. If the ANSI code page is configured for UTF-8, -A APIs typically operate in UTF-8. This model has the benefit of supporting existing code built with -A APIs without any code changes.

This requires Windows 10 version 1903 or newer. As the version number suggests, this feature is only four years old, and using them prevents your program from working on older versions.

[0]: https://learn.microsoft.com/en-us/windows/apps/design/global...

Yes, precisely. I have never tested it conclusively, but (taking it as a given that we are targeting >=1903) is it not sufficient and optimal to use `-utf-8` and -A APIs?

UTF-8 should never contain surrogates as they serve no purpose when all code points can be encoded directly. That doesn't prevent programmers doing stupid things with them but that's not a Unicode problem.

Correct – but that’s exactly why you can’t just say “all strings are UTF-8, we’re done here”. Because Windows filenames can contain unpaired surrogates, which don’t have a sensible UTF-8 translation. (Unix has a similar problem with filenames being arbitrary bytes that may not be valid UTF-8.)

Note that Rust has OsString to handle both of these cases, but it’s somewhat hard to use.

With networking code you sometimes need an OsString for the platform you're not on. Rust doesn't seem to support this, other than by writing your own.

> the native encoding is UTF-16

There's no reason all the rest of your software needs to pay for Microsoft's mistake.

> and converting to/from UTF-8 on every Windows API call is not only a pessimization both in runtime and memory usage, but also introduces complications in having to handle possible conversion failures (involving unpaired surrogate characters, in particular).

If the text isn't actually Unicode, then all you're demonstrating by not handling that in another language is that you didn't care about correctness. There's no magic here, unpaired surrogates aren't somehow actually valid Unicode so long as you stick with C.

> There's no reason all the rest of your software needs to pay for Microsoft's mistake.

How is it a mistake when UTF was not even a thing when the NT kernel was developed?

Not every mistake is something you could have prevented/ avoided. Sometimes you just get unlucky.

Windows APIs (including any COM and .NET libraries you may interoperate with) can serve you strings containing unpaired surrogates. That has nothing to do with that I “didn’t care about correctness”. Processing Windows UTF-16 as-is simply avoids complications that you would add by converting back and forth from and to UTF-8. If you’re working with other I/O (network requests, file contents) that is UTF-8-based, it makes much more sense to strategically pick the places where you perform conversions within your application. The conversion then is also limited to where data actually crosses between Windows APIs and other I/O within your application.

Traditionally, strings in C are encoding-agnostic. By default, the encoding of the current locale is assumed, and applications are expected to convert to other encodings when specifically needed. Forcing everything into “there can only be one encoding we can work with” isn’t a great solution. One can wish that everything was UTF-8 from the beginning, but that’s not the world we live in.

Rust has an abstraction for the system encoding: OsStr. Though I think most Rust programs just assert everything is UTF-8, the language goes to great lengths to ensure it is possible, and not even particularly difficult, to write Rust code that can handle Windows's broken UTF-16, yet which isn't Windows-specific.

Eh, the `OsStr` API has warts still, some parts are unstable (https://github.com/rust-lang/rust/issues/111544).

> These days one of the systems level programming languages (Rust) is natively UTF-8.

Except that the geniuses who maintain the extant Unix systems still couldn't bring themselves to at least start transitioning pathnames to utf8, so now we're forever stuck with the terrible OsString cruft and wasting brain and CPU cycles converting from and to OsString in some unsatisfactory manner.

At least ZFS (which also incidentally is pretty much the only non-terrible filesystem) has an utf8only flag.

the geniuses that maintain the extant Unix systems invented UTF-8 precisely because they value backward compatibility above everything else and won't contemplate breaking stuff for the sake of programmer convenience.

UTF-8 was invented by people who were sufficiently fed up with the shortcomings of Unix that they created a quite incompatible but much improved evolution of it. And it's not just some hack for the sake of backwards compatibility at the cost of other properties, it's a vastly superior design compared to the then existing unicode encodings.

And there is no serious backwards compatibility problem. You just introduce a mount option to enforce utf-8, flip it to default after a say a decade, and people who then still have file systems with pathnames in latin-1 (or other craziness) can then flip it off and get a few extra decades to migrate their stuff. In the meantime the rest of the world just writes software under the assumption that it will be deployed to non-crazy installs, and software becomes magically more reliable and quicker to write.

And it's not like even 1% of software now would handle non-utf-8 filenames robustly anyway (I mean even if you are aware of the problem and diligent about it, there is absolutely no good way to deal with non-utf8 filenames if you need to output them as text for consumption by humans or other programs, which is close to 100% of cases, because at the very least you will need to do so in error messages if there is an IO problem).

Then what happens when you mount it with the utf-8 option on one machine, and without it on another. The one without the utf-8 options writes a filename that breaks utf-8 encoding. You then try to mount it again with the utf-8 option. What happens then?

Not only mount. What happens when you unpack an archive that uses non-fully utf-8 compliant names. Or checkout the history of a git repo with non-utf-8 filenames. Network filesystems are also a source of pain.

And even if you settle for utf-8 you'll still have to deal with differences with filesystems automatically canonicalizing names or being case insensitive.

> What happens when you unpack an archive that uses non-fully utf-8 compliant names.

Exactly what you would want to happen (this is the actual error message you will get, BTW)?

   > (cd /mnt/funkyfilesystem && touch $'\370' && tar cf my-bad-file.tar $'\370')
   > tar xf /mnt/funkyfilesystem/my-bad-file.tar
   tar: \370: Cannot open: Invalid or incomplete multibyte or wide character
   tar: Exiting with failure status due to previous errors

Or are you trying to tell me that your life isn't complete without tar or a git checkout silently dropping some nonsense byte sequences into your filesystem? In both cases you can map the names to something saner explicitly and continue with whatever you were doing, or checkout to e.g. a tmpfs where you flipped off utf8-enforcement. BTW, unless you work with fairly unusual git repos or tar archives on a regular basis, this is very unlikely to ever happen to you.

> And even if you settle for utf-8 you'll still have to deal with differences with filesystems automatically canonicalizing names or being case insensitive.

(Case-preserving you mean? I think truly case-insensitive died with 8.3 DOS filenames). These are also annoying, but a much more minor issue, e.g. you don't need to create a new weird string type just because of them.

Your tar example appear to Work On My Machine. Probably I don't have a funky enough filesystem. Of course if the filesystem itself doesn't support invalid unicode names, there isn't much the os or tools can do.

But hey, if enough tools and applications stop supporting non-utf-8 names, it is possible that in a a few decades you might get what you want as de-facto non-utf8 files will just disappear.

I agree it's something that needs to be dealt with, but can you name a plausible scenario in which this would actually happen and in which the desired behavior would be that you get silently some garbage filename written to your disk?

Let's say you get an EILSEQ on accessing the dirent with readdir and now have to debug how you got some corrupted filename you clearly didn't want in the first place, remount and fix it. How is that worse than having the corrupted filename and not knowing about it before something more insidious happens?

Oh, and BTW: Plan-9, the OS utf-8 was specifically invented for did actually enforce utf-8 for filenames.

> UTF-8 was invented by people who were sufficiently fed up with the shortcomings of Unix that

Uh, not exactly, not quite. UTF-8 was invented by Unix people who were fed up with all the alternative proposals at the time. Those people did not then go and "fix Unix" in any way regarding this. Those people were none others than Ken Thompson and Rob Pike, who were among the creators of Unix -- you could only have gotten closer to "those people" being the creators of Unix by having had Denis Ritchie at that diner table on that fateful day.

> Except that the geniuses who maintain the extant Unix systems still couldn't bring themselves to at least start transitioning pathnames to utf8, [...]

As far as POSIX and Unix `open(2)` and friends go, pathnames are opaque binary with only two special byte values: NUL (0x00, because it's C strings, which are NUL-terminated) and '/' (because that's the path component separator). Any codeset and form that is compatible with that will "work" -- for some value of "work" where if the codeset isn't Unicode in UTF-8 then you'll be sad.

Nothing keeps users / sites from declaring that thou shall use UTF-8 on the filesystem, or, even better, that thou shall use UTF-8 locales only, as the latter is the only simple way to [mostly] get the former.

> At least ZFS (which also incidentally is pretty much the only non-terrible filesystem) has an utf8only flag.

Note that ZFS can't tell if the strings from user-land are in UTF-8. ZFS can only tell if they're not valid UTF-8.

There is a simple solution: use UTF-8 in the middle and push all codeset conversions to the edges. And, ideally: deprecate non-UTF-8 locales.

Falling back to emoticons is always an option ;-)

Back when I was stil single some people unmatched me on dating apps for using them. Great way to filter out people not worth your time, really

You could do that, or you could set your age minimum to 30 or so.

I'm sure it would have been worse with people in their tweens, but it's not exclusive to them weirdly enough.

Another good one is being completely honest about your height (mostly just filters out American expats but still).

Anyway, the only reason I used emoticons was because I use this keyboard:

http://www.exideas.com/ME/index.php

... but it apparently had some side-effects.

The inability to represent text in the vast majority of the world’s languages is a far bigger issue than the lack of emoji.

Reply on news.ycombinator.com