Filenames with Accents (2011)
nedbatchelder.com
nedbatchelder.com
I’m guessing that, by now, most developers on Unix-like systems would be using UTF-8 for filenames – though a decade after these articles were published, there still doesn’t seem to be any good/universal solution to the problem of characters with multiple Unicode representations.
¹ https://dwheeler.com/essays/fixing-unix-linux-filenames.html
So if you have four possible normalizations: NFD, NFC, NFKD, NFKC and your string has N ambiguous codepoints, the number of possible strings you need to try is N^4.
Nowadays, there’s an understanding to assume those bytes are strings encoded in some ISO-8859 variant or UTF-8, but technically, the creat system call doesn’t receive strings; it receives byte arrays.
Historically, that was the (somewhat) right decision because it meant file systems didn’t need to know much about character encodings (they only needed to know the byte value of ‘/‘ and that zero is the name terminator), giving you a nice separation of concerns.
With Unicode, if you want to normalize names on write, or even only reject incorrectly normalized names, or have case-insensitive file names, your file system code needs to know a lot of Unicode. That can be problematic on small embedded systems.
I know two sort-of examples of this. Firstly, there’s MS-DOS long file names. That’s a hack (in the positive sense of the word) that gives code that doesn’t know about long file names the 8.3 file names that it expects.
It works, but code that isn’t aware of long file names will only write 8.3 ones, so even a simple file copy using an old copy tool will drop long file names.
Secondly, there’s macOS. It has a minor thing with directory separators. The Unix layer thinks ‘/‘ is the directory separator, old Mac code thinks it’s ‘:’. That works relatively well, mostly because old style Mac code doesn’t use file paths in the UI.
And of course, every desktop OS has to deal with it when it mounts various drives that handle file names differently. HFS+ used NFD normalization, most other file systems that normalize names use NFC, disks may be case preserving or not, case insensitive or not, normalization insensitive or not, etc.
On Unix-like OSes a path /a/b/c/d/e/f can walk six different file systems, each with different rules (even without soft or hard links). It wouldn’t surprise me to find bugs in programs there.
NFD was used by HFS+, but got abandandoned by the current insecure APFS. (which uses unidentifiable names). NFD is faster to produce, but NFC is complete and needs less space. with NFD you can still have reordered sequence variants, and thus nonidentifiable names.
and the NFKD, NFKC hacks should only be used with python internally, because they didn't understood Unicode. or just read the TR's without understanding it.
it will need several more decades until filesystems will find out about their wrong decisions. maybe I'll bug them with CVE's some day.
I know zsh handles auto-complete well but I can `y-a-w` out of my `nvim` `:terminal` a lot easier.
In Mac OS extensions are important but they are hidden by default. I always force them to be shown when I setup a new Mac. Hiding them obscures important information. Hiding them offers nearly no benefit.
Any tool that can't handle that is faulty, and I'll either stop using it or make a temporary symlink to get the job done. But most tools are fine with unicode these days. Of course I speak from the perspective of a personal computer situation with no deadlines or business requirements or software limitations.
I still use the old cmd.exe with the Terminal font. It doesn't even render most unicode properly, they come out as ?. I deal with it because I like cmd.exe and I like Terminal and I'd rather see ? than butcher the file's name by romanizing it.