Hacker News Reader

Comment by magicalhippo | Hacker News Reader

How does that affect filenames?

IIRC the lower levels of Windows will happily work with filenames that are not valid Unicode strings, for example if you use the kernel API rather than Win32.

But what about Win32? If you create a file before normalization and then open it using the normalized form, will it open the same file or return file not found?

What about other systems? For example AWS' S3 allows UTF-8 keys, with no mention of normalization[1].

On the phone so can't try myself right now.

Anyway for general text I agree, but for identifiers, filenames and such I prefer to treat them as opaquely as possible.

[1]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/object...

View on HN

No need to go to the kernel API to create filenames that are invalid UTF-16. The Win32 API will happily let you do it.

I was AFK so couldn't check, but yeah you're right.

Just made two files named ä.txt which happily sat next to each other, one being NFC and other NFD.

So yeah, don't mess with the normalization of filenames.

They're both valid UTF-16, though. Can you create a filename with only half of a surrogate pair in it?

I don't use Windows, so I can't check. Linux literally allows any arbitrary byte except for 0x00 and 0x2F ('/' in ASCII/UTF-8). It's a problem for programming languages that want to only use valid Unicode strings, like Python. Rust has a separate type "OsString" to handle that, with either lossy conversion to "String" or a conversion method that can fail. Python uses the custom use Unicode range to represent invalid byte sequences in filenames. It's all a mess. JavaScript doesn't give a damn about the validity of their UTF-16 strings.

(Note that Rust's OsString is different from it's CString type. Well, I guess under Unix they're the same, but under Windows OsString is UTF-16 (or "WTF-16", because it isn't actually valid UTF-16 in all cases).)

Yeah seems to be same with Win32.

I tried using U+13161 EGYPTIAN HIEROGLYPH G029[1], which resulted in a string of length 2 as expected.

Using both chars (code units) and just the first char (code unit) worked equally fine. In Windows Explorer the first one shows the stork as expected, while the second shows that "invalid character" rectangle.

So yeah, treating filenames as nearly-opaque byte sequences is probably the best approach.

[1]: https://en.wiktionary.org/wiki/%F0%93%85%A1

But ä.txt is a perfectly normal filename! WTF is there to "normalize" about that?!?

Äh, nu går vi och gör något annat.

Either ä as a single code unit or a + ¨ as two code units.

Win32/NTFS treats them as two different filenames, as it doesn't normalize them before storing/comparing.

> What about other systems?

The Linux kernel doesn't validate filenames in any way, so a filename in Linux can contain any byte except 0x2F ('/', which is interpreted as directory separator) and 0x00 (which signals the end of the byte string).

ETA: of course some file systems have other limitations, for example '\' is not valid in FAT32.

Reply on news.ycombinator.com