Hacker News Reader

Comment by masklinn | Hacker News Reader

Windows paths are sequences of utf-16 code units, so they can contain unpaired surrogates which are not legal in unicode (and therefore UTF-8).

That’s why on windows Rust’s OsStr is an extension of UTF8 called WTF8, which allows surrogates (then again “traditional” unix paths are just bags of bytes so it’s not like windows is any worse).

View on HN

NTFS is old, so I think they're actually just sequences of 16-bit values, once upon a time these were presumed to be UCS-2 characters, now they're presumed to be UTF-16 code units, but either way just as Unix filenames are actually just [u8] the Windows filenames are just [u16]

Bags of bytes is at least the honest, "we know that none of you know jack shit about encodings and if we gave you nice APIs some Go developer will ignore them to avoid dynamic linking so the only safe thing to do is poke file paths as opaque blobs and let people optimistically decode them.

MS's approach is better in the case where we're all good stewards of the filesystem but the edges are sharp when someone isn't.

Windows is actually quite a bit better. The APIs require you to use an encoding. The multibyte `A` APIs can be whatever encoding the current process is using and will be automatically converted to valid UTF-16. The `W` APIs are for UTF-16. The issue is the latter does not enforce valid UTF-16.

So in practice the only way invalid UTF-16 shows up is from malicious programs (or people testing their handling of non-unicode).

The W mode was the only way to do Unicode until relatively recently, A was just legacy pre-Unicode encodings. So there's presumbably a fair chance for older Unicode apps to accidentally mess up the encoding, especially if it's really old stuff from back during the UCS2 era which didn't have paired wide characters (although I suspect at that time most stuff would use A functions in order to work on Windows 9x).

If they were doing conversions manually, I suspect such a major encoding error would have shown up quite quickly. It's bound to cause errors when a file is unopenable in a lot of programs that can't handle non-unicode (heck, Microsoft's own VS code can't handle them).

I doubt even really old NT programs (pre 2000) using UCS-2 would have included unpaired surrogates because IIRC those code points were never mapped to actual characters so I don't think that's an issue in practice.

The problem is not people wilfully introducing lone surrogates but with incorrect string manipulations (e.g. slicing or transformations) not accounting for astrals and thus fucking up on surrogates.

Lone surrogates regularly show up in any environment which allows them.

Reply on news.ycombinator.com