Hacker News Reader

Comment by ChrisSD | Hacker News Reader

Windows is actually quite a bit better. The APIs require you to use an encoding. The multibyte `A` APIs can be whatever encoding the current process is using and will be automatically converted to valid UTF-16. The `W` APIs are for UTF-16. The issue is the latter does not enforce valid UTF-16.

So in practice the only way invalid UTF-16 shows up is from malicious programs (or people testing their handling of non-unicode).

View on HN

The W mode was the only way to do Unicode until relatively recently, A was just legacy pre-Unicode encodings. So there's presumbably a fair chance for older Unicode apps to accidentally mess up the encoding, especially if it's really old stuff from back during the UCS2 era which didn't have paired wide characters (although I suspect at that time most stuff would use A functions in order to work on Windows 9x).

If they were doing conversions manually, I suspect such a major encoding error would have shown up quite quickly. It's bound to cause errors when a file is unopenable in a lot of programs that can't handle non-unicode (heck, Microsoft's own VS code can't handle them).

I doubt even really old NT programs (pre 2000) using UCS-2 would have included unpaired surrogates because IIRC those code points were never mapped to actual characters so I don't think that's an issue in practice.

The problem is not people wilfully introducing lone surrogates but with incorrect string manipulations (e.g. slicing or transformations) not accounting for astrals and thus fucking up on surrogates.

Lone surrogates regularly show up in any environment which allows them.

Reply on news.ycombinator.com