Hacker News Reader

The W mode was the only way to do Unicode until relatively recently, A was just legacy pre-Unicode encodings. So there's presumbably a fair chance for older Unicode apps to accidentally mess up the encoding, especially if it's really old stuff from back during the UCS2 era which didn't have paired wide characters (although I suspect at that time most stuff would use A functions in order to work on Windows 9x).

View on HN

If they were doing conversions manually, I suspect such a major encoding error would have shown up quite quickly. It's bound to cause errors when a file is unopenable in a lot of programs that can't handle non-unicode (heck, Microsoft's own VS code can't handle them).

I doubt even really old NT programs (pre 2000) using UCS-2 would have included unpaired surrogates because IIRC those code points were never mapped to actual characters so I don't think that's an issue in practice.

The problem is not people wilfully introducing lone surrogates but with incorrect string manipulations (e.g. slicing or transformations) not accounting for astrals and thus fucking up on surrogates.

Lone surrogates regularly show up in any environment which allows them.

Reply on news.ycombinator.com