Hacker News Reader

Comment by inferiorhuman | Hacker News Reader

Ideally… yes. Consider that Windows APIs use UTF-16, not UTF-8, for wide characters. Microsoft's extensions to ISO 9660 used (big endian!) UCS-2 (a.k.a. UTF-16) for long filenames, NTFS uses proper UTF-16 I believe.

View on HN

I think NTFS is UCS-2, that is why WTF-8 was invented

https://simonsapin.github.io/wtf-8/

Surrogate pairs are interpreted by client software, so it’s UTF-16 in that sense. The file system just doesn’t ensure that there won’t be unpaired surrogates, or other noncharacters. This is similar to strings in .NET, Java, and JavaScript.

Which unfortunately means you can't rely on it being UTF-16.

Nor should you. Even a well-formed sequence of utf-16 codepoints can be utter nonsense; there's approximately no level of abstraction between "sequence of fixed-width code units" and "run it through a full-blown a font rendering stack" where it makes sense to assume your input is "well-formed".

You are right: NTFS exclusively runs on UCS-2, now "UTF-16 without validation". Back in the 1980s "Unicode" referred to a way to represent the ISO standard "Universal Character Set", i.e. UCS-2, instead of being synonymous to the entire character set like today. The wisdom of i18n back then was to use "Unicode" text encoding because it covered the known universe of writing. Qt made a similar choice.

Actually, you can manifest your windows app as UTF8 aware and all the win32 "-A" (formerly ascii only) APIs switch to handling UTF8 :3

This is a feature of Windows 10+ https://learn.microsoft.com/en-us/windows/apps/design/global...

Or notepad++ could just use the WinAPI (WideCharToMultiByte) to convert UTF-16 to UTF-8 instead of hand writing their own conversion routine.

Reply on news.ycombinator.com