Hacker News Reader

Comment by INTPenis | Hacker News Reader

As a developer who has had to integrate data between Windows and Linux systems, but lacks deeper insight into Windows, I'd say another really important milestone would be a complete switch from CP1252 to Unicode.

Countless times that codepage thing has haunted me when moving data from Windows workstations into automatic processing systems hosted on Linux.

View on HN

As of Windows 10 1903 you can set the codepage of the application to UTF-8 - in fact for UWP Microsoft recommends you do that (https://docs.microsoft.com/en-us/windows/uwp/design/globaliz...).

Unfortunately I don't see that being common enough for data ingestion tools for this to be relevant for a very, very long time.

For now, it's just another pain point because you can't assume Windows == UTF-16 lookalike anymore.

Is it now recommended rather than experimental? I expect some legacy apps that uses local OEM codepage would be broken.

Windows is UTF-16 internally while Linux is UTF-8.

If Windows goes to unicode by default, it will make things a lot worse because it won't be the same unicode.

Well, they say UTF-16, but they still allow unpaired surrogates in some places, so it is really more like UCS-2.

Linux also allows invalid utf-8.

It is, Windows NT's adoption of Unicode predates UTF-16 and it's impossible to really make it UTF-16 compliant without breaking obscure corners of compatibility.

Most programs assume it's all UTF-16, and it happens to work most of the time... at least until you encounter those sequences that are invalid.

WTF-16 makes the distinction that invalid sequences are allowed.

* http://simonsapin.github.io/wtf-8/

You can switch 1-byte encoding to UTF-8. But many applications probably won't work. And working applications probably are using unicode API anyway.

It was explicit for a long time that ANSI is a legacy feature, and new API functions are often added without ANSI version, like CreateFile2.

Reply on news.ycombinator.com