Hacker News Reader

Comment by ezoe | Hacker News Reader

I wonder how long Windows can hold UTF-16 as the default native encodings. They recently added UTF-8 locale.

CP65001 has existed for quite a while. There just have been plenty of places where it never really worked (I think the console was one of them).

Nonetheless, UTF-16 will most likely stay. They might add wrappers around all API functions that convert arguments first, so we'd get CreateWindow8 alongside CreateWindowW and CreateWindowA, but I wouldn't hold my breath. The duality of API functions was to enable programs to be built for NT and 9x with the same codebase. It was never really intended for people to call them directly to enable non-Unicode-aware applications on NT.

The file system cannot migrate from UTF-16 anyway without breaking things, so that will stay as is.

IMHO there's pretty much nothing to gain, except a lot of risk in trying to change Windows to UTF-8 internally and the current interface might be an inconvenience to some developers, but not as much as to justify changing the whole OS.

It's even harder than that - APIs have expected performance characteristics, and APIs that involve string manipulation can change their performance when needing to work directly with UTF-8, e.g. working with file paths might become slower.

When the world uses UTF-8 and you have to constantly recode from/to with every WinAPI call, that's definitely an unnecessary overhead.

Obviously it is unnecessary but I doubt it affects the perceived speed of the average desktop application in any way.

To be clear, the Windows filesystem isn't even UTF-16. It's UCS-2, which is simply a sequence of 16 bit values. The difference is that not all sequences are valid UTF-16 (see surrogate pairs).

Thus you have to be very careful when handling filesystem paths.

The problem is, the rest of the world decided to settle on UTF-8. We have some normalization disagreements but it's UTF-8 anyway.

Now we have surrogate pairs and the IVS, there is no advantage of using UTF-16 at all.

I will not surprise the Microsoft someday convert everything including kernel to filesystem to UTF-8 someday... or more likely they lost the market share.

Reply on news.ycombinator.com