Hacker News Reader

Comment by feldrim | Hacker News Reader

> On Windows, C and C++ is so poorly hooked up to operating system interfaces that most portable or mostly-portable software — programs which work perfectly elsewhere — are subtly broken on Windows.

I believe that the problem is about the definition of "elsewhere". The elsewhere is called POSIX. The portability is based on POSIX. So the article can be summed as "Windows is not POSIX compliant".

View on HN

> The UTF-8 encoding was invented in 1992 and standardized by January 1993. UTF-8 was adopted by the unix world over the following years due to its backwards-compatibility

I'd like to add some historical background the UTF-8 problem mentioned there. UTF-8 was standardized in 1992. But the first Linux using UTF-8 as the default encoding was RedHat in 2002 [1]. 10 years later!

On the other hand, Windows started using Unicode before it was standardized, and back then there was no UTF-8. Or as Raymond Chen put it, "Windows adopted Unicode before the C language did" [2].

[1] https://www.cl.cam.ac.uk/~mgk25/unicode.html#linux

[2] https://devblogs.microsoft.com/oldnewthing/20190830-00/?p=10...

*nix got lucky by always treating filenames mostly as bag of bytes (except for / and NUL), which delegated encoding to applications and typical passthrough from argv to fopen worked fine. Arguably WinNT way is more ambitious and reliable, but I think ignoring UTF8 existence until 2019 was huge mistake.

But the first Linux using UTF-8 as the default encoding was RedHat in 2002

And then Microsoft spent the next 20 years explaining/complaining to everyone (and this includes Chen) why they couldn’t make UTF-8 a code page because it would mean revalidating all of those -A functions for 4-byte characters when they were originally intended for only 3-bytes?

There was a time window of a few months between the invention of UTF-8 and the release of Windows NT with UNICODE support. If the two involved parties would have known of each other (and assuming the Windows devs would have understood what a great deal UTF-8 was), I guess Windows could have used UTF-8 right from the start.

>If the two involved parties would have known of each other [...], I guess Windows could have used UTF-8 right from the start.

Probably not. The author of this thread's article has incorrect history about UTF-8 being widely accepted in 1993. At least 4 different systems independently chose UCS2/UCS4 instead of UTF-8 in the early 1990s:

- Microsoft Windows NT 3.1

- Sun Java in 1995

- Netscape Javascript in 1995

- Python 2.x

Why? Because before 1996, the Unicode Consortium initially thought 16-bit 65k characters was "more than enough" based on the CJK unification first recommended by language experts from Asia. The wikipedia page on Unicode revision history shows the explosion of extra CJK chars in Unicode 3.1 which exceeded count of 65k didn't happen until 2001: https://en.wikipedia.org/wiki/Unicode#Versions

The prevailing standard in 1993 was UCS-2 and not UTF-8. Considering that a huge revision of an operating system like Windows is a multi-year effort, they were really looking at what the standard was circa ~1991. So Rob Pike's napkin idea of UTF-8 in 1992 and then the later presentation at USENIX in January 1993 -- is really not relevant for a July 1993 release of Windows NT 3.1. The author writing in Dec 2021 has hindsight bias but MS/Java/Python just went with Unicode 1.x UCS2 which was logical at that time.

Even by 1999, when Qt 2 was released, the new "Unicode-aware" QString class used UTF-16 internally, not UTF-8. This remains the internal Qt encoding even today (see https://wiki.qt.io/QString#Unicode_methods_in_QString), although the support for UTF-8 has improved substantially, to the point that Qt assumes that 8-bit strings not provided an alternate encoding are UTF-8, that 8-bit filenames can be expressed as UTF-8, and so on (see https://doc.qt.io/qt-6/unicode.html).

In 1991, the decision was way beyond made: https://betawiki.net/wiki/Windows_NT_3.1_October_1991_build

There're several mistakes in the history. Just like any article, it is based on the biases of the author.

Ken Thompson was the one scribbling on a napkin. Rob was the one telling the story later on.

The addition of Unicode support to Windows NT was probably done a great deal in advance of its actual release--it's not the sort of thing you can throw in the last minute.

It's also worth noting that dealing with UTF-8 correctly is more than just declaring "the encoding of this set of characters is UTF-8"--it does require you rethink how string APIs are designed to work well. If the alternative is a 16-bit fixed-with character format, UTF-8's variable-width format doesn't necessarily look like a wiser idea. It's not until 1996, when Unicode moves from 16-bits to 25-bits, that UTF-8 actually looks like a good thing.

They did build a POSIX interface, mostly so they could mislead the DOD into buying NT to meet POSIX needs. https://en.wikipedia.org/wiki/Microsoft_POSIX_subsystem

They didn't mislead. I don't think the DOD mandated full POSIX compliance, and they really only wanted POSIX-style fine-grained ACLs.

Yes. It did. It's just another subsystem over the kernel. Kernel did not care about POSIX. The POSIX subsystem is yet another abstraction layer, not too different from CygWin on MinGW for the use case. Since it was limited, it would not solve the portability problem that the author mentioned.

It also means mixing windows and posix code was impossible. If you chose posix, you were locked in an extremely limited text-only world with no good path to any extras windows could offer.

This architecture made it possible to claim posix compatibility on paper, while making it clear microsoft would make your life hell if you tried to actually use it

Except you're missing the part that it was later replaced with SFU.

https://en.m.wikipedia.org/wiki/Windows_Services_for_UNIX

However most people didn't care enough to use them.

Having WSL now in the box to do what many of us have been doing with VMware, is more an attack on Apple to cater to those that do not pay Linux OEMs to develop for Linux than anything else out of their good hearts.

Reply on news.ycombinator.com