Hacker News Reader

Comment by estebank | Hacker News Reader

> if you have been handling Unicode and using wide characters, you have not been handling Unicode properly.

Paradoxically, trying to do "the right thing" and being an "early adopter" of (the now called) UCS-2 was a "mistake", as both Java and Windows can attest, by getting "stuck" supporting the worst possible Unicode encoding ad-infinitum. UTF-8 is the "obviously correct" choice (from the hindsight afforded by us talking about this in 2021).

I still find it funny that emojis of all things are what actually got the anglosphere to actually write software that isn't completely broken for the other 5.5 billion people out there.

View on HN

Correction to the date and timeline. UTF-8 has been the correct choice since:

" >From ken Tue Sep 8 03:22:07 EDT 1992 "

As discussed last week in the quoted history https://news.ycombinator.com/item?id=26735958 of UTF-8's mail messages: https://doc.cat-v.org/bell_labs/utf-8_history

It appears that Unicode arose between 1990 and 1991 for the initial versions https://en.wikipedia.org/wiki/Universal_Coded_Character_Set#... with the first published version of the specification in 1993:

"ISO/IEC 10646-1:1993 = Unicode 1.1"

As discussed another time on Hacker News https://news.ycombinator.com/item?id=20600195 https://unascribed.com/b/2019-08-02-the-tragedy-of-ucs2.html

It was around 1996 when it became clearly obvious (software got shipped to end users who cared and they complained back) that UCS-2 (16 bit characters) would be insufficient.

+++

Pragmatically this is forever, as long as backwards compatibility must be maintained, the exiting APIs which are built around the crazy 16 bit standard need to exist; but there's little reason they have to be native, rather than wrappers for UTF-8 compatible APIs.

It would even be a good time to standardize on a single user space programming API and have implementations on every operating system. Preferably including basic drawing and font layout functions. So that finally, most programs could be written once, compiled on a platform of choice, and work.

> actually write software that isn't completely broken for the other 5.5 billion people out there

I thought that Chinese and Japanese are the only languages that UCS-2 has trouble fully representing. I believe all the other living languages can actually be represented by UCS-2.

So using UCS-2 would actually work for almost everyone except maybe 1.5 billion people.

Do we not consider emoji a language unto itself?

I was actually quite bemused to discover that some code review software I was using allowed me to "cursor" halfway through a smiley face emoji and enter a space (typing too fast to pay attention)... causing the infamous "box characters" because I'd accidentally split the smiley down the middle.

I get the need for extreme backward compat in browsers, but... this seems like one of those things that just might be worth fixing. Maybe a "use utf8" directive? :)

Even with proper UTF-8, you'll get situations where you can insert a space in the middle of an emoji and split it into other emoji + unprintable characters. The encoding is irrelevant, you need proper Unicode support to avoid these problems.

Unfortunately proper unicode support for many operations means carrying around tons of data since important properties of code points cannot be derived from the codepoints themselves.

Exactly. UTF-8 doesn't and cant fix this problem, you need a full Unicode library if you want to correctly handle human text. If you don't, why bother with UTF-8 instead of something simpler, like ASCII?

> If you don't, why bother with UTF-8 instead of something simpler, like ASCII?

Because unicode support isn't binary. Being able to pass along and not mange blobs of unicode is already a lot better than ASCII-only.

Code that treats UCS-2 as "wide ASCII" can be subtly wrong even for western european languages. For multiple reasons, Unicode has multiple representations for the same glyph[1], so if you have ü, it can either be a the two bytes 00FC[2], or u followed by the ¨ diacritic U+0308. If you don't account for this things like "reversing a string" or "give me a substring" or "how long is the string on screen" will be subtly buggy. If you handle UCS-2 correctly, then it's fine and a reasonable technical limitation, but the emphasis of that sentence is on correctly.

[1]: https://en.wikipedia.org/wiki/Unicode_equivalence

[2]: https://www.compart.com/en/unicode/U+00FC

> Code that treats UCS-2 as "wide ASCII" can be subtly wrong even for western european languages.

It can be. But critically for this conversation, fixing your code to support emoji and other non-BMP characters doesn't necessarily fix those problems.

Except it does since a single emoji can also be made up of multiple code points.

But that only causes subtle wrongness, which can continue to go unnoticed.

> So using UCS-2 would actually work for almost everyone except maybe 1.5 billion people.

One of the great things about using Rust is that I don't have to have this argument. There doesn't have to be a debate about whether we should invest in fixing subtly broken code. Rust string-handling which generally works for Europeans will also work for Chinese!

This isn't the fault of the languages which were designed for UCS-2 (then known just as "Unicode"). But the fact that Rust emerged after UTF-8's ascendance means that Rust's users mostly get to avoid the UCS-2/UTF-16 legacy tarpit.

> Rust string-handling which generally works for Europeans will also work for Chinese!

I doubt that is any more true than Java. You can easily write code in Rust that assumes you can split a String anywhere and get two valid strings, that you can compute the length of a String and get information about how long the printed representation will be, that you can find a substring in that string by simply iterating through UTF-8 code points etc. All of these assumptions are about as wrong in UTF-8 Rust as they are in UTF-16 Java.

Java hasn't been UCS-2 since Java 5.0 - nearly a decade.

Well if memory cost wasn't an issue then UCS-4 would be nicest. But 4x the memory for most strings is currently unacceptable.

There are a lot of instances where I wouldn't mind the memory cost, but would very much mind not automatically being compatible with ascii strings.

That is true, but the benefits of UTF-32 is minor compared to UTF-8, and might not be worth the cost.

And I say that as someone who is developing a language which only has UTF-32 support.

The problem is that even with UTF-32, doing things like splitting strings is inherently unsafe, so you are still going to need a Unicode library to do proper splitting by grapheme cluster. In practice, almost all string splitting works on ASCII text, and assumes everything else is data that should not be manipulated. For this, UTF-8 is perfectly acceptable.

UCS-4 is still a variable-length encoding, because various accented characters and emoji use multiple code points. One advantage of UTF-8 is that it makes you confront variable-length characters head on.

Reply on news.ycombinator.com