Hacker News Reader

Comment by throw0101a | Hacker News Reader

> […] why limit yourself to 24-bit numbers?

For compatibility with UTF-16:

    o  Restricted the range of characters to 0000-10FFFF (the UTF-16
       accessible range).

* https://datatracker.ietf.org/doc/html/rfc3629#section-12

* https://en.wikipedia.org/wiki/UTF-16

The original spec had 31 bits (the UTF-32/UCS-4 range):

* https://datatracker.ietf.org/doc/html/rfc2279

* https://en.wikipedia.org/wiki/UTF-32

View on HN

We really ought to deprecate UTF-16 someday. The fact that it pretends to be a fixed-length encoding has caused all sorts of bugs over the years, with many people assuming n(UTF-16 codepoints) == n(characters) which breaks when the string contains non-BMP characters.

And also, for personal aesthetic reasons I hate that it limits the Unicode codepoint range to an awkward non-power-of-two number (now there are 0x110000 codepoints in total). UTF-8 and UTF-32's 2^31 feels much more natural.

I wonder what it would take. For things like Java, JSA, and c# there would have to be things like a parallel utf8 api (yes please), but the really hard parts is the things that are as old as time (windows).

I don't think it will ever happen, but one can dream.

Reply on news.ycombinator.com