Hacker News Reader

Comment by hlandau | Hacker News Reader

The Unicode space used to be 31-bit. UTF-8 defined 5 and 6-byte encoding sequences for this reason. When UCS-2 had to be extended to create UTF-16, the solution chosen effectively limited things to a 21-bit space. Unicode was then redefined to be 21-bit, and the 5 and 6-byte encodings of UTF-8 were removed from the standard, though old systems might still accept them.

It is worth noting that if we ever run out of space in 21-bit Unicode, one could conceivably re-extend it back to the original 31 bits by throwing UTF-16 under the bus. The only reason Unicode is limited to 21 bits is because of UTF-16.

View on HN

By throwing UTF-16 under the bus, or by extending it, in the same way that UCS-2 was created by extending UTF-16.

Just like a number of UCS-2-representable codepoints in the basic plane were given up, in order to allow that number squared of non-UCS-2-representable codepoints to be representable in UTF-16, a number of UTF-16-representable codepoints in a supplemental plane could be given up, in order to allow N^2 non-UTF-16-representable codepoints in not-yet-existing supplemental planes to be representable in UTF-16-but-worse.

Yes, I'm talking about putting surrogates in the surrogates.

There's also no reason to assume that if you are throwing out UTF-16 that you at that point won't also also be planning to extend past 31-bytes and out into the next plane (out into 63-bits).

Assuming UTF-32 is UCS-4 and will never address future planes is how Unicode got into the situation that UTF-16 isn't infinitely expandable and is stuck at the 21-bit boundary.

Look, we’ve currently got about a million code points, and a trajectory and principles that should leave that easily enough for at least five thousand years, by my simplistic estimate <https://news.ycombinator.com/item?id=32286193>. There’s no point in forecasting that far in advance.

And you’re concerning yourself with filling a space of two billion code points? A Unicode that would be in any danger of doing that would not be in the slightest bit recognisable as the Unicode of today.

(Yeah, I know they once thought they could squeeze Unicode into 65,536 code points. I honestly don’t understand how they ever thought that would be sufficient even with Han unification, and I wish they hadn’t made that misstep that gave us the abomination that is UTF-16. But they knew running out was a hazard from the start, they just miscalculated the risk. No one that examines the matter is concerned about even the slightest risk of the current million code points being insufficient in the usefully-predictable future.)

But JS is UTF-16 native. Would it even be feasible to migrate?

"UTF-16 native" doesn't mean your "UTF-16 unit" (i.e. what would `charCodeAt` return) should not exceed 16 bits. JS implementations already do not use UTF-16 as a sole native representation anyway, so the migration should be relatively easy.

Reply on news.ycombinator.com