The secret life of Unicode (2001)

Unicode is limited to a little over 20 bits, a bit over a million characters. This limitation is because of UTF-16, which I hold to be the worst thing that ever happened to Unicode. Had it not been for UTF-16, the limit would be 32 bits, four billion characters.

As it stands, Unicode is about 25.5% full¹, up from 22.4% a decade ago² and 21% two decades ago³. (These figures are the designated counts; 12.5% of the total space was basically reserved from the start (Unicode 2.0), in control characters, private use areas, surrogates (because UTF-16) and noncharacters.) And by this point, we’re running out of real things to add. (You run out of languages that you want to represent, which leaves you with just the occasional emoji or other symbol that you want to add, and in the last decade emoji have accounted for about 1% of the additions.) Under current policies of what to include, it should easily be good for at least another five thousand years, and I don’t think it’s worth trying to forecast that far in advance.

—⁂—

¹ https://www.unicode.org/versions/stats/charcountv14_0.html

² https://www.unicode.org/versions/stats/charcountv6_1.html

³ https://www.unicode.org/versions/stats/charcountv3_2.html

Reading https://en.wikipedia.org/wiki/UTF-8#Encoding, when we get to 8-byte long UTF-8 code points, the first byte would have to be all ones (11111111)

Those would have room for 7 bytes with 6 bits of data each, or 42 bits.

Of course, if the need arises (which won’t be soon) we could choose to add another byte to encode the length of the actual byte stream.

(Currently, even going beyond four bytes already would break code, as Unicode explicitly limited the UTF-8 encoding in https://datatracker.ietf.org/doc/html/rfc3629 to allow round-tripping to UTF-16. I also think the above goes beyond what was proposed historically (https://en.wikipedia.org/wiki/UTF-8#History))