Comment by vorg | Hacker News Reader

Comment by vorg | Hacker News Reader

Parent Full thread

> It basically defines a ginormous table of 1,114,112 code points that can be used for all sorts of letters and symbols. That's plenty to encode all existing, pre-historian and future characters mankind knows about. There's even an unofficial section for Klingon. Indeed, Unicode is big enough to allow for unofficial, private-use areas.

The private use areas only encode about 137,000 codepoints (U+e000 to U+f8ff & U+f0000 to U+10ffff) and are running out quickly. Most of U+e000 to U+f8ff is used by many different private agreements, and some pseudo-public ones like the Conscript registry which encodes Klingon, linked to in the article. Conscript also uses a large chunk of plane F to encode the constructed script Kinya, i.e. the 3696 codepoints in U+f0000 to U+f0e6f, see http://www.kreativekorp.com/ucsur/charts/PDF/UF0000.pdf . It takes up so much room because it's a block script like Korean Hangul and is encoded by formula just like Hangul. Each Korean Hangul block is made up of 2 or 3 jamo: one of 19 leading consonants, one of 21 vowels, and optionally one of 27 trailing consonants, giving a total of 19 * 21 * 28 = 11,172 possible syllable blocks, generated by formula into range U+ac00 to U+d7a3. Kinya also uses such a formula to generate its script, and I'm sure many other constructed block scripts will make their way into the quasi-official Conscript Registry. I'm even working on one of my own.

In fact, rather than filling up U+f0000 to U+10ffff, such conscripts only need to fill up the first quarter of it (i.e. U+f0000 to U+f7fff) for Unicode to run out of private use space, because the remainder (U+f8000 to U+10ffff) is needed for a second-tier surrogate system (see https://github.com/gavingroovygrover/utf88 ) to extend the codepoint space back up to 1 billionish codepoints as it was originally specified by Pike and Thompson until it was clipped back down to 1 million in 2003.

So Unicode is not "plenty to encode" or "big enough to allow for" all known, future, or private-use characters.

> see https://github.com/gavingroovygrover/utf88

This is the most stupid way to extend UTF-8 I've seen. The only acceptable solution is to remove the restriction of using only four byte per sequence which would allow to encode these easily keeping all the advantages of UTF-8.

Doing it like they does add an additional layer of encoding and so a lot of complexity a room for bugs.

It was probably made for compatibility but a lot of software will do bad thing with these new "surrogate" pairs so this solution is not really more compatible in practice. And updating software to handle UTF-8 sequence longer than 4 bytes is a lot more easier than updating them to handle such encoding.

> The only acceptable solution is to remove the restriction of using only four byte per sequence which would allow to encode these easily keeping all the advantages of UTF-8

I agree. Extending UTF-8 with surrogates like this is intended to be temporary, only used until the pre-2003 2.1 billion codepoint limit for UTF-8 and UTF-32 is reinstated by the Unicode Consortium. Then any software using UTF-88 can easily swap the encoding to the 1 to 6-byte sequences in "reinstated" UTF-8. This surrogation scheme is actually intended for UTF-16 to use as a second-tier surrogate scheme so it can encode the same number of codepoints as UTF-8 and UTF-32. I wrote all this under "Rationale" at the bottom of the linked page, did you read that far?

Hopefully, though, UTF-16 will be on its way out when pre-2003 UTF-8 and UTF-32 are reinstated so this surrogation scheme wouldn't even see much use there.

But "temporary" is a thing who exists only in theory. In practice its always never or (almost) forever. As soon as a few applications start using this "new" form of UTF-8, some of them may have to keep supporting it forever.

Why not directly going for the pre-2003 UTF-8 encoding ? It would even put a bit of pressure for restoring them and would show that this is the good way. It is also the only way I think to convince people to start implementing it.

> As soon as a few applications start using this "new" form of UTF-8, some of them may have to keep supporting it forever

Not if it's used through a 3rd-party library such as the Go-implementation of UTF-88 I've provided.

> Why not directly going for the pre-2003 UTF-8 encoding ? It would even put a bit of pressure for restoring them

Because it's not a valid encoding under the current scheme, whereas using surrogates with UTF-8 is, using as it does the 2 private use planes to implement the surrogates. The goal is for restoration by the Unicode Consortium, but based on their public utterances it's not going to happen easily or quickly, and in the meantime we need an encoding that's valid under the current scheme because it may need to be used for 10 or 20 years. Of course I could have used UTF-16 with a doubly-directed surrogate system but that would be even more error-prone, and I expect whatever 2nd-level surrogate system is eventually provided with UTF-16 will be legally available with UTF-8 and UTF-16 anyway.

UTF-88 is an attempt to showcase both a surrogation scheme implementable in current UTF-16 and the fact that UTF-8 is the best encoding.

Reply on news.ycombinator.com