Hacker News Reader

Comment by raphlinus | Hacker News Reader

I would go further and say that UCS-4 is also a variable length encoding, mostly because of emoji and its endearing ZWJ sequences, not to mention the encoding of flags as two Regional Indicator Symbol codepoints (both in the astral plane, of course), which makes Unicode not self-synchronizing.

So the idea of ever having a fixed-length encoding for Unicode is basically impossible now. Best to just use UTF-8 for everything and logic to group it in to code points, grapheme clusters, or whatever other granularity is needed.

View on HN

Those are all shaping (and bidi) issues though, independent from encoding. UCS-4 is a single Unicode code point and the simplest encoding. Shaping may combine code points many to one, one to many, have ordering issues, etc. I think it is important for people to understand how shaping works and the code point/glyph distinction. Trying to duplicate the effort of the shaper when processing strings UCS-4 is going to lead to a world of hurt. Even English text with strings like "1/4" and ligatures in the font may display as a single glyph (that's font specific shaping, not encoding).

I've heard more than one person tell me they don't need to worry about text shaping since they are using UTF-8. (That statement doesn't make any sense) There is a lot of confusion with Unicode text rendering stack.

Sure. I guess what I'm trying to say is: when do you care about code point boundaries at all? In my experience, it's actually fairly rare. If you're drawing text, you care about the result after shaping. If you're moving the cursor, you care about grapheme cluster boundaries. If you're doing stuff like sorting and collation, you should use specialized Unicode libraries that get this stuff right. When is the last time you wrote code that really needed to iterate Unicode code points specifically?

> When is the last time you wrote code that really needed to iterate Unicode code points specifically?

We do it all the time. Splitting a string at U+002C or U+003B or U+000A, looking for U+007B or U+007D, etc. In my experience, it's really common.

Right, and if you're doing processing of ASCII code points (this is 95% of Markdown parsing, for example, the major exception being case folding and Unicode whitespace), then it's just as convenient to do those in UTF-8 as any other representation.

Those aren't unicode code points; those are bytes.

Not on a UTF-16 platform.

Fun fact: because flags are a political nature, and Unicode did not want to limit the length of country codes, any number of Regional Indicator Symbol code points can be placed back to back and treated as an extended grapheme cluster.

That means that what Unicode today considers "one character" can be infinite in size.

And worse, the ‘operators’ are postfix or infix, so if you're handling text sequentially — and let's say, split across network packets with an arbitrary delay in between — you can't even tell you've read a complete character until you receive the next one.

Typing in many Latin languages generates diacritics using ‘dead keys’, which are prefix operators. (Originally, on a physical typewriter, these were keys that simply struck the paper without triggering the mechanism to advance the carriage.) If Unicode had taken this hint, life would be easier.

(ISO/IEC 6937 had prefix diacritics, but it was too late.)

Complete recognition before processing. It's not the law, just a good idea, but you should do it anyway. In cryptography this lesson had to keep being learned. "Oh, I received this partial data, and I processed it, and then in the next packet I received the MAC and I realised the bad guys altered the data, oops"

Process whole strings, the meaning of a partial string may not be what you hope, even without fancy writing systems and Unicode encoding.

"Give the money to Steph" - OK, will do

"...enson" - Crap, OK, somebody chase Steph and get our money back, meanwhile here's James Stephenson's money

"... once he gives you the key" - Aargh. Somebody chase down James and get the key off him.

Wow so interesting an idea. But so many streaming algorithms etc. people stick megabytes of crap into one framed message.

Sure, but you can also just stack infinite accent marks.

Either way there's already a recommendation in this area. I'm not sure if it strictly fits this particular scenario but the "UAX15-D3. Stream-Safe Text Format" suggests that you should support sequences at least 31 code points long.

To be fair, I believe this was true well before flags—diacritics played this role previously, as in "Zalgo Text".

What do you think a character is? ZWJ and combining characters are needed for a few Indian languages spoken by millions of people, more then speak Dutch or Norwegian.

Some low level text libraries (FriBiDi, HarfBuzz) use UTF-32 as it makes code simpler when you have to work on it at code point level. Most code does not need that of course and is better off working as a different level than code points.

> both in the astral plane, of course

Well never mind that then; how do you like U+41,U+300 aka U+C0 latin capital letter a with grave accent. Not just in the BMP but in the part of the BMP that's only 2 bytes of UTF-8.

And there's a whole bunch of similar characters that don't have NFC encodings.

Reply on news.ycombinator.com