Ask HN: What are your Unicode woes? | Hacker News Reader

Ask HN: What are your Unicode woes?

5 pointsRendello11 comments

I've always worked with text, but I only started digging deep into understanding Unicode this year.

What do HN people have to say about Unicode and UTF-{8,16,32}? Are there parts you've never really understood? Have you had unexpected bugs due to misunderstood properties of text?

View on HN

I (OP) have been working on some Unicode visualization tooling for a while now. The idea started when I had some buggy string-matching code. I was matching case-insensitively, then using those ranges to highlight the original text.

Turns out, sometimes changing case changes not only the number of bytes (in UTF-8), but the number of encoded characters! This led to my post "UTF-8 characters that behave oddly when the case is changed" [1], which inspired a lot of conversation that taught me a lot. After that, I started reading Unicode documentation in earnest, and building up an idea of what a new tool should show. I'm trying to make clear things I didn't (and sometimes still don't) understand, so I'd love to know what causes pains in the wild / gaps in people's understanding.

1. https://news.ycombinator.com/item?id=42014045

I guess its kind of annoying that letters with diacritics can be represented in multiple different ways

That's true, and even with normalization, there's four normalized forms for strings. The -k- forms are mostly for searching, but that still leaves NFC and NFD.

The normalization forms are explained, in order of approachability (imo), in this random Youtube video, the Unicode Annex #15, and the Unicode Core Spec:

https://www.youtube.com/watch?v=ttLD4DiMpiQ

https://unicode.org/reports/tr15/

https://www.unicode.org/versions/Unicode16.0.0/core-spec/cha...

Comparing strings by bytecode equality is kinda dubious anyway.

String comparison is a difficult problem. Consider:

Å (ANGSTROM SIGN)

Å (LATIN CAPITAL LETTER A WITH RING ABOVE)

Å (LATIN CAPITAL LETTER A) + (◌̊ COMBINING RING ABOVE)

А̊ (CYRILLIC CAPITAL LETTER A) + (◌̊ COMBINING RING ABOVE)

Of these, the Angstrom Sign is considered deprecated and won't show up in any normal forms. The second is the NFC (composed) form, and the third is the NFD (decomposed) form. The Cyrillic one looks the same, but is not the same abstract character, so isn't connected in any normalization form.

Normal forms also reorder the diacritics if there are multiple. The strings could be compared through their normalized encoded forms (like UTF-8), which I think is what you meant, or their normalized code points directly. I agree it can be messy, but I'm curious what you meant by dubious, do you think there's a better way?

I don't understand the difference between a character, a codepoint, a glyph, and whatever else makes up a single "thing" in unicode.

That tripped me up too. The Unicode Core spec is quite good at explaining things and introduces some terminology you don't really hear outside the document. Chapter 2, General Structure, is worth reading in its entirety. I've linked some bits that might help:

> *2.2.3 Characters, Not Glyphs*

> The Unicode Standard draws a distinction between characters and glyphs. Characters are the abstract representations of the smallest components of written language that have semantic value. They represent primarily, but not exclusively, the letters, punctuation, and other signs that constitute natural language text and technical notation. [...] Letters in different scripts, even when they correspond either semantically or graphically, are represented in Unicode by distinct characters.

> Characters are represented by code points that reside only in a memory representation, as strings in memory, on disk, or in data transmission. The Unicode Standard deals only with character codes.

> *2.4 Code Points and Characters*

> The range of integers used to code the abstract characters is called the codespace. A particular integer in this set is called a code point. When an abstract character is mapped or assigned to a particular code point in the codespace, it is then referred to as an encoded character.

> *2.5 Encoding Forms*

This deals with UTF-{8,16,32}, which is a tricky bit and tripped me up for a long time. If the document is too dense here, there's a lot of supplementary material online explaining the different forms, I'll link a Tom Scott video explaining UTF-8.

---

The long and short of it is: the atomic unit of Unicode is the character, or encoded character, which is a value that has been associated with a code point, which is an integer usually represented in hex for as U+XXXX. Unicode doesn't deal with glyphs or graphical representations, just characters and their properties (eg. what is the character name? what should this character do when uppercased?). As you probably know, many characters can combine with others to form grapheme clusters, which may look like a single (abstract) character, but underneath consist of multiple (encoded) characters. Every character is associated with an integer index (a codepoint), and those integers can be represented in three formats (this sort of happened by accident): UTF-32 (just represent the integer directly), UTF-16 (was originally supposed to represent the integer directly, but there were too many and it got extended), and UTF-8 (which has different byte lengths to encode different characters efficiently).

[spec] https://www.unicode.org/versions/Unicode16.0.0/core-spec/

[2.2.3] https://www.unicode.org/versions/Unicode16.0.0/core-spec/cha...

[2.4] https://www.unicode.org/versions/Unicode16.0.0/core-spec/cha...

[2.5] https://www.unicode.org/versions/Unicode16.0.0/core-spec/cha...

[Tom Scott UTF-8] https://www.youtube.com/watch?v=MijmeoH9LT4

The original intent of Unicode was great: a standard that creates a mapping between a unique number==codepoint and specific character of language (and here character means only abstract non-visual symbol==meaning, not visually rendered glyph with stylistic font of any kind). The updates for Unicode versions added more languages, even dead ones. So basically it was a historical knowledge effort also.

Then came emojis, and now the Unicode Consortium's efforts for Unicode version updates seems to be about adding more different kinds of poop emojis and shades of skin colors. Well, maybe it projects accurately the language and culture of this modern time.

UTF-8 is great because it is a superset of ASCII, but because its byte-width varies, it has more complexity for decoding/encoding it (similar to constant/variable width ISA's in CPUs).

Different languages have different concepts, e.g. text direction==flow (left/right, up/down, characters/logograms, different kind of visual cues etc.). Humans create problems when they want to combine different languages at the same time. E.g. mathematical notation is in my opinion 2D graphics, and it cannot be (usually/always) inlined with text glyphs (to be aesthetically pleasing). Same kind of problems may come when trying to inline e.g. languages with different flow directions. Its like trying to combine native GUI widgets in Win32 and Cocoa/SwiftUI and GTK/Qt/WXwidgets - the (visual) languages doesn't have the same concepts or they are conflicting.

For what it's worth, the Unicode Consortium seems to be trying to reign in the emoji explosion in the last few years. For example they won't process any new proposals for flags [1][2]:

> The Unicode Consortium will no longer accept proposals for flags. Flags that correspond to officially assigned ISO 3166-1 alpha-2 region codes are automatically added, with no proposals necessary.

And they decided against adding Multi-skintoned Families to the RGI, as in, vendors can encode them if they really want to, but it's not recommended. Apple for example replaced their more complex family emoji with the recommended silhouettes afterwards [4].

1. https://blog.unicode.org/2022/03/the-past-and-future-of-flag...

2. https://unicode.org/emoji/proposals.html#Flags

3. https://www.unicode.org/L2/L2020/20114-family-emoji-explor.p...

4. https://blog.emojipedia.org/ios-17-4-emoji-changelog/

The emoji horse bolted long before Unicode. Everything from MSN Messenger to web forums had their own implementation of it. And that was a continuation of various ASCII emojis ranging from the simple :-) to the more complex ¯\_/(ツ)\_/¯

To say nothing that various emojis have been part of Unicode since pretty much the start, and was part of other encoding schemes as well (notably in Japan, but also e.g. the the "Outlook J").

And if you actually look at the changes in Unicode versions, you'll see there are tons of language-related changes in every one. To say Unicode updates are just about emoji updates is just silly. The reason you don't notice is because this is mostly for small language, obscure features in larger languages, and historical languages, and things like that.

Great corrections from both of you.

Reply on news.ycombinator.com