Hacker News Reader

Comment by nordsieck | Hacker News Reader

> Not all grapheme clusters have composed forms so normalization doesn't actually gain you anything here.

Just because the worst case can't improve doesn't mean that making the average case better is worthless.

I saw something about Arabic text, where that naive truncation at codepoint boundaries turns one word into a different word! Like the sequence of codepoints generate something that is represented as a single glyph in fonts, but truncated its totally different glyphs. I don't remember more details, I don't know any Arabic, but grapheme clusters aren't just about adding diacritics to latin characters. In other languages it all might work quite differently. So truncating at word boundaries (at breakable white-space or punctuation) is probably best. Though of course that way you might truncate the string by a lot. shrug-emoji

(I don't think the talk where the stuff about Arabic was mentioned was Plain Text by Dylan Beattie, but I haven't re-watched it to confirm. So maybe it is. Can't remember the name of any other talk about the subject right now.)

Randomly truncating words can have the same effect in any language. It's outright trivial to find examples in English or German. I don't understand why one has to invoke Arab script for a good example.

Yes, but you don’t end up with different glyphs. Arabic script has letter shaping, that means a letter can have up to 4 shapes based on its position within the word. If you chop off the last letter, the previous one which used to have a “middle” position shape suddenly changes into “terminal” position shape.

I'm thinking even bog-standard European umlauts, cedillas, etc go multi-byte in Unicode? (Take a string of ÅÄÖåäöÜü and chop it off at various byte limits and see.)

I don't remember the details, only that it was a bigger deal than with umlauts. I'll see if I can find the talk again.

This is just the general behavior of truncating strings by code point when they contain decomposed glyphs. This can also impact accents etc.

Reply on news.ycombinator.com