Hacker News Reader

Comment by flohofwoe | Hacker News Reader

OTH UTF-8 is just one variable-length stream encoding among many others (RLE, LBE128, etc...).

The bonus is synchonizing at arbitrary point in stream and that ASCII is UTF-8

Unless I'm misremembering, even UTF-16 is variable. You need to bump up to UTF-32 to get fixed-width.

Even better, it's arguably both -- surrogate characters are valid codepoint values so technically UTF-16 is fixed-width but programs need to have special handling for surrogate pairs meaning it is practically variable-width.

Truly the worst of all worlds.

Correct me if I'm wrong, but I think all kinds of UTF, including 16 and 32, support arbitrary length for a single effective character. This would be because you can stack modifications as long as you like.

What you meant by "single effective character" is grapheme clusters. This whole discussion is about variable sized code points.

The first comment was kind of iffy when it was also talking about buffers and characters, and focusing on code points is mostly a bad focus. It's worth bringing up so nobody thinks fixed width at a single layer is particularly useful, because other layers will still be variable.

Yes, UTF-16 is the worst of all alternatives and should be abolished rather sooner than later.

UTF-32 is fixed-width for UNICODE code points, but a single visual character (e.g. a "grapheme cluster") can be built from multiple code points. This is separate from the encoding algorithm though, grapheme clusters are mostly a problem for the high level code working with already decoded text data (text rendering, comparison, sorting etc...).

I was working on optimizing a UUID scanner recently, and was baffled when I discovered that Java represents all char primitives in UTF-16, regardless of the original encoding.

So even given an ascii String, if you want to do something like

char c = input.charAt(index)

the JVM is going to jump to that index, check whether the character at that index isLatin(), then cast that single byte into to a 2 byte char... every single time.

in the naive solution, omething like 15% of the CPU cycles were spent checking isLatin() over and over again

Reply on news.ycombinator.com