Hacker News Reader

Comment by ainar-g | Hacker News Reader

I feel like you're confusing it with UTF-32, which is fixed-length. UTF-16 is neither fixed-length, like UTF-32, nor compatible with the old ASCII encoding, like UTF-8.

The historical reasons of why UTF-16 exists are fully understandable, but by now it is clear that it's the worst of both worlds.

View on HN

No, I'm fully aware of it being a variable-length encoding. Yet as such it is still simpler to work with than UTF-8. So if we are to use a variable-length encoding anyway then UTF-16 is a reasonable choice. It is not compatible with ASCII and old APIs, that's true, but I cannot judge how important this is. I guess that for some projects the efficiency of UTF-16 should outweigh the inconveniences of that incompatibility.

Can you explain in what way it is easier to handle than UTF-8? Most code that I've seen and looks easier just mishandles characters outside the basic multilingual plane. Thankfully, due to emojis being a thing, a lot of developers woke up to the fact that Unicode specifies more than 65536 characters.

This is plain decoding of a trusted string:

    utf16 decode
      if (unit <= 0xD7FF || unit >= 0xE000)
        /* one unit */;
      else
        /* two units */
    utf-8 decode
      if (unit < 0xE0) {
        if (unit < 0xC0)
          /* one unit */
        else
          /* two units */
      }
      else if (unit < 0xF0)
        /* three units */
      else
        /* four units */

UTF-16 will take one comparison for all characters up to 0xD7FF; UTF-8 normally takes two. Same when encoding:

    utf16 encode
      if (value <= 0xFFFF)
        /* one unit */
      else
        /* two units */
    utf8 encode
      if (value <= 0x7FF) {
        if (value <= 0x7F)
          /* one unit */
        else
          /* two units */
      }
      else if (value <= 0xFFFF)
        /* three units */
      else
        /* four units */

(We can tune UTF-8 it to take one comparison for ASCII if we expect mostly ASCII.)

Things get even more complex when we are to read an untrusted string. In UTF-8 we have five byte types, invalid bytes, conditionally invalid bytes, and must recognize surrogates (as errors). In UTF-16 all 16-bit units are valid and there are only three unit types: character or surrogate, high or low. I once wrote a decoder for UTF-8/16 and here's my stats:

    UTF-8    : 13 byte types, 13 states, 10 actions
    UTF-16LE :  3 byte types,  4 states,  5 actions

utf-8 has an elegance that utf-16 lacks:

    utf-8 decode
      switch (std::countl_one(unit)) {
        case 0:
          /* one unit */
          break;
        case 2:
          /* two units */
          break;
        case 3:
          /* three units */
          break;
        case 4:
          /* four units */
          break;
        default:
          /* not code point boundary */
          break;
      }

Oh, thanks, this one is good. Must be rather fast on major platforms. But with this optimization UTF-8 still remains harder computationally than UTF-16.

I’m happy every time I see c++20 in the wild

UTF-16 is cheaper to decode. All CPUs have branch predictors, 99.99% of all UTF-16 strings don't contain any surrogate pairs.

For East Asian languages UTF-16 uses 50% less memory, these characters take 3 bytes in UTF-8, but only 2 bytes in UTF-16.

That's 33% less memory, or conversely, UTF-8 uses 50% more memory.

Can you show some concrete instances of it being easier to work with compared to UTF-8? As far as I can see you gain variable length encoding (albeit in a potentially confusing way) while losing ASCII support and also not gaining the pure simplicity of UTF-32. Oh and also endianness, although personally that's probably never an issue I will run into.

(Please see my reply to a sibling comment.)

Perfect, thanks for the heads up!

Reply on news.ycombinator.com