Hacker News Reader

Comment by alexjeffrey | Hacker News Reader

I believe it depends on the encoding - UTF-32 has a limit of 2^32 characters, but UTF-8 could potentially expand infinitely (ignoring the current arbitrary 4-byte limit).

View on HN

That's incorrect - the UTF-8 scheme encodes the sequence length in the first byte as the number of bits set to 1 until the first 0.

The original limit was 6 bytes encoding 31 bits, but this was lowered to 4 bytes encoding 21 bits. If I'm not mistaken, 7 bytes encoding 36 bits (or 8 bytes encoding 42 bits if you make the 0 optional) should be the hard limit.

If we ignore this artificial restriction, we can use arbitrary lengths, turning UTF-8 into varint encoding.

I.e.:

      Oct.1    Oct.2    Oct.3
    11111111 110DDDDD DDDDDDDD (7 octets more)
    \_________/
      10 bits

   ("D"s represent data bits)

This is what'd be necessary to use UTF-8 if we'd need to go beyond the current limits of 0x010FFFF upper bound.

In your scheme, we'll end up with multiple leading bytes, though.

Also, how do you plan to handle 9 leading bits instead of the 10 from your example? That would make the second byte start with 10, the marker for continuation bytes.

Ouch, you're right. Refreshed my memory of UTF-8 encoding and now it seems it's designed so it can't be made to express arbitrary values. Not sure why I imagined the opposite. My mistake.

The current limit (0x10FFFF) is the artificial restriction on UTF-8 to match UTF-16, not the upper bound. UTF-8 supports 2^31 code points with it's current encoding. So nothing needs to change until Unicode consists of 2.1 billion code points or so.

In my mind that will only happen as a result of extreme carelessness.

Reply on news.ycombinator.com