Hacker News Reader

Comment by drdaeman | Hacker News Reader

If we ignore this artificial restriction, we can use arbitrary lengths, turning UTF-8 into varint encoding.

I.e.:

      Oct.1    Oct.2    Oct.3
    11111111 110DDDDD DDDDDDDD (7 octets more)
    \_________/
      10 bits

   ("D"s represent data bits)

This is what'd be necessary to use UTF-8 if we'd need to go beyond the current limits of 0x010FFFF upper bound.

View on HN

In your scheme, we'll end up with multiple leading bytes, though.

Also, how do you plan to handle 9 leading bits instead of the 10 from your example? That would make the second byte start with 10, the marker for continuation bytes.

Ouch, you're right. Refreshed my memory of UTF-8 encoding and now it seems it's designed so it can't be made to express arbitrary values. Not sure why I imagined the opposite. My mistake.

The current limit (0x10FFFF) is the artificial restriction on UTF-8 to match UTF-16, not the upper bound. UTF-8 supports 2^31 code points with it's current encoding. So nothing needs to change until Unicode consists of 2.1 billion code points or so.

In my mind that will only happen as a result of extreme carelessness.

Reply on news.ycombinator.com