Hacker News Reader

Comment by Hondor | Hacker News Reader

Wow, so characters like U+022F (ȯ) and U+042F (Cyrillic letter Я) U+062F (Arabic letter د) are not allowed but nearly everything else is? Some of those are letters used in actual languages. That's sure to make people scratch their heads.

View on HN

    codepoint -> encoding in UTF-8

    U+022F    -> C8 AF  
    U+042F    -> D0 AF  
    U+062F    -> D8 AF

Remember, UTF-8 is self-synchronising: when you pick up at a random point within a stream, there is no ambiguity as to whether you are in the middle of a sequence or not. Valid lower codepoints appearing in the encoding of higher codepoints would violate this property.

When encoding those characters in UTF-8, you will never end up with 0x2F as a byte. On of the properties of UTF-8 is that bytes with the high bit not set (e.g. 0x0 to 0x7e) never appear unless the are representing the 0 - 127 codepoints.

If you encode them in UTF-8, they are allowed. In UTF-8, to represent a given code point, only the first code unit can be in the range 0x00 to 0x7F. So if you're reading UTF-8 content and find a byte with value 0x00-0x7F, you can be sure it is a 1-byte long code unit sequence representing the code points U+0000 to U+007F.

Reply on news.ycombinator.com