Comment by derleth | Hacker News Reader

Comment by derleth | Hacker News Reader

> The original ASCII standard defined characters from 0 to 126.

0 to 127. 127 is a power of 2 minus 1, which should be a hint; in specific, it's two to the seventh minus one, since ASCII defines codepoints for all possible combinations of seven bits, which is 128 possible codepoints, so the enumeration ends at 127 if you count starting from zero, as computer programmers are wont to do.

> all possible 127 ASCII characters

128 characters, as mentioned above.

> the ASCII guys, who by the way, were American

ASCII stands for American Standard Code for Information Interchange. The ethnocentrism was unfortunate but it isn't like you weren't warned.

> The numbers are called “magic numbers” and they begin with U+.

He can call them "magic numbers" but everyone else calls them "codepoints".

> UTF-8 was an amazing concept: it single handedly and brilliantly handled backward ASCII compatibility making sure that Unicode is adopted by masses. Whoever came up with it must at least receive the Nobel Peace Prize.

I'm sure Ken Thompson and Rob Pike will be happy to hear someone thinks that way.

> 0 to 127. 127 is a power of 2 minus 1, which should be a hint; in specific, it's two to the seventh minus one ...

To expand on this a tiny bit, a power of two minus one is called a "Mersenne number". If the number is prime, it's called ... wait for it ... a Mersenne prime.

When expressed in binary, Mersenne numbers are an uninterrupted series of one digits: 11111... of varying lengths.

http://mathworld.wolfram.com/MersenneNumber.html

>> UTF-8 was an amazing concept: it single handedly and brilliantly handled backward ASCII compatibility making sure that Unicode is adopted by masses. Whoever came up with it must at least receive the Nobel Peace Prize.

> I'm sure Ken Thompson and Rob Pike will be happy to hear someone thinks that way.

If that's the case then I think the designers of GB18030 deserve it more, because they achieved an encoding that is able to map all Unicode codepoints while being backwards compatible with GB2312, which is itself backwards compatible with ASCII.

But seriously UTF-8 is like sliced bread after having dealt with we-thought-64K-is-enough-so-lets-all-use-16-bits UCS-2/UTF-16.

And even more so, if the current 20.1-bit format turns out to be insufficient, UTF-8 is happy to expand to at least 31 bits, where UTF-16 would be forced to undergo very awkward redesign, such as dedicating an entire plane to double-surrogates.

The flip side of this is that there is such a thing as invalid UTF-8, i.e. a sequence of bytes that does not form a sequence of valid code units and so cannot be decoded as UTF-8. UTF-16 does not suffer from that particular problem.

Of course, there's a long ways to go from "a sequence of valid code units" to "a valid string," but it's still relevant.

UTF-8 is already much better than the legacy double-byte encodings (Big5, GB2312, Shift-JIS etc etc) because, at least you can tell from any byte whether it is supposed to be a part of a code sequence, without having to start all the way from the beginning of the string.

>UTF-16 does not suffer from that particular problem

what about surrogate pairs? You can't have only one 16 bit word for a pair and have a valid UTF-16 sequence. This problem is real easy to do if you substring a UTF-16 sequence naively.

Right, there's still plenty of ways you can screw up a string. My point was merely that UTF-8 has one failure mode (invalid code units) that UTF-16 does not suffer from, and that has implications in encoding converters and APIs.

Invalid bytes is a feature. It makes it much easier to figure out the bytes you have are not actually UTF8.

This is untrue, see D79 and D89 of Unicode 6.2: an ill-formed UTF-16 string is one containing lone surrogates.

Yes, that's an ill-formed string. As I wrote, there's a long ways to go from a "sequence of valid code units" to "a valid string."

But what I wrote is nevertheless correct, and matters. See for example the special requirements on UTF-8 processing in D93:

    If the converter encounters an ill-formed UTF-8
    code unit sequence...it must not consume the
    successor bytes as part of the ill-formed
    subsequence whenever those successor bytes
    themselves constitute part of a well-formed UTF-8
    code unit subsequence....For a UTF-8 conversion
    process to consume valid successor bytes is not
    only non-conformant, but also leaves the converter
    open to security exploits."

"Ill-formed code unit sequence" is not possible with UTF-16. That's an advantage. Also note the last line: the existence of ill-formed code units has security implications that are not present with UTF-16.

To be fair, UTF-16 has its own unique issues too, like endianness.

> The flip side of this is that there is such a thing as invalid UTF-8, i.e. a sequence of bytes that does not form a sequence of valid code units and so cannot be decoded as UTF-8.

In what sort of situation that is a problem?

It can allow for security exploits, if the different layers of your stack handle invalid code units differently.

There's a related issue with "non-shortest forms:" it's possible to encode the NUL character '\0' without actually using a null byte. This means that (for example) "round tripping" UTF-8 data can introduce a null byte, allowing for security exploits when mixed with C-style string processing.

>ASCII stands for American Standard Code for Information Interchange. The ethnocentrism was unfortunate but it isn't like you weren't warned.

Yea, I think it reflects how much technology development comes from America even back then as well as now.

Rob Pike's story on how he and Ken Thompson invented UTF-8 over dinner:

https://plus.google.com/101960720994009339267/posts/Rz1udTvt...

Reply on news.ycombinator.com