Plain Text Doesn’t Exist: Unicode and encodings demystified | Hacker News Reader

Plain Text Doesn’t Exist: Unicode and encodings demystified

10kloc.wordpress.com

48 pointsperseus32338 comments

Contrary to the editorialized title (which I'm sure will be changed soon), here's the best article on Unicode I've ever read: http://www.joelonsoftware.com/articles/Unicode.html

I feel like we're at a point now where articles that just try to 'demystify' unicode are almost teaching the controversy if they don't come out and actually say how you should deal with encodings in new apps.

It's about time we actually start pressing for the idea that utf-16 was a terrible idea and that utf-8 should be the dominant wire format for unicode, with ucs4 if you really need to have a linear representation.

Utf-16 is confusing, complicated, and implementations are routinely broken because the corner cases are rarer. I really hope we're not still stuck with it in 50 years.

I agree about wire formats. However, programming languages that use UTF-8 as the primary string representation tend to have inferior Unicode support to those that use UTF-16. I think this is because with UTF-8, a lot of stuff just seems to work and so it's easy to ignore the issues, while UTF-16 forces you to come to grips with the realities of Unicode.

For example, consider a function to open a file by name. If your strings are UTF-8, you can just pass your null terminated buffer to fopen() or something and things will work fine for most files. But if your strings are internally UTF-16, you have to think about which encoding to use, and you research it, and you discover that holy crap, this stuff differs across OSes, and so we better take this problem seriously.

> However, programming languages that use UTF-8 as the primary string representation tend to have inferior Unicode support to those that use UTF-16.

I'm really curious which languages drove you to this conclusion. Right now we're in a state where the most mature implementations of unicode in general are utf-16 because of the ucs2 accident, so it wouldn't exactly surprise me, but I'd still like to see the proof.

UTF-16 has the same issue with invalid things appearing to work due to surrogate pairs. A lot of UTF-16 based applications actually only support UCS-2, because you can assume one string index equals one character and get away with it if it never occurs to you to try to use stuff outside of the BMP.

More ethnocentrism. UTF-16 results in less bandwidth for any language that has non-ASCII characters.

I think the Unicode group's handling of the round trip issues faced by Japanese and Chinese users with their own encodings due to Han Unification is a bigger ethnocentric problem by far. I'm not unaware of these issues.

But it's not like utf-16 was invented to solve that problem so much as it was an accident that in the early days of unicode 2 bytes was enough to encode all code points and a 2 byte encoding was appealing as a universal solution in a way that a 3 or 4 byte encoding never would be. Some group of people on the planet will have to suffer 3 or more byte encodings no matter what. And I'd be happy to make it mine if it meant we could go with the better technical solution, if I had the power to do that.

All that aside it's certainly not quite as clear cut as any language. Many languages have a mix of ascii and non-ascii characters and those fare better than they would with utf-16. And in the codepoints below 0x07ff (where utf-8 goes up to 3 bytes) there are character sets like greek, hebrew, arabic, and cyrillic. And as has been pointed out, ascii characters are often used in structural elements of documents and this factors into making it unlikely that the general case is as bad as the worst case (50% inflation) for most practical text. Particularly when you throw compression into the mix.

[edit] fixed some minor things.

That's not true. For anything that uses the latin alphabet utf8 will be strictly better as the majority of characters will be in ASCII (1 byte) and the rest will just need an extra byte, equaling utf16.

Looking at the format definition[1] anything below U+0800 will fit in 2 bytes in UTF8. So the ethnocentrism starts at Samaritan[2]. The Japanese, Chinese and Indian scripts are probably the most important in that range.

[1] https://en.wikipedia.org/wiki/UTF-8#Description

[2] http://codepoints.net/basic_multilingual_plane

I can't find the research hsivonen (probably best known for validator.nu and Gecko's HTML parser) did into this a while ago, but he concluded that for the top 100 (IIRC) CJK sites, they were smaller as gzip'd UTF-8 than gzip'd UTF-16, because they all contained a sufficient quantity of HTML/CSS/JS that the gain from the shorter representation of them outweighed that of the text itself.

What is the argument for utf-8 that doesn't also apply to utf-16, depending on what your common characters are? Applications that use Asian languages take a big hit from utf-8.

>What is the argument for utf-8 that doesn't also apply to utf-16

The comment you're replying to says right there, "implementations are routinely broken because the corner cases are rarer" utf-16 misleads people into using fixed-width code where it won't work right.

>Applications that use Asian languages take a big hit from utf-8.

Not really. It's a moderate hit for pure text, but how often do you see bulk pure text, without any markup, and that can't be compressed?

> utf-16 misleads people into using fixed-width code where it won't work right

This is exactly my point: lots of code also treats utf-8 as if it were ASCII.

> how often do you see bulk pure text, without any markup, and that can't be compressed?

That's a good argument. Thanks :)

> This is exactly my point: lots of code also treats utf-8 as if it were ASCII.

Two things to this:

- In many cases this is an entirely non-destructive mistake to make. This is an advantage of utf-8, that intermediaries that only care about ascii characters (even in the presence of extended characters) can work with it.

- It will become readily apparent that you've got a problem on the first attempt to port your app to any language with a non-ascii script. With UTF-16 it only becomes apparent when you go beyond the BMP.

there are no surrogates in utf-8. that alone is reason to use it over utf-16

I don't get it. Surrogates in UTF-16 are when you have two code units that map to a single code point. In UTF-8, you have that same issue, except it may be two, three, or four code units. So how is UTF-8 better in this respect?

You're probably looking for this:

http://www.utf8everywhere.org/

Interesting talk by Nick Patch of Shutterstock about Unicode, why they use Perl, and what he calls best practices for working with encodings: http://www.youtube.com/watch?v=X2FQHUHjo8M

HN guidelines ask you to retain the original title of the article, not to inject your own editorial opinion[1]. This article is a decent quick overview for someone who has never studied Unicode before, but a bit light on the details compared to what's available, say, in Wikipedia's articles on Unicode[2].

[1] http://ycombinator.com/newsguidelines.html

[2] https://en.wikipedia.org/wiki/Unicode

>Other common myths include: Unicode can only support characters up to 65,536

Not really a myth. This was UCS2 and was the situation when a load of important early adopters started with Unicode. Winodws, Java, JavaScript all got burnt by this and ended up with UTF-16 as a result. Even Python 2.x on Linux is UTF-16 under the covers :(

>Unicode is just a standard way to map characters to magic numbers and there is no limit on the number of characters it can represent.

Unicode now limits itself to 21 bits of data. This is what allows the surrogate pair coding of UTF-16

At first there was no plain text. Then ASCII became standard enough and if you passed a file to someone they would be able to read it without you needing to tell them an encoding. Extensions of ASCII and other options spread, and there was no way to have plain text include characters beyond the 128 of ASCII. Over the past decade or so we've been consensing on utf8, however, and the probability that some random piece of non-ASCII text is utf8 has been getting sufficiently high.

There is plain text, and it is utf8.

By the way: A very interesting article for those interested in a bit of context around unicode and how it doesn't solve everything (especially from an asian perspective) is this one:

http://web.archive.org/web/20090627072117/http://www.jbrowse...

Character encodings are a pain in the ass. If you want some examples, Michael Kaplan posts [1] about these sorts of things in way too much detail.

[1] http://blogs.msdn.com/b/michkap/archive/tags/unicode+lame+li...

this seems strangely similar to Joel Spolsky's article on unicode.

http://www.joelonsoftware.com/articles/Unicode.html

Though not exactly plagiarised, it shares some obvious parallels.

> The original ASCII standard defined characters from 0 to 126.

0 to 127. 127 is a power of 2 minus 1, which should be a hint; in specific, it's two to the seventh minus one, since ASCII defines codepoints for all possible combinations of seven bits, which is 128 possible codepoints, so the enumeration ends at 127 if you count starting from zero, as computer programmers are wont to do.

> all possible 127 ASCII characters

128 characters, as mentioned above.

> the ASCII guys, who by the way, were American

ASCII stands for American Standard Code for Information Interchange. The ethnocentrism was unfortunate but it isn't like you weren't warned.

> The numbers are called “magic numbers” and they begin with U+.

He can call them "magic numbers" but everyone else calls them "codepoints".

> UTF-8 was an amazing concept: it single handedly and brilliantly handled backward ASCII compatibility making sure that Unicode is adopted by masses. Whoever came up with it must at least receive the Nobel Peace Prize.

I'm sure Ken Thompson and Rob Pike will be happy to hear someone thinks that way.

> 0 to 127. 127 is a power of 2 minus 1, which should be a hint; in specific, it's two to the seventh minus one ...

To expand on this a tiny bit, a power of two minus one is called a "Mersenne number". If the number is prime, it's called ... wait for it ... a Mersenne prime.

When expressed in binary, Mersenne numbers are an uninterrupted series of one digits: 11111... of varying lengths.

http://mathworld.wolfram.com/MersenneNumber.html

>> UTF-8 was an amazing concept: it single handedly and brilliantly handled backward ASCII compatibility making sure that Unicode is adopted by masses. Whoever came up with it must at least receive the Nobel Peace Prize.

> I'm sure Ken Thompson and Rob Pike will be happy to hear someone thinks that way.

If that's the case then I think the designers of GB18030 deserve it more, because they achieved an encoding that is able to map all Unicode codepoints while being backwards compatible with GB2312, which is itself backwards compatible with ASCII.

But seriously UTF-8 is like sliced bread after having dealt with we-thought-64K-is-enough-so-lets-all-use-16-bits UCS-2/UTF-16.

And even more so, if the current 20.1-bit format turns out to be insufficient, UTF-8 is happy to expand to at least 31 bits, where UTF-16 would be forced to undergo very awkward redesign, such as dedicating an entire plane to double-surrogates.

The flip side of this is that there is such a thing as invalid UTF-8, i.e. a sequence of bytes that does not form a sequence of valid code units and so cannot be decoded as UTF-8. UTF-16 does not suffer from that particular problem.

Of course, there's a long ways to go from "a sequence of valid code units" to "a valid string," but it's still relevant.

>UTF-16 does not suffer from that particular problem

what about surrogate pairs? You can't have only one 16 bit word for a pair and have a valid UTF-16 sequence. This problem is real easy to do if you substring a UTF-16 sequence naively.

Right, there's still plenty of ways you can screw up a string. My point was merely that UTF-8 has one failure mode (invalid code units) that UTF-16 does not suffer from, and that has implications in encoding converters and APIs.

Invalid bytes is a feature. It makes it much easier to figure out the bytes you have are not actually UTF8.

UTF-8 is already much better than the legacy double-byte encodings (Big5, GB2312, Shift-JIS etc etc) because, at least you can tell from any byte whether it is supposed to be a part of a code sequence, without having to start all the way from the beginning of the string.

> The flip side of this is that there is such a thing as invalid UTF-8, i.e. a sequence of bytes that does not form a sequence of valid code units and so cannot be decoded as UTF-8.

In what sort of situation that is a problem?

It can allow for security exploits, if the different layers of your stack handle invalid code units differently.

There's a related issue with "non-shortest forms:" it's possible to encode the NUL character '\0' without actually using a null byte. This means that (for example) "round tripping" UTF-8 data can introduce a null byte, allowing for security exploits when mixed with C-style string processing.

This is untrue, see D79 and D89 of Unicode 6.2: an ill-formed UTF-16 string is one containing lone surrogates.

Yes, that's an ill-formed string. As I wrote, there's a long ways to go from a "sequence of valid code units" to "a valid string."

But what I wrote is nevertheless correct, and matters. See for example the special requirements on UTF-8 processing in D93:

    If the converter encounters an ill-formed UTF-8
    code unit sequence...it must not consume the
    successor bytes as part of the ill-formed
    subsequence whenever those successor bytes
    themselves constitute part of a well-formed UTF-8
    code unit subsequence....For a UTF-8 conversion
    process to consume valid successor bytes is not
    only non-conformant, but also leaves the converter
    open to security exploits."

"Ill-formed code unit sequence" is not possible with UTF-16. That's an advantage. Also note the last line: the existence of ill-formed code units has security implications that are not present with UTF-16.

To be fair, UTF-16 has its own unique issues too, like endianness.

>ASCII stands for American Standard Code for Information Interchange. The ethnocentrism was unfortunate but it isn't like you weren't warned.

Yea, I think it reflects how much technology development comes from America even back then as well as now.

Rob Pike's story on how he and Ken Thompson invented UTF-8 over dinner:

https://plus.google.com/101960720994009339267/posts/Rz1udTvt...

Reply on news.ycombinator.com