Hacker News Reader

Comment by sp332 | Hacker News Reader

I wouldn't recommend dropping other Unicode encodings. I assume e.g. UTF-16 is more popular in non-ASCII-based locales since it never requires more than 2 bytes for characters in the BMP (basic multilingual plane). I guess you could pretend the 30% of the web that's in ASCII is UTF-8, but then you'd still be missing all the Chinese and Japanese sites that aren't using Unicode yet. That's almost 10% of the web as a whole and obviously if you're targeting those markets it would be a lot higher.

View on HN

There are very few rational reasons to be using utf-16 instead of utf-8 and I would be as bold as to claim that the two aren't even close in popularity for non-ascii texts, utf-8 is clearly well above. Regarding memory consumption, latin texts also primordially use ascii characters, with only the occasional non-ascii glyphs (cedillas, accents, tremas and other such characters), the difference between utf-8 and 16 is that the former uses 1 byte to represent characters in the ascii range and 2 bytes for the occasional â, whereas the latter always uses 2 bytes, which would explain utf-8's popularity in latin based alphabets. Also, because having ordinary characters whose code points are padded with zeros is just wasteful and inefficient, utf-16 and its brethren utf-32 introduced funky attempts at optimizations such as Byte Order Marks. Why would anyone wanna deal with this stuff when writing an internationalized program?

Tell that to the Japanese, whose glyphs are always 3+ bytes in UTF-8!

Very astute point and I would grant that it could be a consideration when writing a program for which you don't intend to ever output content on the web. I personally believe that an extra byte per character is a rather small price to pay for all the good of utf-8 and I would not even give a second look to utf-16.

Now, consider that if you decide to use utf-16 to publish your Japanese text as an HTML document (whose syntax is entirely ascii based), you might lose its "saving advantage" and could easily end up with text that is actually larger in utf-16 than in utf-8.

Also take a look at this: http://programmers.stackexchange.com/questions/102205/should...

I'm curious whether there is more information per byte in UTF-8 hanzi / kanji compared to English, considering each character is about 3 bytes but can represent an entire word. A potential human-readable compression technique! ;)

It doesn't really matter since most document formats already use compression. No matter what encoding you use, the amount of entropy is the same. So UTF-8 usually compresses better than 2-byte encodings for CJK languages. This nearly compensates for the increased size.

Example: The Korean text of the Universal Declaration of Human Rights [1] is 8.1KB in EUC-KR and 11.2KB in UTF-8. When compressed with bzip2, it's only 3.1KB and 3.2KB, respectively. I assume Japanese would behave similarly.

[1] http://www.ohchr.org/EN/UDHR/Pages/Language.aspx?LangID=kkn

The web consists of a lot of markup and javascript. That's all in ASCII compatible characters. So on the web UTF-8 usually wins even for Japanese or Chinese texts. For example take a random page of the Japanese Wikipedia and encode it in UTF-8 and in UTF-16. In all my tries UTF-8 was smaller.

What the Japanese use isn't even UTF-16, it's more often something like EUC-JP which predates Unicode and uses different encodings. Same with EUC-KR in Korea and several different Chinese encodings.

Reply on news.ycombinator.com