Comment by mekoka | Hacker News Reader

Comment by mekoka | Hacker News Reader

Parent Full thread

There are very few rational reasons to be using utf-16 instead of utf-8 and I would be as bold as to claim that the two aren't even close in popularity for non-ascii texts, utf-8 is clearly well above. Regarding memory consumption, latin texts also primordially use ascii characters, with only the occasional non-ascii glyphs (cedillas, accents, tremas and other such characters), the difference between utf-8 and 16 is that the former uses 1 byte to represent characters in the ascii range and 2 bytes for the occasional â, whereas the latter always uses 2 bytes, which would explain utf-8's popularity in latin based alphabets. Also, because having ordinary characters whose code points are padded with zeros is just wasteful and inefficient, utf-16 and its brethren utf-32 introduced funky attempts at optimizations such as Byte Order Marks. Why would anyone wanna deal with this stuff when writing an internationalized program?

Tell that to the Japanese, whose glyphs are always 3+ bytes in UTF-8!

Very astute point and I would grant that it could be a consideration when writing a program for which you don't intend to ever output content on the web. I personally believe that an extra byte per character is a rather small price to pay for all the good of utf-8 and I would not even give a second look to utf-16.

Now, consider that if you decide to use utf-16 to publish your Japanese text as an HTML document (whose syntax is entirely ascii based), you might lose its "saving advantage" and could easily end up with text that is actually larger in utf-16 than in utf-8.

Also take a look at this: http://programmers.stackexchange.com/questions/102205/should...

I'm curious whether there is more information per byte in UTF-8 hanzi / kanji compared to English, considering each character is about 3 bytes but can represent an entire word. A potential human-readable compression technique! ;)

It doesn't really matter since most document formats already use compression. No matter what encoding you use, the amount of entropy is the same. So UTF-8 usually compresses better than 2-byte encodings for CJK languages. This nearly compensates for the increased size.

Example: The Korean text of the Universal Declaration of Human Rights [1] is 8.1KB in EUC-KR and 11.2KB in UTF-8. When compressed with bzip2, it's only 3.1KB and 3.2KB, respectively. I assume Japanese would behave similarly.

[1] http://www.ohchr.org/EN/UDHR/Pages/Language.aspx?LangID=kkn

Reply on news.ycombinator.com