Hacker News Reader

Comment by rwmj | Hacker News Reader

I wonder if we can now start to abandon non-UTF8 charsets? Ignoring Windows which is wilfully incompatible and broken, UTF-8 is used pretty much everywhere that matters, and I would argue that those who don't use it should use it. If there is something UTF-8 or Unicode can't do, let's fix that.

View on HN

I wouldn't recommend dropping other Unicode encodings. I assume e.g. UTF-16 is more popular in non-ASCII-based locales since it never requires more than 2 bytes for characters in the BMP (basic multilingual plane). I guess you could pretend the 30% of the web that's in ASCII is UTF-8, but then you'd still be missing all the Chinese and Japanese sites that aren't using Unicode yet. That's almost 10% of the web as a whole and obviously if you're targeting those markets it would be a lot higher.

There are very few rational reasons to be using utf-16 instead of utf-8 and I would be as bold as to claim that the two aren't even close in popularity for non-ascii texts, utf-8 is clearly well above. Regarding memory consumption, latin texts also primordially use ascii characters, with only the occasional non-ascii glyphs (cedillas, accents, tremas and other such characters), the difference between utf-8 and 16 is that the former uses 1 byte to represent characters in the ascii range and 2 bytes for the occasional â, whereas the latter always uses 2 bytes, which would explain utf-8's popularity in latin based alphabets. Also, because having ordinary characters whose code points are padded with zeros is just wasteful and inefficient, utf-16 and its brethren utf-32 introduced funky attempts at optimizations such as Byte Order Marks. Why would anyone wanna deal with this stuff when writing an internationalized program?

Tell that to the Japanese, whose glyphs are always 3+ bytes in UTF-8!

Very astute point and I would grant that it could be a consideration when writing a program for which you don't intend to ever output content on the web. I personally believe that an extra byte per character is a rather small price to pay for all the good of utf-8 and I would not even give a second look to utf-16.

Now, consider that if you decide to use utf-16 to publish your Japanese text as an HTML document (whose syntax is entirely ascii based), you might lose its "saving advantage" and could easily end up with text that is actually larger in utf-16 than in utf-8.

Also take a look at this: http://programmers.stackexchange.com/questions/102205/should...

I'm curious whether there is more information per byte in UTF-8 hanzi / kanji compared to English, considering each character is about 3 bytes but can represent an entire word. A potential human-readable compression technique! ;)

It doesn't really matter since most document formats already use compression. No matter what encoding you use, the amount of entropy is the same. So UTF-8 usually compresses better than 2-byte encodings for CJK languages. This nearly compensates for the increased size.

Example: The Korean text of the Universal Declaration of Human Rights [1] is 8.1KB in EUC-KR and 11.2KB in UTF-8. When compressed with bzip2, it's only 3.1KB and 3.2KB, respectively. I assume Japanese would behave similarly.

[1] http://www.ohchr.org/EN/UDHR/Pages/Language.aspx?LangID=kkn

What the Japanese use isn't even UTF-16, it's more often something like EUC-JP which predates Unicode and uses different encodings. Same with EUC-KR in Korea and several different Chinese encodings.

The web consists of a lot of markup and javascript. That's all in ASCII compatible characters. So on the web UTF-8 usually wins even for Japanese or Chinese texts. For example take a random page of the Japanese Wikipedia and encode it in UTF-8 and in UTF-16. In all my tries UTF-8 was smaller.

Afaik UTF-32 is relatively convenient as it is fixed-width, and thus many operations are faster to perform than in UTF-8.

You might want to use UCS-4 as an internal format, or some other convenient format (for example ASCII if you know everything is 7/8 bit). But UTF-8 should still be the only thing seen externally.

You still have to worry about combining characters.

Exactly. One UTF-32 character (code-point) is not one displayed character (glyph), and one UTF-32 character is not one character to search.

So basically it's not fixed width in any meaningful way.

I largely share sqrt17's grief.

There is no such thing as "UTF-32 character".

Abstract character is not code point.

UTF-32 _is_ fixed width because it's defined on code points, not glyphs, not characters.

Fixed width in what way? It's not fixed width for display or search, so what's fixed about it?

Reply on news.ycombinator.com