Java May Use UTF-8 as Its Default Charset | Hacker News Reader

Java May Use UTF-8 as Its Default Charset

marxsoftware.blogspot.com

33 pointsingve29 comments

Article is about setting up locale during compilation/runtime. Internally java/jvm uses only UTF-16 and it is not going to change because of backward compatibility reasons.

Char is a 16bit type, so some unicode 'characters' have high/low surrogate pairs, those pairs are encoded in 32bit int as `codePoint`. Example of a problem: String#length returns number of chars, not number of codepoints.

String/Character api is a one big record of utf16 madness: http://grepcode.com/file/repository.grepcode.com/java/root/j... http://grepcode.com/file/repository.grepcode.com/java/root/j...

Most of contemporary technologies use UTF8 for string encoding and exchange format. This back and forth UTF8<>UTF16 conversion in java doesn't come without a computational&memory price.

Yes, please! This aligns with my rants on the subject [1] because I want all software to use UTF-8 by default. In my experience as an application developer nearly all of my character encoding grief has been caused by an application or library somewhere in my overall system architecture not using UTF-8 by default. This is followed by me needing to find how to configure it. I am by no means a character encoding expert, but I want everything to use UTF-8 unless there's a very particular need for something else (I'm still waiting to find out what that might be).

[1] http://tiamat.tsotech.com/utf8-everything

Software that needs to display Japanese names (particularly if it may also need to display Chinese names) tends to use traditional encodings to avoid Unicode's Han unification.

(Some characters are considered by Unicode to be the same character despite the Japanese and Chinese ways of writing that character being visually distinguishable; Unicode regards this as the same character being written in different fonts (just as in our Latin alphabet there are multiple visually distinct ways to write the lower-case "a" or the number 4, one can choose to cross "7"s or not, etc.) but customers sometimes don't find that acceptable; in particular many Japanese people are very unhappy to see their names written with the Chinese versions of those characters, which tend to be how international fonts render them)

So it is the same symbol, recognized as that symbol, meaning the same thing but showing up as what some people consider comic sans?

So the problem is the font?

The problem is not the font. Users of CJK actually want this distinction in codepoint assignments, not font choice. Non-CJK users might think it silly, and maybe it is, but whatever the case, unification has been abandoned.

Yes, theoretically speaking this distinction might have been useful, but so would have been combining hieroglyphs and hangul jamo and the ability to define an entirely new CJK character by specifying components and placements would have been even nicer.

None of this would be solved by devolving back to popular pre-unicode standards like EUC-JP or Big5.

Yes except that there's no way to choose a font that all users will be happy with - if you switch your application to a font that uses the Japanese way of writing the character, some Chinese people with that character in their name will be equally unhappy. (It's hard to create an exact analogy because there's no country that the US has the same kind of tense relationship with, and we don't attach quite the same personal meaning to the exact reading of our name - we're used to seeing our names written in capitals and lower-case and being the same name, we don't tell people the meanings of the characters used in our names as we spell them, but I guess imagine having a 7 in your name that you care about quite personally and then a program you use renders it as a French 7 even though you hate the French - you can tell that it's the same character, but it's still wrong and feels very rude. It's the sort of thing that seems trivial in the abstract but ends up being felt surprisingly personally - I'm always a lot more pissed off than I expected to be when people call me "Imm").

You could do something like switching the font based on the system locale, but at that point you're no better off than when you were using the system's character encoding, and following a less well-trodden path.

Well, there is really no way to solve this other than using a language-specific font or markup to indicate the language of the text. The first solution is probably the more common one, since there are very few international fonts that actually do CJK languages well to begin with.

The number of characters you need to cover is hard enough to make it a daunting task, and if Han Unification didn't exist it could have been 3 times harder...

The TRON Multilingual System (which was championed by the Han Unification opponents in the 90s) basically does second solution. It did avoid character unification between Chinese, Japanese and Korean, but it still chose to specify the language of the text to support different overall processing based on the language (e.g. different ligatures may be activated for different European languages).

If you're talking about Japanese users being unhappy that their name characters change when its rendered _within Chinese Text_ - well, that's how it's always been in newspapers, books and official documents. Signs all over Japan (made by Japanese for Chinese tourists) show the Chinese rendition of Japanese place names such as 涩谷 (Shibuya) and nobody seems to bother. It goes the other way too, of course.

> Well, there is really no way to solve this other than using a language-specific font or markup to indicate the language of the text. The first solution is probably the more common one, since there are very few international fonts that actually do CJK languages well to begin with.

Indeed, but - more by accident than design - traditional codepages end up making language-specific fonts easy, since you'll naturally use different fonts for different encodings (and if you don't, it's easy to switch to doing so, because you were already carrying around the encoding for each piece of text so you know which texts are Chinese and which are Japanese). I mean, using UTF-8 you could carry around that metadata manually and achieve the same result, but that's a lot less fail-fast; you usually only realise you needed that information after you've got a substantial amount of text of unknown language stored in your system. Whereas if you fail to store which encoding text is in then you (hopefully) notice sooner.

> Signs all over Japan (made by Japanese for Chinese tourists) show the Chinese rendition of Japanese place names such as 涩谷 (Shibuya) and nobody seems to bother.

Sure, which I think is what mislead the Unicode consortium - for ordinary text no-one cares, and most people's names aren't affected. But the people whose names are affected can take it quite personally.

Unicode abandoned unification.

That doesn't fix the problem for existing characters though, right? If I run a file through "iconv -f SHIFT-JIS -t UTF-8" then it's going to come out using codepoints that are rendered as the "wrong" characters in most international fonts, right?

Is that conversion standardized and stable?

I would hope and expect so - the current implementation may be "wrong" but having it behave differently in a newer version would be even worse than the current situation.

For many scripts a conversion from non-Unicode to Unicode can't really be standard without also specifying a normalization form for output. I don't know of any standards in this space.

I can't imagine defaulting to anything other than NFC.

Sure, but that's not responsive.

I think I've yet to met a Japanese person who was even mildly unhappy to see their name rendered incorrectly due to Han Unification. It seems like when Unicode came out, the TRON guys and some other Japanese engineer were pretty vocal about the evil of Han Unification, but the reality is that the average Japanese user probably doesn't care. They get their name butchered at least once a month in their bank account statement (in half-width Katakana), and if they have an unusual character or a slightly unusual reading in their name, their fellow compatriots would butcher their name as well.

Compared to that, the number of times people from Japan browsing at a site configured to display Japanese language with Chinese Fonts? That must happen once in Super Blue Blood Moon, but not that often. And "many" is definitely exaggeration, considering the number of Japanese name which use characters subject to heavily modified by unification. In other words, "bone" is not a popular character in Japanese names, who would have figured.

Han Unification is not without its pains, of course, and I still can't tell for sure whether it was a good idea, but using EUC-JP or Shift-JIS is not going to get you around it, since you still need a Japanese font to display the characters correctly as they should look in Japanese. And I would gladly take UTF-8 over EUC-JP and Shift-JIS every day.

Using UTF-16 is silly because it doesn't mean that length in code units == length in codepoints -- that was only ever true of UCS-2. Not that length in codepoints is all that useful anyways (because of decomposition: you can't just split on the Nth codepoint, only on the Nth character). But because that was true of UCS-2, some systems (e.g., Windows) selected UCS-2 and then UTF-16. Sad, but true.

UTF-8 is clearly superior.

It's not obvious that UTF-8 is superior. UTF-8 has subtleties such as non-shortest forms and invalid code units. These issues have lead to security exploits [1] but are impossible in UTF-16 by design.

The internal encoding of strings attracts too much attention. A proper string type abstracts over these details. When doing arithmetic in Haskell, Python, JavaScript, etc. you don't worry about how the numbers are encoded; strings should be the same.

[1] https://unicode.org/reports/tr36/#UTF-8_Exploit

A brief search shows for "UTF-16 vulnerability" finds relevant results.

UTF-8 overlong sequences haven't been a problem for a while.

The choice of encoding is definitely relevant for all external file formats, protocols, APIs. We can't just let the choice of encoding be purely internal, though I agree that programmers should not need to care about what their runtimes use internally for strings, as long as it is Unicode (UTF-8, 16, 32, or some other UTF say).

But no UTF actually gives you random access by character index because a) no runtime is going to be ensuring normalization of all strings to precomposed sequences (not even NFC because NFC is closed to new compositions!), b) not every valid character will have pre-composed codepoints therefore (a) isn't even possible. Given this, UTF-16 and UTF-32 have no real advantages over UTF-8, and the false sense of simplicity they convey is dangerous anyways.

Is that "UTF-8 with BOM" or "UTF-8 without BOM"? - because I've had lots of trouble with things supporting only one of those two options.

Wait there's a UTF-8 with BOM? Why do you need a byte order mark for utf-8? The stream "unit" is a byte. I thought only utf-16 and above had an optional BOM?

They're likely referring to this behavior on Windows:

> "Many Windows programs (including Windows Notepad) add the bytes 0xEF, 0xBB, 0xBF at the start of any document saved as UTF-8. This is the UTF-8 encoding of the Unicode byte order mark (BOM), and is commonly referred to as a UTF-8 BOM, even though it is not relevant to byte order. A BOM can also appear if another encoding with a BOM is translated to UTF-8 without stripping it. Software that is not aware of multibyte encodings will display the BOM as three garbage characters at the start of the document, e.g. "ï»¿" in software interpreting the document as ISO 8859-1 or Windows-1252 or "∩╗┐" if interpreted as code page 437, a default for certain older Windows console applications."

> "The Unicode Standard neither requires nor recommends the use of the BOM for UTF-8, but warns that it may be encountered at the start of a file as a transcoding artifact. The presence of the UTF-8 BOM may cause problems with existing software that can handle UTF-8..."

https://en.wikipedia.org/wiki/UTF-8#Byte_order_mark

I should have searched. This is actually quite useful to be aware of. Thanks.

you should remember it when creating csv's especially if your consumer uses windows. saves a ton of "what is a charset" questions.

I could imagine a UTF-8 BOM to be useful in helping identify that a given file is UTF-8-encoded, when the encoding is not clear from context (such as an outer MIME type specification or explicit encoding specs like those in XML documents).

Without. The BOM is for UTF-16 and UTF-32. Nor would would you want a BOM on every string in memory -- that would make it impossible to cheaply substring, for example. BOMs are for external documents -- for interop.

Ok, but for files, it does happen

https://docs.microsoft.com/en-us/globalization/encoding/byte...

I understand, but it's a mistake and completely unnecessary.

Reply on news.ycombinator.com