The flaws of thinking "Just use Unicode"
itworld.com
itworld.com
We use the IBM ICU library for our product and it handles the vast majority of the issues presented in the article. There is no reason for me to be writing the code to handle the thousands of permutations of language, country, and culture when someone else has already done the heavy lifting here. ICU solves problems such as: sorting, number representation, etc.
The other problems should be things unrelated to Unicode - things your product / business already need.
1)Pick a target market - If you haven't picked a target market worrying about using Unicode to support some languages is putting the cart before the horse.
2)Get a translator(s) that know what they are doing - You already needed a translator - now you need to pick one competent enough to know the difference between Mexican Spanish and the Spanish spoken in Spain.
3) Write software that can switch look and feel on the fly during run time - This has already be done and should be a good philosophy for companies writing GUI intensive applications anyway.
Really - The correct sentence should be, "Just use Unicode, with your pre-existing good business practices."
But honestly, out of all the problems you can have, that's a good one indicative of success.
1. Using variables to put numbers in the string. Singular and plural are not always only two options. Rules in this regard can get quite complex. gettext has this solved by using rules: n%2 == 0? Use A. n==0? Use B. n%2 == 1? Use C. etc. That should give you an idea about complexity.
2. Replacing words into the string is nearly impossible due to declensions.
3. Names. When using a different script, will the name be transliterated? Do the names need to be inflected? Will they be handled correctly when changing writing direction?
3a. Names have a whole load of problems anyway. The "given name" isn't always the first name. The surname isn't always the most important one. Some people don't have family names, some people have two or more. Some people have double-barrelled given names. Etc.
4. Ordinals differ greatly from language to language.
5. String length can mess up your layout. A good translator can help by using different idiom, but sometimes it's just not possible.
6. Layout expectations are different when the writing system is right-to-left. You also need to learn the details of Unicode's direction algorithm, because you will encounter situations where it doesn't work properly without guidance.
These are just some problems we've actually encountered when using multiple languages.
We use IDs and I've not seen too many of the problems you mention. I will, however, agree that the string length can cause serious problems for a layout. We tend towards using very flexible UI designs where the UI elements resize and reposition themselves a long with the strings. However, UI design isn't as stylish for embedded devices so we get let off easy here. One of the biggest problems we have with our newer developers is that they don't know how to write their UIs so that they can easily re-arrange themselves for our various languages. Also, the Left and Right layout changes are all part of our look and feel engine.
I wonder if some of these problems are specific to web applications and some of the 'fancier' UI designs that we don't use? I don't actually know enough about the differences between the two to say.
What letter does the name Van der Waals sort under?
What about D'Arcy
Does McKay come before MacKay ?
Is 7-4-09 an American holiday or a week after April fools?
In many non-Roman scripts the letters change form drastically based on if they are at the beginning, middle, or end. In some, the letters literally switch positions with other letters visually. Current operating system font support, Unicode standards, and fonts simply don't exist to support all of this.
Graphite (http://scripts.sil.org/cms/scripts/page.php?site_id=nrsi&...) is one great initiative to get the ball moving here and is gaining some support (being included as part of OpenOffice 3), however there are HUGE steps that have not yet been taken before applications can even think about scaling to 3000 languages. Unicode is great, but it's not a simple solution/answer/cure-all.
Hopefully we'll be able to scale that large one day, but luckily for now we only need to target < 1% of the world's languages.
How many customers will you have past your particular and local top 35 languages? Enough to warrant the translation and support and testing work inherent in adding another language?
And yes, the target market for and the list of languages used by your customer base(s) may or will differ from mine, too.
Please just use Unicode.
> Further east we hit ideographics. The concept of a "letter" has just flew out of the occidental window. Not only that but the text is laid out top to bottom.
Chinese and Japanese are both read from left to right (although in the past characters were written top to bottom and right to left). The only language that I know of that is presently laid out from top to bottom is Mongolian. I'll be damned if I ever need to write software for people in Mongolia.
"Both horizontal and vertical writing are used in Japan, as well as in Hong Kong, Macau and Taiwan where the Traditional Chinese orthography is used. Traditional Chinese is also used in mainland China in a few limited contexts, such as some books on ancient literature, or as an aesthetic choice for some signs on shops, temples, etc. In those contexts, both horizontal and vertical writing are used as well.
Vertical writing is commonly used for novels, newspapers, comics, and many other forms of writing. Because it goes downwards, vertical writing is invariably used on the spines of books. Some newspapers combine the two forms, using the vertical format for most articles but including some written horizontally, especially for headlines. Musical notation for some Japanese instruments such as the shakuhachi is written vertically."
http://en.wikipedia.org/wiki/Horizontal_and_vertical_writing...
But I agree that you don't really have to worry about that. Just as we're lucky not to have to worry about this monstrosity: http://en.wikipedia.org/wiki/Boustrophedon
Yes, you pointed out some relevant points, notably that vertical writing is still used in some contexts. However, it's a fact that Mandarin and Japanese are predominantly left to right on the web and on computers, which is what the original post was referring to.
PS The vertical writing on book spines is a life saver.
Yes, just thinking about having to worry about boustrophedon makes my head hurt.
But apparently it's faster to read, so it's not all bad ;-)
> PS The vertical writing on book spines is a life saver.
I can imagine. The constant twisted neck when searching for books in a library can't be healthy...
That said, top to bottom is just too much of a pain to deal with on the web until CSS supports it.
Not having gone through the cultural revolution, character simplification and various other imposed measures, top-down writing is much, much more common in Taiwan, and HK. The same is true to a lesser degree in Japan.
> Oh, and there are languages with unbounded sets of "characters" such as Chinese which literally cannot be fully described in Unicode.
I read Chinese just fine in Unicode. Is there something I'm missing? You have 50,000 commonly used characters, 100,000 rare ones - this certainly doesn't bust any limitations within unicode. Makes fonts rather large, but that's another issue altogether.
The rest of the article seems to be confused about what unicode is - I don't think anybody seriously goes into unicode expecting it to practically translate for them. It's an encoding, that's it. It won't convert "," to "." depending on locale, it certainly won't rephrase your messages for you to be sensitive to local cultures. That's your responsibility, not the purview of a form of encoding.
I think the author's point at the end was: "Unicode" as a concept won't localize your programs for you. Well, duh?
That's his point as I understand it. But I disagree with the "duh" part. It's rare to see software that goes beyond simple translation.
This is true. But SIL has what is considered the "standard" for this list, and there are almost 7000 of them. Many of them are unwritten. See http://www.ethnologue.com/ and ISO 639-3.
Who said that you need to support every language in a region to have your software operate there? If someone living in France speaks an obscure dialect then its a pretty safe bet they ALSO speak French. And if they only speak a single obscure language they probably aren't going to be your customer anyway.
If you hit the top 15-20 most used languages in the world then you'll be fine.
Just use unicode.
In fact, the PRC requires that all language-related products introduced into the Chinese marketplace must be able to function in GB 18030.
I don't see anything in it, at all, about character set and encoding choice. And in that realm, yes: you should just be using UTF-8. Not even "unicode" -- you want to be in UTF-8, period. Convert at the edges of your system if you absolutely must handle other encodings. But wherever you can, you should just be using UTF-8.
I'm curious as to why you believe UTF-8 is the only encoding you should use? I know of at least one person that says you should use UTF-16, because then it's immediately obvious when a certain piece of data is in the wrong encoding, whereas with UTF-8 you need to look for non-Latin characters (or accents) to distinguish it from, for example, ISO-8859-1. That line of thinking holds some value to me. Why do you believe otherwise?
The inability to distinguish between UTF-8 and ASCII is a feature, not a bug. If you want to know if it's a valid string, just validate it. UTF-8 has extraordinarily strong validation properties, it's almost impossible for strings in other encodings to parse successfully.
But they will have that same problem with UTF-8, right? Just as soon as you start using non-Latin characters? And if you aren't, there's no point in going beyond ASCII.
The sole exception are algorithms that truly need random access to characters within the string (e.g. "give me the 32nd character"), which can't interoperate with any variable-length encoding. But there are precious few of these. All anyone cares about is substrings.