The pitfalls of unicode sorting (with Perl examples)

I think that'd make a better URL for the main submission. Editor?

I have a file we give our new (Indian) engineers prior to letting them work on projects for our (Japanese) clients.

It is called blue-pill.doc . (The official Japanese title of a similar document that existed a few years ago was Welcome To The Real World. I got to rename it when I was put in charge of it, and I like crosscultural puns.)

Bad news: while the number and identity of sounds in the Japanese language is widely agreed upon, there are two separate orderings on them which are in wide use, plus the "ABC order on the romanization" alternative which we see a bit of. (There are multiple plausible romanizations for many words, but we'll skip that discussion.)

Bad news: it is pretty hard to determine the pronunciation of a significant portion of Japanese algorithmically. You have to use a lookup table.

Worse news: there are multiple possible pronunciations of many Japanese words. It changes based on the context. This results in changing their lexical sort order... some of the time.

Still worse news: chunking natural Japanese text into individual words so that you can look them up is pretty hard, too. (For a taste of the fun, take any English sample document and remove all the whitespace, then tell me how you're going to pick words out.)

Bonus: Everything you know about Japanese lexical sorting is wrong for prefectures. The proper lexical sort is in order of geographic location, north to south. Don't ask why. It just is. Except when it isn't.

(Incidentally: I don't know what other companies do, but here, we put a display order column in all tables that routinely have contents displayed in order, and set the order manually. For things like names which customers can edit by themselves, we have them spell out the pronunciation as well if they want to do lexical sorting on it, otherwise we return the results in arbitrary order (on basis of code points, technically, but good luck explaining that to a non-technical person).

Why oh why did I not take the red pill. :)

Oh sure. I submitted this story for two reasons - (1) the interesting technicalities regarding language-to-language (or even dialect to dialect) variations in sorting rules and (2) the awesome thoroughness of the reply to a seemingly minor question. tchrist can be quite scary, but this was beyond the call of duty.