Hacker News Reader

Comment by masklinn | Hacker News Reader

The semantic impact of diacritics is locale-dependent, some languages treat accented characters as a variant of the base (and thus with the same identity) while others treat them as separate letters.

So while the german or french alphabets are purely latin (the diacritics are not part of the alphabet), that is very much not the case of the 29 letters Turkish alphabet (where accented characters are separate but following the un accented form), or the Icelandic alphabet (where ö does not sort with o and ó), or the Norwegian alphabet (where IIRC ø and å sort at the end of the alphabet, nowhere near o or a).

To say nothing of Welsh, where “ng” is a digraph sorting between “g” and “h” (although I don’t think any welsh word starts with ng so it probably doesn’t matter).

View on HN

It is not just locale dependent, but may also depend on the type of list you are sorting. So while German dictionaries ignore the umlauts as claimed, German phone books sort names with umlauts according their non-umlaut transcription (ä = ae, ö = oe, ü = ue). And I own an old encyclopedia that goes the other way and sorts words that are conventionally written with the non-umlaut transcription as if they were written with an umlaut, which is then ignored.

Dictionary: Goethe, Görlitz, Gotha, Gotik

Phone book: Görlitz, Goethe, Gotha, Gotik

Encyclopedia: Görlitz, Gotha, Goethe, Gotik

And, oh, yes, "ß" sorts right before "ss" and after "sr".

> dictionaries ignore the umlauts

almost, but not quite, since you can have words that only differ by umlaut; in that case, the one with the umlaut should sort after the one without one, so basically your alphabet now is a, ä, b, c, ... l, m, n, o, ö, p, ..., z. This btw is how modern Japanese dictionaries are sorted, too, where two entries with the only difference of a dakuten (as in しゅ, じゅ) are sorted as shown here.

I guess they also sort loan words starting with “ng” with “n” (example: https://en.wikipedia.org/wiki/Bhutanese_ngultrum)

Sorting also is context-dependent. https://unicode.org/reports/tr10/#Introduction: “even within the same language, dictionaries may sort differently than phonebooks or book indices”

Weird example (https://unicode.org/reports/tr10/#Contextual_Sensitivity): “Normally, all differences in sorting are assessed from the start to the end of the string. If all of the base letters are the same, the first accent difference determines the final order. %[…] In some French dictionary ordering traditions, however, it is the last accent difference that determines the order”

(That entire page is worth reading if you want to know about collation)

In Dutch the ij is a digraph. The dictionary sorts words starting with as an i followed by j, though many other places like the phone book sort it together with y.

(... after my boss told me he checked our submissions of free time by simply making a search for the word "vrij" in Slack, [meaning "free"] I started to use the unicode digraph ĳ to try and cheat the system.)

Reply on news.ycombinator.com