Hacker News Reader

Comment by corty | Hacker News Reader

That is insufficient and wrong. "Apfel" should not match "Äpfel" in a search, because those are different words, one is the singular, the other the plural of "apple". However, "Aepfel" should match "Äpfel", since those are the same word, the former is an accepted replacement form. Similarly, "ss" and "sz" (very rare) should match "ß" and vice versa, so "Strasse" is the same as "Strasze" and "Straße".

Of course all of this is locale-dependent and it might be acceptable for ease of use to match "ä" to "a" for international users. But not in a german locale.

Proper I18N is very hard, and supporting Unicode is just a first step. Converting text between lower/upper/title-case can only be done knowing at least the locale of the text, and in rare cases a case-folding-roundtrip can even change the meaning of the text, so you will have to be able to speak the language.

View on HN

> Similarly, "ss" and "sz" (very rare) should match "ß" and vice versa, so "Strasse" is the same as "Strasze" and "Straße".

This is unworkable, since "Busse" (several autobuses) should not match "Buße" (repentance). But "Busse" (incorrect spelling of "Buße" due to character set limitations/Swiss German spelling I believe) should match "Buße" by your argument.

Anyway, modern Firefox lets you choose whether to match "Apfel" against "Äpfel".

You are right in that my argument is somewhat incomplete, one could be stricter and not match "Buße" when searching for "Busse" because that would be a somewhat uncommon almost-misspelling in German German.

But: "Busse" is not a always incorrect spelling, it's just an "emergency" one, if you really can't use "ß" for some reason. One example is allcaps text, here "BUSSE" and "BUSSE" are not distinguishable, although they have different meanings. Also, there is Swiss German, where replacing ß with ss is the normal form and not at all "accepted in an emergency" like in German German: https://www.galaxus.de/de/page/schweizerhochdeutsch-fuer-anf...

Incidentially this is one of the instances where you really have to know the language and the context to be able to do a lowercase -> uppercase -> lowercase roundtrip, otherwise you might screw up with "Buße -> BUSSE -> Busse", changing the meaning. Or not, if the current locale is de_CH instead of de_DE.

Together, since very often search will be case-insensitive, I think that while you may be correct, being as strict as to not match here would not be what the user would expect.

Yes, again, I18N is hard and sometimes it is impossible to do correctly for a machine.

Text search implementations do not have the luxury of assuming that the text they are used against us correctly spelled. Indeed one common usecase for find and replace is searching for misspellings.

Well, I think now the Germans invented a uppercase "ß" letter (in 2017), although there didn't used to be.

Yes, I know. Just more headaches, because you have one more equivalent variant to worry about (use of ẞ is correct but very rare), and fonts that do not support the new character ẞ yet or ever (so it will not be standard capitalisation for quite some time, if ever).

Even in a German locale, it doesn't work. "Müller" and "Mueller" are two different names, there is no band called "Moetley Cruee", etc. Also, "sz" hasn't been a replacement for "ß" since 1901.

So maybe in addition to locale-dependent behavior there has to be a time-dependent one :-)

Or, even better, the "find" feature should support multiple user-specified options about what to care about in the search, like is done for the "match case" option. Simply allowing regex covers most issues.

I agree about regex, it is a good idea to have. I would much rather that Firefox's search function supports regex rather than the incremental search it has, because I think regex would be useful.

But, regex is a separate issue than languages of text.

Or perhaps better, weighted finite state transducers, since some errors are more likely than others.

> Similarly, "ss" and "sz" (very rare) should match "ß" and vice versa,

Szene should match ß?

That again is a special case: Sz is never ß at the start of a word... Since there are composite words in German, you have to pay attention to "Schlusssszene" being neither "Schlusßzene" nor "Schlussßene" but "Schlußszene". But I would not blame anyone or any program for getting this wrong.

Also, in this case, Sz is pronounced as the two distinct letters it is composed of. Whereas when "sz" replaces "ß", the pronounciation is just very similar to a plain "s".

That's the same point though - it's just as "wrong" to match 'ß' twice in Schlu[ß][sz]ene as it is to match 'a' once in [Ä]mpel.

Admittedly German isn't my first language. But it seems odd to be permissive in one case, but more restrictive in the other. In my experience there are two cases for searching: exact matches, like you'd expect an editor's find and replace, and more permissive searches, where you might expect to skip through several matches to find what you're looking for. In this second case, what's the advantage of being more exclusive in one case, then less exclusive in the other? Why not have a more permissive default (as mentioned by the other user's reply regarding Firefox's configurable search), even if it's technically incorrect?

Because the ß == ss case is special in that there are numerous words that can be written both ways correctly, there are numerous words that can be written both ways sometimes and often the meaning depends on whether the most common form is written with ss or ß. And Swiss German allows replacing ss for ß as a normal way of writing everything.

Whereas for umlauts ä, ö and ü, the forms with ae, oe and ue are always "emergency replacements", even the Swiss use äöü very frequently (though not always). Also, there are almost never ambiguities when replacing ä, ö, ü -> ae, oe, ue. Yet there are very frequent ambiguities when replacing ä, ö, ü -> a, o, u, because singular/plural, conjunctive and other forms are derived that way and sometimes there are just different words that only differ in one vowel being an umlaut.

So there is a difference in the chance of being wrong when picking one form of a word for another possibly equivalent one. When you do ss == ß, you are often correct. When you do ä, ö, ü == ae, oe, ue, you are almost always correct. When you do ä, ö, ü == a, o, u, you are almost always wrong.

> Swiss German allows replacing ss for ß

requires, not just "allows".

> even the Swiss use äöü very frequently

I can't think of any case where Germans would use one of these and German speaking Swiss would not. But as another Swiss peculiarity, some of our words contain "üe", e.g. "Üetliberg", "gmüetlich".

Reply on news.ycombinator.com