Hacker News Reader

Comment by varenc | Hacker News Reader

Another fun fact: Upper-casing is language dependent. In English uppercasing 'i' gets you 'I'. But Turkish has a dotted and un-dotted 'i', each with an uppercase variant. So if your user's language was Turkish, uppercasing 'i' would give you 'İ', and lowercasing 'I' would give you 'ı'.

Makes me wonder how case insensitive file systems handle this...and for more fun, handle the situation where the user changes the system language. I know that the Turkish 'I' delayed at least one big company's Turkish localization efforts for awhile.

View on HN

> Makes me wonder how case insensitive file systems handle this...

They generally don't. It is true that several case-insensitive file systems (including NTFS, exFAT and ext4 [1]) maintain some sort of configurable case-folding maps but they are mostly used to guard against the future Unicode update and do not vary across locales.

[1] https://dfir.ru/2021/07/15/playing-with-case-insensitive-fil... (ext4 supports an optional case insensitivity)

Another example is that in Dutch, the bigram 'ij' is considered a single letter, and so at the beginning of a word, both have to be uppercased. See for example the Dutch Wikipedia page for Iceland: https://nl.wikipedia.org/wiki/IJsland.

And it's present in Unicode also! Ĳ and ĳ are single characters (try selecting only the i or only the j in those). Their use is discouraged though:

> They are included for compatibility and round-trip convertibility with legacy encodings, but their use is discouraged. Therefore, even with Unicode available, it is recommended to encode ij as two separate letters.

A somewhat surprising and interesting side effect of this can be found in the blog post "Hacking GitHub with Unicode's dotless i" [1], which is now fairly well known.

[1] https://eng.getwisdom.io/hacking-github-with-unicode-dotless...

>Upper-casing is language dependent.

It goes beyond just language... uppercasing can also be locale-dependent. In Microsoft Word, for example, uppercasing an "é" gets you "É" in the French (Canada) locale but "E" in the French (France) locale.

My understanding is that case insensitive filesystems that wish to be portable have a mapping table in the metadata. A quick search showed references to an 'upcase table', although I'm not sure of the accuracy of the source, so I won't link it.

Just because the user changed the system language doesn't mean the system should be expected to change the upcase table though. That operation would need to be very carefully managed; you can't change the rules if there are existing files that would conflict in the new mapping. And you might have symbolic links that matched because of case insensitivity that won't anymore... Pretty tricky.

I think this goes to show that the file system is not the right place to implement natural language handling to its full extent, like it's probably not a good idea to implement time zones as a primitive function in your new programming language.

What is the right place to have two different systems agree on natural language handling, then?

And of course there are widely recognized problems with time zones too.

Uppercasing is also language dependent in the trivial sense that most scripts in use have no concept of "upper" and "lower" case at all.

Wouldn't the 2 Is be different characters than the ASCII I and therefore each be subject to different capitilazation rules?

This could've been a solution but is not what Unicode chose to do (in this particular case). Also relevant: discussion around unifying glyphs like A, B, C, E and so on across Cyrillic, Latin, and Greek, all with their language-specific casing behavior.

Coincidentally, Cyrillic I/i is also a thing (and is separately encoded!):

https://en.wikipedia.org/wiki/Dotted_I_(Cyrillic)#Computing_...

That's correct, but this could be very surprising for anyone not familiar with Turkish. For example if you have an English text which has a Turkish word in it, maybe a place or person name. Or for example when doing OCR.

Reply on news.ycombinator.com