Truths programmers should know about case (2018)

We're at the point that it might make sense to map languages to a digital format rather than map an extensible digital format to myriad different languages. I mean, it will never happen, but if we had a much simpler format than Unicode that just supported a number of characters, but no combining characters, no accents (and perhaps even no case), and let each language decide how they wanted to represent themselves within that medium (are accents important enough to require a separate character, or is leaving it off or just using two characters next to each other sufficient?).

That isn't to say we wouldn't want accurate representations of languages (as Unicode makes an attempt to do), but there's a difference between trying to represent a language, and trying to represent the data within a language. In some cases, the former is important, in others, the latter vastly outweighs the former, and making that easily accessible, categorized, compared, etc has value.

I think it's only even conceivable at this point because of how much communication is done online, and in abridged form. We've already adopted pidgin languages for ad-hoc communication (as in IMs, SMS, and tweets), so I don't think people would find this that hard to grapple with at this point.

No. Those 3 points are not that important. West Europe works using ISO-Latin-1. Many European countries even had a national 7-bit version. We also had our own computer companies, but they died out because they couldn't grow.

The 2 main reason are:

1) you have one Sillicon Valley in USA, so all competent people were in one location. If you wanted to start Intel/Apple/Google in Denmark you'll run out of Wozniaks even before you got going because they are spread out all over the map.

2) All of USA speaks the same language, so you can easily do complex communication with companies in other states (e.g. Intel/Gateway in Texas). Getting a French person to speak good English in the 60-70'ies was a, ahem, challenge.

There is no Sillicon Valley in Europe precisly because there is no single place, and therefore Europe miss out on the "everybody important in same place" effects.

China's government pointed to one place at the map when they wanted a SV. That's why China got Shenzhen. Europe/EU still can't do that politically.

There was once a great post on ready comparing home computers with Japanese counter parts, and how much more expensive they were due to extra memory to hold the alphabet, as well as how unnatural typing was (could only type katakana and latin, still needed more memory than the US version - those kind of things)

The authors conclusion was that this turned Japan into a single-use device nation (because a nintendo doesn't need you to type) and the west much more focused on PC's. I think he even went so far as to say the iPod couldn't have been invented in Japan, because it was too dependant on a PC to get music onto it (the article was pre-iphone, or around that time)

> It is relatively easy to do a case-insensitive comparison.

Case-insensitive comparison only became a relevant concept in informatics because it was easy on the countries that advanced on it first.

> With the exception of numerical strings, sorting and ordering is relatively straightforward.

That's another one inverting causation. If informatics developed on a place where different kinds of sorting made sense, we would be using those different kinds.

I do think your point about the small alphabet stands. But keep in mind that it applies to all languages that took influence from the ancient Greek, what is basically only excludes South-Eastern Asia.

That is the topic of the very last part of the article -- on case folding.

No, not without additional information. You can't tell whether a name that says i is equivalent to one that says I without knowing more about where the names come from. And in Japanese even two identical codepoint sequences might represent different names (kanji with multiple readings).

The part "Case-insensitive comparison requires case folding" says there is an algo for that, but it's up to the language to implement it.

E.G: in Python 3, it's recommend to use str.casefold() instead of str.lower() for this:

    >>> "Straße".upper().lower()
    'strasse'
    >>> "Straße".lower()
    'straße'
    >>> "Straße".casefold()
    'strasse'
    >>> 

Depending of the context, a little str.strip() and/or a str.split() + str.join() may also help. User usually don't want to consider blanks when entering data (e.g: an address), while it's important for machines (e.g: a password).

Now like @lmm said, this only normalize the codepoints. Meaning is of course, impossible to be sure that way.

If you mean human names, this is fundamentally impossible no matter what you do because of things like "MᶜDonald" vs "McDonald" vs "MacDonald". If you mean user or variable names, you should probably just treat them as case sensitive.

And in french we keep them:

étude -> Étude français -> FRANÇAIS

While it was customary to omit the accent in uppercase letters due to technical limitations, this is not (and never has been) accepted by the Royal Academy:

http://www.rae.es/consultas/tilde-en-las-mayusculas (link in Spanish).

Personally (I'm a nobody that doesn't always agree with the Academy) I don't like omitting the accents at all in any circumstances.