"one code point in unicode does not necessarily map to one character on the screen."
A "character" can, and often does, get constructed from multiple code points.
This doesn't help an already complicated sorting issue (who knew "sort these names alphabetically" could be an ambiguous statement).
Also, importantly, some characters are not represented at all. For example, there is no codepoint or combination of characters that distinguishes a capital Turkish dotless I from its identical looking but conceptually different sibling the Latin capital I. Similarly for capital Turkish dotted İ.
I find it extremely weird that two codepoints couldn't have been spared, yet we have gradations of skin tone in emoji. One can use composition to deal with the round-trip from i -> İ -> i (within a closed ecosystem), but even that fails when it comes to ı -> I -> ı.
A case in point are the product pages for my friend's book on Amazon. Compare "Sınır Ötesi" to "Sinir Ötesi". The former means "beyond borders" whereas the latter means "exceedingly irritating".
Yet, because there is no unambiguous representation of the Turkish I's, the rendering makes an assumption on the basis of the domain. Note that all other non-US ASCII letters involved are rendered correctly.
\c[PERSON FROWNING, ZERO WIDTH JOINER, PROGRAMMER]
[tr]: https://www.amazon.com.tr/Gezging%C3%B6z-S%C4%B1n%C4%B1r-T%C...
[us]: https://www.amazon.com/Gezging%C3%B6z-Sinir-T%C3%BCrkiye-Mir...
The Turkish alphabet ISO 8859-9 was published in 1988, and it doesn't distinguish between I and I, since why would it? It's a byte encoding designed specifically to accommodate Turkish.
Unicode has a principle of adopting whatever makes it easy to convert from the preferred legacy encoding to Unicode, and for Turkish that meant using the same code point for I in both Turkish and practically everywhere else, since I is in ASCII and the bulk of Latin-based character encodings are based on ASCII.
I point this out because it could indeed have been the case that they ended up with separate encodings, which would have solved your problem.
But it wouldn't have solved the actual problem, which is the collation and casing are locale-specific according to Unicode, and indeed they must be. Example: in Dutch the letters 'ij' are one letter for collation purposes, as well as for casing, see https://en.wikipedia.org/wiki/IJsselstein for an example.
This is of course impossible to get correct if the application doesn't know it's dealing with Dutch, which it often can't know. The very same problem you have with Turkish.
Not really the same. The brokenness of more complex text transformations (collation, alphabetization, upper-casing, lower-casing) seems a bit less catastrophic than breaking the fundamental ability to unambiguously draw the character.
If you don't think that's an important goal.... then what are "character sets" for anyway?
This may not be a solved problem in a given programming language's standard library, and/or one might see a bug such as the one GP was reporting, but neither of these conditions is inevitable, here's how you do it in Swift:
"i".uppercased(with: Locale(identifier: "tr_TR")) // returns "İ"
"i".uppercased(with: Locale(identifier: "en_US")) // returns "I"
It's not a bug that Unicode just threw its hands up and said that complex string operations are locale-specific. The more you learn about the various scripts it represents, the more you will appreciate that there is no other choice.
Hopefully this makes what I meant a bit more clear.
The current state of affairs categorically rules that out it contrary to the situation with any other language.
This is another question which doesn't belong in the Unicode character specification, but should be resolved when necessary at a higher level of abstraction.
People shouldn’t expect to know what language a Unicode string is in without external help, even if heuristics work often.
And in this case Unicode fails catastrophically by not providing codepoints for the uppercase of "ı" and the lowercase of "İ" while potentially allowing for ethnically sensitive renderings of shit.
That historic process does not preclude introducing two extra codepoints to at least allow for the possibility of correctly encoding the upper case of "ı" and the lower case of "İ".
The past isn't something you can argue with, no more than the wind.
That already is ambiguous without considering text encodings or even computers.
Sort order depends on locale (example, “in Lithuanian, "y" is sorted between "i" and "k"”) and within some locales even on the goal of the sorting (example: “in German dictionaries, "öf" would come before "of". In phone books the situation is the exact opposite.”
(Both examples taken from https://unicode-org.github.io/icu/userguide/collation/, which has many more examples)
Also, on that “one code point in unicode does not necessarily map to one character on the screen”, I think it’s important to use consistent terminology. The screen doesn’t show characters but glyphs.
Thus alphabetical order becomes ambiguous depending on where you are.
_then_ you discover some symbol sets / languages don't have an order at all. Oh the world is complicated. I wish I was 4 again.
Like "the" word, or person with some titles... While ofc. the title itself is sorted in that position...
I am only aware of “of” before “öf”, e.g. in Grimmʼs dictionary, the greatest dictionary of German itself, “offen” ‘open (adj.), openly’ comes before “öffen”, a variant of “öffnen” ‘to open’.
When I have to sort strings in German, I make a sortstring by throwing out diacritics (via Unicode Normal Form “K”anonical Decomposition) and changing “ß” to “s”. Then I order by a tuple of this sortstring and the original string.
What you infer about sorting is better described as collation and is a separate set of problems. https://www.unicode.org/collation/
There can be multiple collations for the same encoding and even for the same language inside the same encoding.
PS: As a matter of fact in some database like PostgreSQL you can define different collations for each columns if needed.
I did a quick check and saw that I can specify Hungarian in MySQL as a collation for a table and I also know that it's available in Java and even JavaScript.
⸻
1. This sorting is wrong not just for Hungarian but for every language including English which would expect, e.g., naïve to be sorted between nag and nanny and not after nay.
the second character of sœur (o or the ligature?)
the second character of حبيبي (the canonical form ب or the contextual form ﺒ ?)
the third character of есть (Cyrillic t with or without soft sign, which is always a separate code point and always displayed to the right but changes the sound?)
the first character of 실례합니다 (Korean phoneme or syllabic grapheme?)
the first character of ﷺ or ﷽ ?
The main issue isn't programming language support, it's ambiguity in the concept of "character" and conventions about how languages treat them. Imagine how the letter i would be treated if Unicode were invented by a Turk. The fundamental issue here is that human communication is deeply nuanced in a way that does not lend itself well to systematic encoding or fast/naive algorithms on that encoding. Even in the plain ASCII range it's impossible to know how to render a word like "ANSCHLUSS" in lower case (or how many 'characters' such a word would have) without knowledge of the language, country of origin, and time period in which the word was written.
All of this already standardised in detail in Unicode. There is no ambiguity. Read it some time to mend your ignorance, it answers all your questions with justification, I am willing to spend time on the answers only.
> the second character of sœur (o or the ligature?)
the letter œ, both character (C) and grapheme cluster (GC)
the letter o does not exist in this text (again, according to the standard)
> the second character of حبيبي (the canonical form ب or the contextual form ﺒ ?)
the letter ب, both C and GC
the letter ﺒ is a presentation form and does not exist in this text
> the third character of есть (Cyrillic t with or without soft sign
the letter т, both C and GC
the letter ь, both C and GC, is the 4th in the text
> the first character of 실례합니다 (Korean phoneme or syllabic grapheme?)
the letter 실, both C and GC
the text does not contain the jamo ᄉ
Unicode is not concerned with phonemes, only with writing
> the first character of ﷺ
is entirely itself, the letter ﷺ, both C and GC
> or ﷽ ?
is entirely itself, the symbol ﷽, both C and GC
Just say "I don't have the faintest idea how to do this properly". It would be much less of a problem if people would admit they don't know how and need somebody else's help than it is when they insist it "doesn't really matter" as you have and produce garbage that doesn't work but insist that's just because everybody is holding it wrong.