IMO, it was a mistake for Unicode to provide multiple ways to represent 100% identical-looking characters. After all, ASCII doesn't have separate "c"s for "hard c" and "soft c".
This rabbit hole goes very, very deep. In Dutch, the digraph IJ is a single letter. In Swedish, V and W are considered the same letter for most purposes (watch out, people who are using the MySQL default utf8_swedish_ci collation). The Turkish dotless i (ı) in its lowercase form uppercases to a normal I, which then does _not_ lowercase back to a dotless i if you're just lowercasing naively without locale info. In Danish, the digraph aa is an alternate way of writing å (which sorts near the end of the alphabet). Hungarian has a whole bunch of bizarre di- and trigraphs IIRC. Try looking up the standard Unicode algorithm for doing case insensitive equality comparison by the way; it's one heck of a thing.
People somehow think that issues like these are only an issue with Han unification or something, but it's all over European languages as well. Comparing strings for equality is a deeply political issue.
Actually, there is just only one trihraph. "dzs" almost exclusively used for representing "j" from English and other alphabets, for example "Jennifer" is "Dzsennifer" in Hungarian or "jam" is "dzsem" in the same way.
Trigraph and digraphs actually make sense, at least as a native as these really mark similar sounds what you would think you will get by combining the given graphs. These letters doesn't cause too much issues in search in my opinion, but hyphenation is a form of art (see "magyar.ldf" for LaTeX as an example).
To complicate the situation even further we have a/á, e/é, i/í and o/ó/ü/ő and u/ú/ü/ű letters, all of those considered to be separate ones and you can easily type them in a Hungarian desktop keyboard. On the other hand, mobile virtual keyboards usually show a QWERTY/QWERTZ layout where you can only find "long vowels" by long pressing their "short" counterparts, so when you are targeting mobile users you maybe want to differentiate between "o" and "ö", but not between "o" and "ó" nor between "ö" and "ő".
Unicode shouldn't be responsible for making such searches work, just like it's not responsible for making searches for "analyze" match text that says "analyse".
The "proper" way of sorting and comparing Unicode strings is part of the standard; it's called the Unicode Collation Algorithm (https://unicode.org/reports/tr10/). It is unwieldy to say the least, but it is tuneable (see the "Tailoring" part) and can be used to implement o/ö equivalence if desired. I think it's great that this algorithm (and its accompanying Common Locale Data Repository) is in the standard and maintained by the consortium, because I definitely wouldn't want to maintain those myself.
So now we are saddled with an encoding that has to be bug compatible with any encoding ever designed before.
There's history here, with Unicode originally having just 65k characters, and hindsight is always 20/20, but I do wish there was a move towards deprecating all of this in favour of always using pre-composed.
Also: what you linked isn't "ASCII" and "extended ASCII" doesn't really mean anything. ASCII is a 7-bit character set with 128 characters, and there are dozens, if not hundreds, of 8-bit character sets with 256 characters. Both CP-1252 and ISO-8859-1 saw wide use for Latin alphabet text, but others saw wide use for text in other scripts. So if you give me a document and tell me "this is extended ASCII" then I still don't know how to read it and will have to trail-and-error it.
I don't think Unicode after U+007F is compatible with any specific character set? To be honest I never checked, and I don't see in what case that would be convenient. UTF-8 is only compatible with ASCII, not any specific "extended ASCII".
The pre-composed characters are necessary only for backwards compatibility.
It is completely unrealistic to expect that Unicode will ever provide all the pre-composed characters that have ever been used in the past or which will ever be desired in the future.
There are pre-composed characters that do not exist in Unicode because they have been very seldom used. Some of them may even be unused in any language right now, but they have been used in some languages in the past, e.g. in the 19th century, but then they have been replaced by orthographic reforms. Nevertheless, when you digitize and OCR some old book, you may want to keep its text as it was written originally, so you want the missing composed characters.
Another case that I have encountered where I needed composed characters not existing in Unicode was when choosing a more consistent transliteration for languages that do not use the Latin alphabet. Many such languages use quite bad transliteration systems, precisely because whoever designed them has attempted to use only whatever restricted character set was available at that time. By choosing appropriate composing characters it is possible to design improved transliterations.
I agree it's unlikely this will ever happen, but as far as I know there aren't really any serious technical barriers, and from purely a technical point of view it could be done if there was a desire to do so. There are plenty of rarely used codepoints in Unicode already, and while adding more is certainly an inconvenience, the status quo is also inconvenient, which is why we have one of those "wow, I just discovered Unicode normalisation!" (and variants thereof) posts on the front-page here every few months.
Your last paragraph can be summarize as "it makes it easier to innovate with new diacritics". This is actually an interesting point – in the past anyone could "just" write a new character and it may or may not get any uptake, just as anyone can "just" coin a new word. I've bemoaned this inability to innovate before. That is not inherent to Unicode but computerized alphabets in general, and I that composing characters alleviates at least some of that is probably the best reason I've heard for favouring compose characters.
I'm actually also okay with just using composing characters and deprecating the pre-composed forms. Overall I feel that pre-composed is probably better, partly because that's what most text currently uses and partly because it's simpler, but that's the lesser issue – the more important one that it would be nice to move towards "one obviously canonical" form that everything uses.
Many of the existing typefaces, even some that are quite expensive, do not contain all the pre-composed characters defined by Unicode, especially when those characters have been added in more recent Unicode versions or when they are used only in languages that are not Western European.
The missing characters can be synthesized with composing characters. The alternatives, which are to use a font editor to add characters to the typeface or to buy another more complete and more expensive version of the typeface, are not acceptable or even possible for most users.
Therefore the fact that Unicode has defined composing characters is quite useful in such cases.
The ‘early’ Unicode alphabetic code blocks came from ISO 8859 encodings¹, e.g. the Unicode Cyrillic block follows ISO 8859-5, the Greek and Coptic block follows ISO 8859-7, etc.
But it does, IIRC, for both Bengali and Telugu.
But the English letter "c" and the Russian letter "с" are completely different characters, even if at a glance they look the same - they make completely different sounds, and are different letters. It would be ludicrous to suggest that they should share a single symbol.
1. Unicode isn't a method of storing pixel or graphic representations of writing systems; it's meant to store text, regardless of how similar certain characters look.
2. What do you do about screen readers & the like? If it encounters something that looks like a little half-moon glyph that's in the middle of a sentence about foreign alphabets that reads "Por ejemplo, la letra 'c'", should it pronounce it as the English "see" or as Russian "ess"?
I'm not sure that that is really possible without something way bigger or more complicated than Unicode. Consider the string "fart". In English that means to emit gas from the anus. In Swedish it means speed. Does that mean Unicode should have separate "f", "a", "r", and "t" for English and Swedish?
> 2. What do you do about screen readers & the like? If it encounters something that looks like a little half-moon glyph that's in the middle of a sentence about foreign alphabets that reads "Por ejemplo, la letra 'c'", should it pronounce it as the English "see" or as Russian "ess"?
What would a human do if that was in a book and they were reading it aloud for a blind friend?
(IIRC, she learned the language entirely from books so has no idea of the correct pronunciation and thinks she's fluent)
2. I think the pronunciation should not be encoded into the text representation on a general scale. You would need different encodings for "though" and "through" in english alone. Your example leaves the meaning open, even if being read as text. If I was the editor, and the distinction was important, I'd change it to "For example, the cyrillic letter 'c'".
I understand that Unicode provides different code points for same-looking characters, mostly because of history, where these characters came from different code sheets in language-specific encodings.
“Cyrillic” isn't the same everywhere. Bulgarian fonts differ from Russian fonts, some letters are “latinized”, some borrow from handwritten forms:
https://bg.wikipedia.org/wiki/Българска_кирилица
Colored example has the third alternative for Serbian cursive.
So without some external lang metadata we don't even know how your message should look.
However, Russian “Кк” traditionally is different from Latin “Kk” in most recognized families. In the '90s, font designers regularly thrashed ad-hoc font localization attempts which ignored the legacy of pre-digital era, and blindly copied the Latin capital into capital and minuscule forms.
It was a joke, by the way.
Or the whole eh/ye flip En/UK/Ru Eh/е/э Ye/є/е
г/е are unified and that's probably as it should be but there are downsides.