How much of this is still a problem with modern software/font stacks and harfbuzz?
How much of this is still a problem with modern software/font stacks and harfbuzz?
This repository has a good outlook: https://github.com/harfbuzz/harfbuzz-wasm-examples
> The inflexibility persisted and has arguably only become more aggravated in the 20th century
What about 21th century? Digital printing can overlap characters just fine. And modern fonts support context sensitive ligatures and glyph substitutions.
Second/third example those seemed to be caused by more by someone who doesn't understand the language copy pasting stuff.
PDF -> that's just PDF being bad. Text and text search in PDFs tends to mes up even or English.
> with unicode number U+0623, but one can also type أ, which is an alif and a high hamza, represented by unicode numbers U+0627 and U+0654.
That's what Unicode normalization and locale settings are for. Same thing applies to large fraction of latin based scripts other than English, anything which has letters with diacritic marks.
> for كثيره and كثيرة will in most cases yield different results
Similar thing in almost any non English language for example cafe and café or ABC and ⒶⒷⒸ. Although at least some systems handle it reasonably. Not sure how much it is heuristics based on large data (hard to scale across software), and how much it's good application of Unicode character decomposition/normal form tables. Which Arabic letters lack appropriate Unicode decomposition (and other) tables and what are the best practices of unicode normalization/decomposition/locale handling for search (applicable for all languages) are more interesting and actionable topics.
> Not even the simple idea of CJK has been implemented.
Many users of CJK language would argue that CJK unification was a mistake. If different languages prefer different forms of the glyph, they should better be separate characters. Having separate Chinese and Japanese fonts because CJK unified too much just introduces additional points of failure.
Luckily it's not a decision without turning back. In most relevant contexts you should know the input language and can select a Font specifically using said variations. Of course this information will not be present in plain text, but if it turns out to become an issue I'd wager, since language codes do exist, that a control code-point for language selection can be added to the specification. There's already so many special cases in Unicode that it shouldn't be a huge issue (apart from backwards-incompatibility that would lead to tofu instead of no rendered glyph).