Comment by adontz | Hacker News Reader

Comment by adontz | Hacker News Reader

I am not sure if native speakers are involved in Unicode standardization at all.

Personal example.

In Georgian (a few millions native speakers) we had historically three different independent alphabets: mtavruli, nuskhuri and mkhedruli. All represent same letters, but different ways to write them. Alphabets are represented in Unicode under different code points. Only mkhedruli is used currently (last few centuries) and therefore most fonts have glyphs for mkhedruli only.

Also there is no concept of capitalization in Georgian, there is no "a" and "A", we have only "ა", one does not start a sentence, a word or a personal name with some variant of a letter. Anna is ანნა in Georgian, the first and the last letters are written the same way.

Now, because of some reasons, letters of mtavruli alphabet are registered as uppercase versions of letters of mkhedruli alphabet. Mind blown.

And since well written software uses rules defined by Unicode standard, via ICU library or some platform specific routines, many desktop and mobile applications as well as major web-sites are hard to use. First letters of sentences are capitalized, but since mtavruli glyphs are missing from fonts, they are rendered as ⍰.

So instead of

მაგათი დედაც ვატირე.

we see

⍰აგათი დედაც ვატირე.

and have to guess what was the first letter. But who cares, if it is not used in U.S. right?

> I am not sure if native speakers are involved in Unicode standardization at all.

Definitely. For Georgian alphabets that were one of the original scripts included in Unicode 1.0, as an example, the reference fonts were designed by Georgian type designers Anton Dumbadze and Irakli Garibashvili. If something was very off they must have noticed that.

Doing a bit of research, this mkhedruli/mtavruli case relationship was only added in Unicode 11, which means you can go and read the documents you used to justify it.

And it seems that the Georgians themselves were the ones who pushed for this change: https://unicode.org/L2/L2016/16034-n4707-georgian.pdf. It seems this isn't a case of "ignorant Westerners screwing up Unicode" but "ignorant programmers who don't understand how Unicode works screwing it up," as noted in the sibling comment.

Or "ignorant programmers who don't understand how text works or can't bother to support it properly getting to decide character encoding standards".

We have English speaking programmers swearing by ASCII despite it not having all the characters needed for English (I can't find it now but there was a report by the Library of Congress that came to that conclusion). I suspect their Georgian counterparts have similar attitudes towards text.

> not having all the characters needed for English

I've heard this argument a few times, and each time it's just as contrived, particularly if you stick to American English. While I'm happy to acknowledge it's not good enough for other languages, ASCII is plenty good for English. Unless you're trying to type cafe with an accented e to be fancy or type that old conjoined-ae character, ASCII is fine for everyday usage. See the fact that most Americans have never installed a separate keyboard layout.

The 'long s' (ſ) was still in fairly common use in English less than 200 years ago. Consequently there are a lot of historically relevant documents written in English that can't be faithfully reproduced using ASCII. This may not be relevant to most people, but it seems plausibly relevant to libraries or archivists (for instance the Library of Congress.)

Example: https://upload.wikimedia.org/wikipedia/commons/7/79/Bill_of_...

And that's exactly why Unicode was invented.

Funny you mention American English given the fact that a great many Americans have various diacritic marks in their names. You can make the claim that those names are "foreign" but the most common ancestry group in the US is German.

Regardless of the technical arguments, from the perspective of non-programmers, when they see things like the CA DMV being incapabale of doing things as simple as recording a diacritic over a person's name, it doesn't reflect well on the software industry and programmers as a professsion.

> You can make the claim that those names are "foreign" but the most common ancestry group in the US is German.

While true, I think the overwhelming majority of German-Americans anglicized their names several generations ago. For instance, my great-great grandfather switched from Müller to Miller. This is probably less true of more recent arrivals.

It's naïve to assume that American English does not use diacriticals.

I newer saw correctly spelled name of Kyïv (Kyiv, Kiev) city in English texts.

Kyiv is the official form according to the Ukraine government/mapping agency (according to Wikipedia).

The most common English spelling is Kiev. That reflects the English pronunciation. (The British one anyway.)

But many significant European cities with a long history have a different name in English than the local languages: Cologne, Munich, Florence, Naples, Warsaw, Copenhagen, Gothenburg, Prague.

This isn't just Anglophones being boorish and Anglocentric, either. Those cities also have different names in German or, more obviously, Chinese.

Indeed, and naturally British and Irish cities have different names in other (European) languages.

In French: Douvres, Londres, Édimbourg.

In German: Edinburg.

In Italian: Edimburgo, Glascovia, Londra, Novocastro, Dublino.

In Chinese: Jiànqiáo (剑桥) (lit. "sword bridge", Cambridge)

Have you ever wondered why Russians are indifferent to the fact that Moskva is spelled Moscow in English?

Because we don’t really care.

I'd hate to work for the New Yorker without ö on the keyboard, but for 99.99% of American usage ASCII is plenty good enough.

ASCII doesn't even have proper quotation marks or dashes. There's a reason software for text production has long made it easy to use characters outside of the ASCII range.

You're talking past OP. Most people don't care about matching quotations or dashes.

Most people will care about something not in ASCII.

Windows-1252 encoding was supported on American computers long before Unicode. I remember “smart quotes” being promoted as a feature of word processors.

Even for people who didn't care for the quotes:

• everyone appreciates ¢, ° for °F or 46° N, ¼, ½, ¾, × and ÷,

• people handling large texts want §, †, ‡ and maybe ¶,

• businesses want ©, ® and ™,

• scientists and engineers need µ, ±, ², ³, ·.

Web browsers have made it more difficult for the average user to enter these characters, but software like Microsoft Word has supported easy ways to insert many of them automatically.

> While I'm happy to acknowledge it's not good enough for other languages, ASCII is plenty good for English.

It’s good enough for the style of American English adopted as a concession to typewriter limitations, since that’s pretty much exactly what ASCII is designed to support. OTOH, American English as used outside of that context doesn’t use straight quotes, but rounded ones in each direction with different meaning, uses various diacritical marks, and also uses the cent sign (¢), the obelisk (†), diesis (‡), pilcrow (¶), section sign (§), division sign (÷), multiplication sign (×), less-(and greater-)than-or-equals sign (≤ and ≥), angle brackets (⟨ and ⟩, which aren’t the same as less-than/greater-than symbols), the interrobang (‽), and—the next three distinct from the hyphen—the minus sign, en-dash, and em-dash.

> See the fact that most Americans have never installed a separate keyboard layout.

Common software used for text, like MS-Word, provides convenience methods for many non-ASCII characters without an keyboard layout change.

Some operating systems extend this to work across all apps.

Since I know that macOS conveniently allows to type en and em dashes (option+hyphen and shift+option+hyphen respectively) I'm using both frequently in the appropriate situations instead of just putting hyphens everywhere.

I don't see at all how not being able to type certain non alphabetical symbols means it isn't good for English. Half you quoted are general math symbols that have nothing to do with English.

I want you to explain to your grandmother why the software you wrote can handle dollar signs but not cent signs.

FIELDATA didn't support the cent sign either, which my grandmother is much more familiar with, so I think it would not be hard for her to grasp this concept.

> I don't see at all how not being able to type certain non alphabetical symbols means it isn't good for English.

It means that because the English language uses punctuation and other non-alphabetic symbols, not just letters of the alphabet, as an examination of any substantial corpus of written material in the language not solely consisting of material where the writer was constrained by ASCII (and, heck, even those where they were, though the set of non-alphabetic symbols used would be narrower) would instantly reveal.

> Half you quoted are general math symbols that have nothing to do with English.

5 (7 if you mistakenly include angle brackets, which do have textual, non-mathematical uses) out of 15 is not half, but mathematical symbols are used in written English.

This is like saying that a python interpreter isn't good for interpreting python, because frequently python programs might be invoked using a bash script, and so python is incomplete without bash.

If my weird squiggley that I use in my specific niche field isn't represented by any Unicode code point, does that mean that Unicode isn't good for representing English?

I didn’t know ‘cafe’ was even a thing and wondered what’s fancy about café, so I had to Google a bit. I learned that, yes, Americans do spell ‘cafe’, but the most common spelling still is café. Even https://simple.wikipedia.org/wiki/Café redirects cafe to café.

https://www.merriam-webster.com/dictionary/cafe‘: “variants: or less commonly cafe”

https://dictionary.cambridge.org/dictionary/essential-americ...: “noun (also cafe)”

https://www.dictionary.com/browse/cafe also prefers to see the accent

That spelling preference may be shifting. I couldn’t find evidence for it, though.

> the most common spelling still is café

Really? The data say otherwise: https://books.google.com/ngrams/graph?content=caf%C3%A9%2Cca...

Do you have another source that supports the popularity of the diacritic variant? Rather, café doesn't increase in popularity until after 2000, and thus would appear to be an affectation of sophistication.

Pretending ASCII is enough for English is at least understandable. I've seen engineers demanding that we find a localized string for "middle name" in Korean. For a social network product. (Hint: There are no middle names in Korean.)

Some people just don't care.

Not a great example if it's not a platform for only Korean people. If you have a field for middle name, there will probably be some-one with the language set to Korean still be interested in reading or writing that field (ie expats learning the language, or perhaps people with mixed heritage?). I think assuming because Korean (people) have no middle names you don't need to translate it is more careless than the other way around.

The uppercase mapping of Mkhedruli is Mtavruli in Unicode, but the titlecase mapping is not. If you use the uppercase mapping to automatically capitalise the first letter of a sentence, you are not following the recommendations of the Unicode standard.

TIL about ToTitleCase. An easy-to-visualize example of the difference is

    ToUpperCase("ǳ") => "Ǳ"
    ToTitleCase("ǳ") => "ǲ"

But how many programmers are even aware of the fact Unicode has both an Uppercase and Titlecase mapping? Let alone that they can do different things!

I routinely see code that treats titlecase as “set individual character in this Unicode string to its upper case equivalent”... It wouldn’t surprise me at all to find out this kind of titlecase vs uppercase mistake is a very widespread issue.

I for one never heard of "titlecase" before. I guess that's what I get from learning text encoding from APIs exposed in programming languages.

Interesting, I didn't know that there was a distinction between uppercase and titlecase. Perhaps unsurprisingly, unicode has a FAQ around just that: https://unicode.org/faq/casemap_charprop.html

The thing is one cannot type mtavruli letters. No modern keyboard layout which I am aware of allows that.

If you press "s" key it types "ს" (S), but if you press "shift+s" it types "შ" (SH), which is entirely different letter. Georgian alphabet has 33 letters, 7 more letters than Latin, so letters ჭღშჟძჩთ are mapped to shift+something keystrokes.

Russian language has 33 letters too, but Russian layouts override 7 punctuation keys ,.;'[]~ so pressing "," will not type a comma, but a letter "б", pressing "]" will type "ъ", so Russian layouts remap punctuation symbols to shift+number keystrokes. Georgian layouts keep punctuation key assignments same as for English and use shift+letter for additional letters.

Modern fonts may have mtavruli styled glyphs, that is true, but they are placed on mkhedruli code points anyway, which is exactly the problem, no widely used fonts support this case switch anyway and render ⍰.

In Arch-based Linux distros you can use Caps Lock. When Caps Lock is active, each letter comes out capitalized (including shift+ combinations).

Mtavruli has already been supported by the following fonts: Google Noto (Android 11 DP, Arch Linux), SegoeUI, Calibri (Windows 2019 update), Helvetica Neue (macOS Catalina, iOS 13).

> who cares, if it is not used in U.S. right?

This isn't intended to sound rude, but why should an American care? Dismissing the argument that for better or worse many people hold doesn't do any good. Making it might convince them otherwise.

Please don't take HN threads further into nationalistic flamewar.

That strikes me as much closer to linguistic than nationalistic; there's more than one nation that speaks English. Could you clarify what about that statement was flame-y? All I did was ask that adontz make the argument rather than making a dismissive statement.

You took a sarcastic question that was being asked from a defensive place and reflected it back to the person in an amplified way that was guaranteed to rub salt in their wound.

Reply on news.ycombinator.com