A Spectre Is Haunting Unicode (2018)

https://news.ycombinator.com/item?id=226853 (18 comments)

As a contrived example if you had a symbol for 'happy' you want to be very cautious that it doesn't get converted to 'gay' because in your language gay and happy mean the same thing, in some repressive regime it means the leadership gets to execute you with the approval of the law.

The important thing for engineers to note is a technical shortcoming caused a tragic misunderstanding. Focusing instead on the well-known fact that some people have poor impulse control, knowing full well that is a non-controllable input, instead makes an excuse for poor engineering and implicitly expresses powerlessness to do anything about the problem.

My name got changed when I moved to Spain and it never bothered me, while I have met people who took great offence at the use of standard nicks that they had not explicitly sanctioned in advance. I know a guy who makes a new name up for everyone he meets. Like or lump it. If you are too sensitive about your name, you risk people not using it at all.

Nevertheless there have been countless times where people automatically substitute the more common name, or even worse in text messages manage to misread it and reply incorrectly.

It sometimes upsets her. The npc analogy is very apt, i guess many people are just very preoccupied?!

There's a trick in Chinese to explain which character you are referring to when it could be any number of homophones- you repeat it as part of another well-known word. To use a crap analogy, you might say "Je like Jeep" or Ge like geography" if your name is Jennifer or Gene. The latter is a bad example since Gene is itself a single syllable, but hopefully it is illustrative enough. On the other hand, if you have a special stroke in a common character (or remove a stroke, as in the post above), I am guessing it's harder to explain.

That's not how it works. Most Chinese characters stem from a character C having a pronunciation A referring to a meaning M being used to note another word of meaning M' with same pronunciation A (sometimes slightly different A'). This of course doesn't scale really well, hence the existence of determiners in logographic scripts, which are words used without their pronunciations placed before or after another to give a semantic clue. The innovation of Chinese (which I think is why it's still an efficient script today) was to incorporate the determiner in the character itself to give birth to a character C' where a part refer to the pronunciation and another acts as the determiner, instead of padding the main text with (a lot of) determiners.

There is a fundamental difference between pictographic languages where glyphs have intrinsic meaning, and alphabetic languages where letters reflect sounds.

Old English was much better spelled than current English because it didn't have a spelling. People wrote what they heard. The current mess is because we have 5 centuries of bad standards that can render ghoti as fish. I think you'll agree that the digraph ti, as in nation, is just as nonsensical as sh for the same sound and we'd be much better served by a single glyph for both.

We in fact have that already: https://en.wikipedia.org/wiki/International_Phonetic_Alphabe... English uses somewhere around 45 of those sounds depending on accent. Th for example renders two distinct sounds: ð and θ. þ is not any better than th, apart from brevity, because it also rendered to ð or θ when spoken depending on context.

Chinese is of course as much a pictographic language as English is an alphabetic one. A substantial number of glyphs come from combinations of simpler glyphs which have the same sound as the word you're trying to write.

Native speakers can probably recognise at least 3-4k kanji if not more but can probably only write around 2k from memory, depending on how well-read they are.

嘘 (lie) is the best example of an incredibly common word whose kanji form (which is used fairly often) is not in any official government list.

That means that if you know 4800 characters, and you read a text that is 1000 characters (equivalent to around 700 words) long, there's likely one character you won't recognize.

The funny thing is, if you recognize only the top six characters, you already know 10% of the characters in a typical text. The distribution is very top-heavy, but with a long tail that you do have to learn to become literate.

0. https://lingua.mtsu.edu/chinese-computing/statistics/char/li...

But copying how every Chinese dictionary renders characters as trees of simpler characters seems like a much better approach than the arbitrary letter to number mapping of unicode. That it works well in English is only due to the fact it has no accents.

I can confidently say that native solutions for scripts with accents were _not_ unicode like but overstrike. My grandfather has the source code, in Romanian, of a 1960s computer payment system he worked on which had to deal with both Romanian and Hungarian names.

The combinatorial explosion of possible letters and accents made unicode like encodings an obvious non-starter. Historically names could pick up any accent (some times more than one) on any letter. When you have 26 base letters and 6 possible accents you'd need 26 + 26 * 6 (182) unique representations for single accented letters and 26 + 26 * 6 + 26 ^ 2 (1118) for double accented letters. That makes a language which is fundamentally alphabet based unusable on a keyboard. Something that Unicode is still sweeping under the rug.

Even typewriters worked with a giant table: https://en.wikipedia.org/wiki/Chinese_typewriter

This is separate to the question of encoding -- phonetic-based input systems (IMEs) are a far more likely cause (there are less-widely-used shape-based IMEs which still spit out a Unicode codepoint). The same is happening to Japanese natives, though it should be noted that it's not the case that they cannot write 漢字 normally, they just might forget how to write a relatively rare one (just like how you might forget how to spell a word in English because of a dependence on autocorrect and spellcheck).

That's a weird goal for Unicode to have. We've already accomplished that; a PDF file does the job better (note: PDF documents already support every character existing in the past, present, or future!) while being less complex.

I guess math has a similar representation in unicode as well.

All that said, I think people use runes to express magic and spells (even to this day). I don’t think all the magical runes are expressed in unicode (and perhaps they shouldn’t). If you want to use a rune in that way, you might have to draw it out in SVG or something and then email it to your friends.

One of the biggest problems in the study of these cultures is that they left no written records. We know they had a writing system, the runes, but as far as we can tell they almost never used it for anything. Quite the opposite is true of Mesopotamian cultures, where we're buried in more records than we have the manpower to translate.

> Documents are also distinguished from "realia", which are three-dimensional objects that would otherwise satisfy the definition of "document" because they memorialize or represent thought; documents are considered more as 2-dimensional representations.

I think "realia" - a term I had never heard before - describes runestones better than "document".

Or you just make something up. If you’re coining a new character, you probably don’t care about whether the pronunciation is already known.

There's also 奭 https://en.wiktionary.org/wiki/%E5%A5%AD which is occasionally used as a censorship workaround to mock one of Xi Jinping's gaffes in an early 2000's TV interview where he bluffed about being able to carry two hundred "catty" (~100kg)'s worth of wheat on rural mountain roads. The character is composed of two 百 ("hundred") and one 人 ("human/person/people") which is a pitoral euphemism to that line he said on TV. I can't find any sources about this one that's in English so please bear with my half-assed explanation.

The unihan database records pronunciations as well but I don’t think anyone takes it seriously. I’m not aware of any software that “prescribes” specific meaning or pronunciations to characters, so basically if you want you can pick a character and assign it a meaning and convince everyone else to use it. For languages that don’t have a fully standardized writing system , it’s something you can do and people have done it. (There’s also a bunch of people trying to convince people to not use the defacto standardized characters in favor of archaic ones that they claim are more “authentic”… but that’s a story for another day