Hacker News Reader

Comment by anigbrowl | Hacker News Reader

Can't blame them, given how mediocre computer representation continues to be.

Just hop onto any Wikipedia page about Chinese characters, and be perplexed at the non-displayability of many characters, eg https://en.wiktionary.org/wiki/Appendix:Chinese_radical/%E4%...

Unicode just isn't a good solution for pictographic languages where new characters can be invented the same way new words can be coined in alphabetic languages. In practice this doesn't happen that often, but there's no reason they can't be; there's an annual creative kanji contest in Japan: https://sousaku-kanji.com/archive.html

View on HN

AFAIK the construction of new kanji is a novelty, not something of any real importance in communication.

Japan has embraced UTF-8, and before that they embraced Shift-JIS - a homegrown encoding that also didn't support constructing new characters. Is there any encoding that works well for this particular case? AFAIK there aren't any of any prominence, which suggests that it's a non-issue - surely if constructing new characters was important to them, Japanese or Chinese or Taiwanese computer scientists and standards organizations would push for some kind of encoding that enabled it.

You're correct. Parent comment says something true (some / many characters don't have a representation in most fonts), but the inference it makes is wrong (this is a bottleneck for communication)

And specifically these characters that are not present in mainstream fonts are getting so far down the long tail that most natives wouldn't know how to write them, so would result in e.g. hiragana in Japanese if they knew the word at all. To some extent computers have even increased the variety of kanji usage as you'll get textbooks that describe a word as "usually written in kana" because it's infrequent enough that an average Japanese person wouldn't be confident to handwrite it from scratch, and so twenty years ago that meant they had to use Hiragana.

But in these days of computer IMEs when they type it in phonetically and get a list of options presented to them, they're confident enough to pick one, and so the kanji gets used a lot more.

so far down the long tail that most natives wouldn't know how to write them

True, but if we can't even display them then chunks of language are going to effectively disappear because people will have less and less opportunity to encounter them.

AFAIK the construction of new kanji is a novelty

Yes, that's why I referred to it as a 'creative competition.'

I think you're missing the main point that more than half of the characters on a relevant Wikipedia page don't display properly.

Does this come from actual experience trying to communicate in chinese or japanese? At least for chinese (but I'm pretty sure it applies to japanese too), what you're saying doesn't ring as relevant (i.e. it is true that some characters don't have a graphic representation in many fonts, but it doesn't cause problems for communication / creation of new words)

Plenty of fonts can represent a vast amount of characters (search around, but even free fonts can have >50k), much more than even highly literate people could ever use (a very educated person might know 8k characters, but would use much less actively).

Every new release of Unicode releases new characters that are slowly incorporated into fonts...but for many years these are on the truly long tail of characters, nothing that any common person would use. There might be a few exceptions with new characters (the new japanese emperor characters comes to mind), but the thing to bear in mind is that new words don't necessarily involve new characters. A big chunk of nouns in chinese are composed of multiple characters, so making a new combination can generate a new word (电视 is television...electric + vision). This is probably even easier in japanese were they can use katakana to spell an English word and create a new japanese word.

Yes. Han Unification is something people read about online and decide is bad or whatever, but it actually isn’t relevant to using Japanese with computers. People use Shift-JIS and fax machines because legacy technology takes a long time to die everywhere and longer in Japan. It’s not more complicated than that.

This is correct: as much as it's wrong, Han Unification isn't a problem so long you stick to one language to support and remove all conflicting fonts for the other two locales, which are rarely needed anyway in Asia, it just comes back when you try to support multiple CJK languages at the same time. Unicode coverage of each languages are totally fine for 99% of use cases, the Unification problem is just it's reusing the same address space for all CJK languages.

Well, we all know Unicode Han Unification is fine for not just 99 but 99.99% of use cases. It is those 0.01% that we care about.

Indeed, the only problem is you’re not going to be mainstream in Japan, but that honestly hardly matter…

The problem is probably less prevalent in Japanese. There's like 10x less kanji than hanzi, and only 2-3k are normally used. Also faxes in Japan are generally sent using printed documents, so handwriting a very rare character doesn't explain the usage of faxes.

Actually Unicode has something for character creation: 'IDS', or Ideographic Description Sequence. For example, the Chinese character for the word Biang[1] can be describe with:

⿺辶⿳穴⿲月⿱⿲幺言幺⿲長馬長刂心

[1] https://en.wikipedia.org/wiki/Biangbiang_noodles#Chinese_cha...

> In practice this doesn't happen that often

The primary reason that this doesn't happen that often is that neither Chinese nor Japanese use Chinese characters in a primarily pictographic manner anymore. Only a small subset of the characters are pictographic (and even they are only vaguely so), and unlike in the distant past, most words are actually compositions of several characters, so new words can be represented via either combinations of existing characters if applicable, or via borrowing existing characters for their pronunciation (or in the case of Japanese, using the syllabic characters).

most words are actually compositions of several characters

For that matter most characters are combinations of other ones. Briefly, there are 27 different stroke shapes and 214 different 'radicals' in kanji that are recombined in various ways to form new characters. Japanese kanji overlap heavily with Chinese Kangxi, but they're gone in somewhat different directions over the last few centuries. (I'm interested in both languages but less familiar with Chinese than Japanese because I can't speak the former at all.)

https://en.wikipedia.org/wiki/List_of_kanji_radicals_by_stro...

I'm not a linguist, but I don't think you can understand languages that use Chinese characters if you're not willing to look at the components those characters are built out of. I mean, you could just learn them by sight and remember the ~2000 most common ones, but that seems like both a more shallow and difficult approach.

Anyway, a lot of the more obscure characters are not displayed properly on most web pages like the Wikipedia example above because they're in one of the CJK extension tables, and this seems like a problem, because few people are going to want to dig through the unicode charts to find them. This isn't a problem with unicode as such, but with the way browsers and operating systems implement it.

https://unicode.org/charts/

You couldn't simply do an encoding that generates kanji procedurally from the list of component radicals because of the differing placements, and sometimes significant modifications of the radicals to fit in the kanji.

Take the kanji 勉 which is made up of the radicals 免 and 力. You could maybe have a "right half" slot for the 力 radical which would figure out the stretching but the fact that the leg of 免 extends underneath it is something that only exists for that kanji and then you'll need to special case it.

I probably couldn't do it, and while I might have the skills in future it would probably be a fool's errand to build a system for that just to demonstrate that it could. But I don't see any fundamental barrier to the idea; it's a comprehensible system, after all. Maybe I'll sic an LLM on it when I'm feeling less conscious of heat loads.

That's super interesting: so even the 'well meaning' Unicode committee got a basic part of writing culturally very wrong: the assumption was that these are Alphabets, rather than living things. Alphabets change as well over time, but much more slowly than the words. Thanks for pointing this out.

I think emojis were a mistake to include in Unicode.

But they actually prove that we could continue adding characters, if the design allowed enough foresight in space designated to these evolving sets.

To an extent, the fact that they made so much software care about non-BMP characters that otherwise wouldn't and the fact that they made English speaking users care about characters outside Latin-1 makes it worthwhile. Even from an English speaking country, running into software that wouldn't accept the € sign was a problem until the early 10s, and I think displaying emojis drives American companies more than displaying the euro sign or kanji.

So I'll give emojis in Unicode a wide pass on criticism for their effect in making everyone actually support all of unicode, and making people give up things like MySQL's utf8mb3.

Reply on news.ycombinator.com