Hacker News Reader

Trying to do something smart is usually the wrong approach and leads to users tangling themselves in invisible state that they do not understand and can't change. The best thing would have been for Unicode to not do Han unification. The second best would be to provide alternate glyphs now. The third best is to display the characters either in the language they're written in, when you know that for sure (usually for text that you wrote yourself), or in the user's most likely locale, when you don't. For locale I would go down this list of traits and pick the first that matches:

1) The language setting on your website if you have one and have translated it to C/J/K. You may use different TLDs for the different languages and discern that way, too.

2) The list of preferred languages from the browser. This is usually unreliable, but if someone has gone to the trouble of inputting "english=1;japanese=.9;chinese=.8", then it's a fair bet they want Japanese Kanji usually and will be understanding if you use them in place of Chinese Han characters.

3) The country to which the user's IP belongs. The least ideal option, but if you're in Korea and reading a random string of Hanzi, you probably expect them to look like Hanzi.

You will show the wrong characters to some users, but the behaviour is understandable. "Oh, the site is showing me Korean characters because I'm in Korea." is a lot easier to grasp than "The site is showing me Chinese characters because I clicked a dropdown one time that I forgot about and now I have no idea why my name is written wrong!"

You can argue about point 2) that some users might set their language preferences and forget about it, but so far I have never observed a user who doesn't know about them messing with the setting.

The problem is btw. not specific to Han unification/asian languages. It's that for every western language if you use screen readers.

My thought whilst reading the article was that the Han unification would actually help screen readers. IIUC The meaning of the glyph is the same across all the languages, so the screen reader will get the correct meaning and can present it according to local settings. The problem with the European languages is that the different characters (letter variations, accent variations) can change the meaning of the word they're part of.

Or have I misunderstood?

The same Han character can have wildly different pronunciations even in a single language (being a logogram, they represent a word and not a sound). KS X 1001, the primary Korean character set, even duplicated same characters according to their readings so that they can be almost [1] correctly converted back to Hangul. In practice they didn't work well though, and Unicode assigns all but one duplicate characters into the compatibility region.

[1] These readings didn't take account for systematic variations like the initial sound law (두음법칙, for example 이 vs. 리 at the beginning of words).

You can't split CJK sentences into individual letters, inspect them one by one and decipher its exact meaning. If you present Chinese writing to a Japanese speaker, they'd only see complete gibberish consisting of letters they may or may not recognize. It also works the other way around too.

On top of that, Kanjis aren't the only type of letters Japanese folks use. They also use Hiraganas and Katakanas, which are phonetic symbols and totally unrecognizable to non-Japanese speakers.

I'm not sure I understand what you're saying here. Are you saying that screen readers don't know which language to interpret text as?

If you don't tell them, yes.

As many websites do not convey the language properly they try to fix that using some heuristics, but especially for small texts (like messages) that can easily fail hard leading to completely wrong pronunciation.

Mail for example tried to fix that by allowing you to annotate the language for UTF-8 embedded in mail headers and the content. Ironically while that mechanism works for display names it doesn't work for email addresses itself. And I'm not sure any mail program uses it.

For what it's worth, mainstream OSes will also have poor handling of these cases, which will eliminate the most tricky cases by the sheer inconvenience it causes.

As far as I know textfields only have one font applied, so entering both languages in a single field won't be optimal. And if you're not doing anything fancy with your fields, they will all take the same font as well.

So even at the input level, the user switching languages will already be mildly screwed, and the best solution would probably be to change pages for each language.

> I cannot set a single lang: attribute for the whole page because it'll be wrong for the other two languages.

> What does that leave me? A dropdown list "C/J/K/Other" besides every single text field?

If you need to control the language display at the granularity of a single text field (rather than a user or a page or a website), then yes, you need a tool that operates on a single text field. This shouldn't be too surprising.

Surely you can get away with one of your other solutions, though. In particular, you can guess the language and be right most of the time.

It is surprising to me, because I can mix Czech, English and German just fine on a single screen, even in a single text field. From that perspective having to say "this whole text is in language X" seems backwards.

OK, yeah. If Unicode hadn't gone with Han unification (couldn't they know we'd have space to put zillions of characters in a font just a few years later?) you'd have the same flexibility in mixing C/J/K.

> (couldn't they know we'd have space to put zillions of characters in a font just a few years later?)

Representation matters. There should have been plenty of experts who understood that Han unification was problematic. It seems they were not in a position to do anything about it.

Part of the problem back then is they really really wanted to fit it all in 16 bits, and given there are more hanzi than that to start with this was a bit of an issue (back in the 80s the largest dictionary listed >54000 characters, now it’s >100000, and Japanese and Korean total about 50000 each).

A few years later they increased the character space by 5 bits and it wasn’t an issue anymore, but the original legacy of han unification remains.

Yes, and the people who prioritized the 16-bit size were OK with Japanese looking ridiculous in order to achieve their goal. From the article:

> if the equivalent symptom was happening with English text, ιҭ wѳuld bє lѳѳκιng sѳmєҭЋιng lικє ҭЋιs.

From a page linked by the article, "I Can Text You A Pile of Poo, But I Can’t Write My Name":

> To help English readers understand the absurdity of this premise, consider that the Latin alphabet (used by English) and the Cyrillic alphabet (used by Russian) are both derived from Greek. No native English speaker would ever think to try “Greco Unification” and consolidate the English, Russian, German, Swedish, Greek, and other European languages’ alphabets into a single alphabet.

If there had been a proposal to sacrifice English in order to cram Unicode into a certain code space size, is there any question that the people on the panel whose first language was English would have quashed it?

But people who might have spoken out for Japanese, or for Indian languages, etc. were seemingly not in a position to do anything.

Greco Unification wouldn't have been a priority because it couldn't have saved enough bits to matter.

Yes, of course — but the point is that native English speakers would have blocked such an absurd proposal no matter what.

Han Unification is just as absurd and just as unacceptable to a native Japanese speaker as Greco Unification would be to a native English speaker. But Han Unification went through because native Japanese speakers were not in a position to block it. Representation matters.

Were they? Or were they worried that their characters would be the ones given 4 bytes, and unifying means not squabbling with China and Korea over who goes first? It's not like Japan was not a technical center at the time. Shift-JIS is still around.

(EDIT: for screen reader users, a translation follows.)

¢σηѕι∂єяιηg тнє нιѕтσяι¢αℓ αηтιραтну вєтωєєη נαραη αη∂ ¢нιηα ιт'ѕ ρяєρσѕтєяσυѕ тнαт тнє נαραηєѕє ωσυℓ∂ υηιℓαтєяαℓℓу ѕα¢яιƒι¢є тнєιя ¢υℓтυяє ιη ѕυ¢н α ωαу.

(EDIT: Considering the historical antipathy between Japan and China it's preposterous that the Japanese would unilaterally sacrifice their culture in such a way.)

It makes sense given constraints of the time to try to fit the character set into 16 bits, but Unicode has variation selectors, why not use those for the ambiguous characters? They could have easily done something like HAN_CHARACTER_X + VARIANT_KANJI. It would take up an extra 16 bits, but given the density of CJK text relative to Latin text that may not be a big issue.

Edit: s/variant/variation/.

Could this be done now?

So such a variation selector would have to be used in front of every individual character? Or could it be applied across a substring, similar to how the text direction is flipped?

Either way this seems to be the most painless "fix" for this given the current situation. Setting a language for the whole text fails as soon as the languages are mixed.

Suppose you're blind and use a screen reader. How does German sound when pronounced by an English text-to-speech engine, or vice versa? Which should be used to read out that "single text field" that contains a mix of languages?

When reading mixed-language text we can't know if the "Tee" in the middle of an English sentence is German for "tea" or English slang for "t-shirt". We have to use imperfect context clues, so screen readers will have to do the same, it should be doable with today's technology. And if the software doesn't get it right every time, that's fine, because neither do humans.

> And if the software doesn't get it right every time, that's fine

It really is not, a screen reader using the wrong language is way, way worse than a human with the wrong pronunciation. The first time voiceover decided to switch to russian in the middle of english text I thought the os had crashed, the mangling is quite extreme.

> unix time is non-monotonic and discontinuous at random

Well, depends on your definition of unix time; if you use time() as the definition then it is actually monotonic because the integral part only repeats on leap seconds?

>How is a Japanese user asking for help learning Chinese supposed to write?

They write with japanese kanji. I imagine minuscule differences in kanji forms are negligible compared to general unfamiliarity with the foreign language.

Deprecate languages. We will all be writing exclusively in emojis from now on.

ツ