Hacker News Reader

Comment by bane | Hacker News Reader

Oh wow, that's a great personal story.

I would imagine things begin to improve around 2000 due to the broad adoption of unicode? I remember there being an absolutely huge number of encoding systems for the various CJK languages back then, but I think Windows eventually guessed/settled on UTF-16 IIR.

I didn't live in Asia during this time, but was heavily involved in writing some multilingual capable desktop windows software and was very aware of these challenges. I remember one colleague who worked on our Chinese language material having to buy an expensive copy of a Chinese British telegraph code book.

View on HN

> I would imagine things begin to improve around 2000 due to the broad adoption of unicode?

Yes. I don't remember the exact timing, but it was the switchover to Unicode by Windows and Mac that finally resolved the character encoding conflicts (mostly).

I don't know how much attention this got outside Japan, but there was significant opposition here to Unicode for a while from some Japanese intellectuals. Handwritten and, sometimes, typeset Japanese has traditionally allowed for a lot of minor variations in the forms of kanji. A person might write their name with an extra stroke in one character, for example. Some of those variants are ignored in Unicode as well as in the Japanese encodings on which it is based, and some people kicked up a fuss about it in the Japanese press. I remember reading rants accusing Bill Gates of trying to suppress Japanese culture by imposing a homogenized character encoding through Windows.

The controversy has long since died down, though there have been efforts to develop more comprehensive character sets of Japanese and other Asian languages, such as TRON Code and Mojikyō:

https://en.m.wikipedia.org/wiki/TRON_(encoding)

https://en.m.wikipedia.org/wiki/Mojiky%C5%8D

Oh it gets attention on Hacker News to this day. Any time there's a thread about something Unicode one or another of our Japanese colleagues will show up to make a fuss about how Unicode is quite insufficient for Japanese, for this reason or that.

In 2024 I'm out of sympathy for this. There's an entire tertiary plane for extended Hanzi/Kanji/ideographs, if there are still missing glyphs please take it up with the Unicode Consortium, because there's plenty of room and it is reasonable to support one (1) character encoding now.

But it is in fact Microsoft's fault that the farcical attempt to squeeze "CJK" into a two-byte encoding was even attempted. They were firmly committed to the idea that UCS-2 could ever be viable and we're stuck with various consequences of this fatuous premise to this day.

In fact, I occasionally run (currently maintained) CJK commercial software on my en-US Windows installation and I still run into ??? from time to time and have to guess what the text is supposed to be. It’s a shitshow.

There is a system wide setting that changes all non-Unicode text encoding to another code page e.g CP932 for Shift-JIS. Third party tools are available to do the same conversion on a per application basis.

It’s not as bad as trying to load some really old CJK web pages on mobile devices: few mobile browser has an accessible option to select character encoding and there appears to be none on iOS. The only option is to change the system language and that didn’t always work for more obscure character codes.

https://xupefei.github.io/Locale-Emulator/

Heck, the stupid Windows application for the "my number card" has this problem, and it is an official application coming from the government itself! How is it possible that in 2024 we still have this problem?

Yeah and a part of e-tax literally asks you to have Japanese Windows in fine print or it fails silently without any warning messages (Fixed by changing chrome locale to Japanese thankfully). Gotta love Japanese websites. I just go into the office and hand write the forms because it beats trying to debug cryptic issues.

Many Japanese PC games and desktop applications aimed at the home market, from small to large developers, also do sort of these things. Just look up the word "AppLocale".

Between 2010 and 2020 I've ordered something from Amazon around 5 times, and each time they spelled my legal name wrong, each time in a different way (due to encoding issues). And I'm an european, living in a large country. So I'm not surprised CJK languages have this problem.

For the record: this happens because text encoding problems was solved by broad adoption of not Unicode but by localized Windows 9x. Windows NT and apps on it to this day uses whichever encoding used by equivalent ancient regional versions of Windows 95 for language/region it booted in, e.g. cp1252/ISO-8859-1 for American English and cp932/Shift_JIS for Japanese.

True unification into Unicode only happened during the mobile shift in the form of iOS/Android.

Reply on news.ycombinator.com