Automatic conversion between simplified and traditional Chinese | Hacker News Reader

Automatic conversion between simplified and traditional Chinese

meta.wikimedia.org

I asked in Wikimedia IRC before and never got a response, for any one with a system locale defaults to English, any Chinese page you visit are default to Chinese Simplified, even if you are from Taiwan or Hong Kong, where everyone uses Chinese Traditional.

I don't see a question in your statement. Is it, "Why does Wikipedia default to simplified Chinese?"

Probably because > 90% of native Chinese speakers use the simplified script. Now, I don't know what Wikipedia statistics are -- maybe usage skews heavier in areas where traditional characters are used, so that would be a reason to not do so, or to ask first. But without that information, defaulting to the more-commonly used script seems like a reasonable measure.

zh-cn (at least) is blocked in the PRC. I assume that even if it is unblocked at some point it will be subject to heavy censorship

It is periodically blocked and unblocked. The GFW is extremely inconsistent on Chinese wiki.

I don't think Wikipedia defaults to Simplified Chinese. It seems to default to whatever was entered by the user editing the page. So if users using Simplified Chinese and Traditional Chinese both edit the same page, it will end up with a mixture of characters.

For example, the page on Cryptography https://zh.wikipedia.org/wiki/%E5%AF%86%E7%A0%81%E5%AD%A6 contains both 密码学 and 密碼學, sometimes in adjacent sentences.

If you select the versions for Mainland China https://zh.wikipedia.org/zh-cn/%E5%AF%86%E7%A0%81%E5%AD%A6

or Taiwan https://zh.wikipedia.org/zh-tw/%E5%AF%86%E7%A0%81%E5%AD%A6

those get cleaned up to the appropriate shape.

That could be fixed simply with a browsing extension that automatically appends ?variant=zh_TW to every wikipedia URL (possibly checking the the subdomain). I do that manually when I want to force the character set. Implementation left to the reader ;)

Edit: not sure if it’s the correct parameter. But something like that exists for Wikipedia.

You can have multiple language preferences in your browser. What happens if English is your first and Traditional Chinese is your second?

Note that regardless of language tag, IIRC the article you read defaults to an unconverted version, which results in a mish-mash of simple and traditional characters (many simple characters and traditional characters have different codepoints and are not transformed by han unification language tags).

Yeah I always used to use the PHP implementation from mediawiki to achieve this, for over 15 years I think. However, last week we hit on https://github.com/BYVoid/OpenCC ... not sure which is superior yet.

What's interesting to me about the Mediawiki implementation is it has some features of a fully fledged rule-based machine translation system. It implements several layers of translation. In increasing level of precision, but also increasing effort and decreasing recall:

1. character to character conversion - originally based on tables from unicode

2. word/phrase type structure level matching - based on their own lists

3. domain specific word/phrase lists, (I'm not sure if this is based on category, opt in by including a template or both) see https://zh.wikipedia.org/wiki/Category:%E5%85%AC%E5%85%B1%E8... (I can't read Chinese - figured it out with Google Translate)

4. rules which apply for a whole article, see https://zh.wikipedia.org/wiki/Help:%E4%B8%AD%E6%96%87%E7%BB%...

5. rules which apply for a single word, see previous link

The whole thing is explained here: https://zh.wikipedia.org/wiki/Help:%E4%B8%AD%E6%96%87%E7%BB%...

The last 3 points address level 4 of Chinese character conversion mentioned in this article: http://www.cjk.org/cjk/c2c/c2cbasis.htm

I think OpenCC only implements level 1 and 2. I've looked at the source code to see that OpenCC gets "words" by using greedy longest prefix matching rather than fancier "Chinese word segmentation" -- I would guess MediaWiki does the same.

They say you need to support this conversion in the markup but gives no examples.

More in-depth info available at https://zh.m.wikipedia.org/wiki/Help:中文维基百科的繁简、地区词处理#转换技术

Don't know why this was downvoted, probably because not in English?

Anyway IIRC (and as the help page confirms) there are multiple conversion layers: letter-by-letter conversion tables (provided by MediaWiki itself), customizable word-by-word conversion tables [1] and special markup `-{...}-` for explicit variant selection [2].

[1] https://zh.m.wikipedia.org/wiki/MediaWiki:Conversiontable/zh... etc. You can also put them into the article itself, allowing domain-specific or article-specific conversion table.

[2] Mentioned under the heading "控制自动转换的代碼"; also can be used to turn off the wrong automatic translation.

Reply on news.ycombinator.com