How to tokenize Japanese in Python | Hacker News Reader

How to tokenize Japanese in Python

dampfkraft.com

59 pointspolm2320 comments

If you're interested in this you should take a look at Chrome's break engine source code for japanese/chinese: https://github.com/unicode-org/icu/blob/778d0a6d1d46faa724ea... https://chromium.googlesource.com/v8/v8/+/refs/heads/master/...

So that works on similar principles to MeCab but it's more limited and performance is... not very good. I wrote about some of the issues with it recently here.

https://github.com/polm/fugashi/issues/23

(Also, if you learned about Chrome's Japaense tokenization on HN, that was me too!)

For someone in the know, how different is tokenization strategy between Japanese and Chinese? The grammar/sentence structure is different and Chinese does not use kana or qualifiers for things like verbs in the same way but then both have compound words and lack of spaces in text.

Do languages like German which is rife with compound words have any overlap in tokenization or do you end up with completely different strategies?

The basic strategy is always the same: you use a dictionary to identify possible words, and if there's overlap use a language model to determine the most likely segmentation. The details can be very different, e.g. Fugashi is a wrapper for MeCab https://taku910.github.io/mecab/ which uses conditional random fields, while the most popular Chinese tokenizer is probably Jieba https://github.com/fxsjy/jieba which uses a combination of word frequencies and hidden markov models.

For German, the task is much easier, since most words are already separated by spaces, so only compound nouns might be problematic, but it's rare for those to have more than one possible segmentation. (There are of course exceptions, like e.g. Wachstube, which could be either Wach-Stube or Wachs-Tube. EDIT for non-German speakers: cognates are watch-stove = guardroom, wax-tube = tube of wax.)

Thank you - this was useful! I hadn't considered that in German compound nouns there will usually only be one way to cut the word which seems obvious now. (edge cases notwithstanding of course)

Ofc in chinese (and japanese) the ambiguity is much higher. Especially when abbreviation comes into the mix

My strategy for tokenising Chinese is brute-force: read 5 characters ahead, is that a word? No -> 4 characters, No -> 3 characters, No -> 2 characters -> Yes, copy to output, move 2 characters ahead, read the next 5, 4, 3, 2. It's very slow.

This is how humans do it though. https://pingtype.github.io/docs/advanced.html#spacing

If you have some Chinese text that you want to read, throw it into Pingtype (my program) and it should help you to pick up the vocabulary! https://pingtype.github.io/

You should be able to speed it up a bit by precomputing a prefix tree of all words. Then you can stop after three characters if there's no word in the tree that begins with those three characters. That would also allow you to do away with the 5-character limit and recognize longer words like 中华人民共和国 without a performance penalty.

Humans do a bit more than simple prefix matching, though. E.g. most people won't read 有权利 as 有权/利 because that doesn't make much sense.

>read 5 characters ahead

alright programmer question: why was the magic number 5? Are there no chinese words that are longer than 5 characters? Or does 5 characters cover 99.999% of all possible cases, and is therefore "good enough"?

It's the maximum limit for names on Ruten, a shopping site in Taiwan. That's also the reason I used the name 尸三十三尺 for deliveries for a while! ASCII characters weren't allowed.

There are some words that are longer (e.g. Mesopotamia) but in practice those are basically all proper nouns.

I enjoyed your first link. I remember a friend chuckling when she heard me speak chinese because I would speak "yizi yizi" which is very childlike. I still struggle neurologically to tokenize chinese in my head properly. Although I suspect it is personality driven as my music has been accused of being the same in the past!

It seems a bit weird that the example somehow doesn't consider 'fugashi' a compound noun and claims it's 2 nouns, 'fu kashi'.

Maybe that's intended but then apparently this is pointless if you want to know how things are pronounced.

The key aspect not captured here is "rendaku" -- the voicing of a normally unvoiced consonant in a compound word.

https://en.wikipedia.org/wiki/Rendaku

But rendaku rules are extremely complicated and often unpredictable , eg 中島 can be Nakashima or Nakajima depending in how the person chooses to pronounce it.

This is part of a tradeoff that UniDic makes between "natural" word boundaries and other factors like reproducibility and usefulness for search indexes.

UniDic has a rule that a "word" (or a "short unit word" 短単位, the unit used in UniDic) can't be more than two "minimal units", where an example of a minimal unit is a kanji with a Chinese reading. So 麩菓子 can't be a word, and in this case it's 菓子 with 麩 on the front.

You can read more about short unit words here.

https://unidic.ninjal.ac.jp/glossary#suw

It's not a perfect tradeoff, and I think it can be improved, but it does have its advantages.

There have been some recent attempts at creating EBNF grammar definitions for Japanese that try to balance simplicity and coverage https://learn-japanese.org/2020/01/04/japanese-grammar-in-eb...

If I'm not mistaken, that's a post by yourself, so you could just say "I recently attempted..."

The moment I saw the title I thought of you Paul (polm). No doubt you are the world's expert on tokenizing Japanese.

Also cool: https://www.aclweb.org/anthology/Y03-1034.pdf (parser from 2003 with 97% coverage on native text)

Be careful what you do with a Japanese tokenizer. Just because you got an ability(or I'd like to call it probability since it's not perfect) to tokenize Japanese doesn't mean you can process Japanese like the language you familiar with. Please, don't do clever things. It's better to just leave Japanese string as binary blob and treat it as is, passing left to right without any modification.

It is my observation that the programmer of native Indo-European languages tend to have a delusion that their native language is so fundamental that any language processing they deemed necessary must be applied to the any and all languages in the world.

Which cause the locale library in most programming languages tend to have a feature that's unfit for Japanese and it break the Japanese. Yes, the presence of the locale library break the Japanese language support.

These locale libraries modify the string binary and break the Japanese encoding. You can't easily implement no-op local for Japanese because these modification are often cause by the fundamental common part of the library rather than language specific part.

People have a good intention. They want to support multiple languages. but it's a delusion that passing the string binary to locale library magically solve the problem. The locale libraries were often written badly for Japanese so it break the Japanese. I would like to call it locale-tainted.

Since The locale library is part of the standard library of the programming languages and many software relies on it. We can't just avoid using locale library without massive refactoring of the source code which is impossible given the available resource for localization. So if we, the unfortunate Japanese programmer, got a job of localizing locale-tainted existing software to Japanese, we have to come up some workarounds.

Modifying the source code to remove the locale library usage is the best but often not possible. Hooking locale library to replace it with Japanese-safe version are dirty workaround but it requires to reimplement most of the text manipulation which cost many man-monthes and rigorous testing, Imagine implementing a printf/regex library to replace the standard library that satisfy all the use case in the wild. The worst workaround is let the stupid locale library do whatever the stupid Japanese breaking string binary transformation they damn want, then, reverse-transform after the string binary passing through the locale library to fix the mess. A mess that was unnecessary in the first place.

I was involved in C++ Standard Committee but people out there are so delusional they think it's a good idea for a new standard library to depend on broken locale library because it help the localization. Those delusional people are also designing Unicode library now. They won't fart out a good library under such delusion. I lost all hope for this industry.

So please. Don't introduce stupid unnecessary language processing to the language you don't understand. Make our job less horrible.

Tokenizing strings is usually not something that happens unnecessarily, but it's required for the implementation of e.g. keyword search. So it's not something people can just avoid doing.

> These locale libraries modify the string binary and break the Japanese encoding. You can't easily implement no-op local for Japanese because these modification are often cause by the fundamental common part of the library rather than language specific part.

Can you give an example of a situation where that happens? Is it an issue with e.g Shift-JIS encoding vs. Unicode?

Reply on news.ycombinator.com