Hacker News Reader

Thanks! The tricky bit is to make this work in different languages where the "space" is not used to separate the different words, such as Chinese. We should implement a real Chinese lemmatizer there to chunk the words.

Not sure if you saw it, but we already have pinyin in there. If you open up the settings and tick "show pronunciations" they will appear above the chat messages.

View on HN

> We should implement a real Chinese lemmatizer there to chunk the words.

Or find all substrings that are listed in a dictionary (≈everyone uses cc-cedict https://www.mdbg.net/chinese/dictionary?page=cc-cedict ) and give translations for all of them. That way, the user won't be limited to any particular chunking granularity, which is always a finicky aspect of word segmenters to fine-tune.

At least for chinese there are off-the-shelf word segmenters you can use like jieba[0]- I used it in gptlingo and it Just Works(TM).

The "show pronounciations" setting just turns on pinyin above characters - what I want is to type pinyin and enter chinese characters. Actually showing the pinyin above characters is quite distracting!

[0]: https://pypi.org/project/jieba/

Reply on news.ycombinator.com