PyCantonese: Cantonese Linguistics and NLP in Python | Hacker News Reader

PyCantonese: Cantonese Linguistics and NLP in Python

pycantonese.org

69 pointsteleforce13 comments

As a person learning cantonese, pycantonese and cantonese.sheik.co.uk were so helpful!

The trickiest thing to explain to people is that Cantonese is a diaglossika. It's spoken, but to write you would use mandarin. So there aren't a ton of giant written language corpuses to train on.

AFAICT that's why tools like this often use simple heuristics to do word segmentation. I remember going down a deep rabbit hole, before finally just packaging a small tool to do cantonese word segmentation using the A* algorithm:

https://github.com/machow/cantocut

Cantonese can be written, it is just that the character is not as common as those you see in Mandarin.

Not 100% Accurate.

A lot of us who speak Cantonese as mother tongue use "Cantonese" as written languages on online forums / medias. Written Mandarin normally appear on news or more formal environment.

I thought the same thing as you, but spurred by the other commenters, found this: https://en.wikipedia.org/wiki/Written_Cantonese

This is a minor point, but I'm cantonese and aware it can be written. (This is why I linked to a tool to segment written cantonese :).

Cantonese is in my opinion a richer and more interesting language than Mandarin, yet much more difficult in terms of finding resources and tools for working with it. I wish I could support this project but for some reason none of Github's paid features, including sponsoring, seem to work on my account.

Seems wild to build a new python NLP library totally disconnected from the current ecosystem. Particularly spaCy, but really any of NLTK, gensim, huggingface. Would love to understand the context of that choice.

Because Academia. "Lee, Jackson L., Litong Chen, Charles Lam, Chaak Ming Lau, and Tsz-Him Tsui. 2022. PyCantonese: Cantonese Linguistics and NLP in Python. Proceedings of the 13th Language Resources and Evaluation Conference."

That said parsing is not a solved problem in a lot of languages (particularly ones without spaces) and frameworks like spaCy just integrate existing solutions into a common pipeline and datastructures.

The first author's an independent scholar so I don't think that's right.

Nice to see some work being done with Cantonese. Large part of North American Chinese speak Cantonese but its rarely supported officially in any tools.

Google's cantonese IME is a godsend for us in this group (that never learned other pingyin)

Whenever someone talks about Cantonese I instantly think of Ah To's cartoons of Cantonese proverbs: https://says.com/my/fun/81-chinese-cantonese-proverbs-in-one... https://writecantonese8.wordpress.com/2014/02/25/cantonese-p...

Wow the authors are from the newest universities in Hong Kong. Hope they can make it big without supports from bigger unis like CUHK!

Reply on news.ycombinator.com