HN
Hacker News
Top
New
Best
Ask
Show
Jobs
Comment by 1024core | Hacker News Reader
Full thread
1024core
·
So what is the most accurate way to tokenize CJK text?
View on HN
edflsafoiewq
·
By hand using a native speaker.
tobyhinloopen
·
Is there an API for that?
cferr
·
Yes, but it hasn't been written yet.
edjroot
·
You joke, but...
https://blog.ycombinator.com/scale/
syrrim
·
you could try something with mturk, I'm sure.
1024core
·
Surely someone must've come up with rules for it by now?
earthboundkid
·
Are there rules (as opposed to heuristics) for splitting up English, like “words—separated haphazardly—by em–dashes?”
Reply on news.ycombinator.com