Hacker News Reader

Comment by anonu | Hacker News Reader

I like where this is going. But I'm still a bit skeptical at the use of neural networks in this use case

How big is the problem space? There's a limited set of roots and morphologies...

Would a more rules based approach work (more accurately)?

It really depends on what dialects of Arabic you are looking at and what your goals are. My starting dataset is very small. I study classical Arabic, so I am only looking at fus-ha or MSA (not quite the same but very close), and I only pulled a small subset of the total number of words because this started out as just a toy project. The total number of fus-ha words in use today is probably in the low millions. But if you extend to all dialects of Arabic and all time periods then you may reach half a billion to a billion words. If you go from written Arabic to spoken Arabic, then there's no telling how big the input set is. Practically infinite at that point.

For the purpose of just using the language, the morphological rules are well-understood. One of the most popular dictionaries (Hans Wehr) is arranged not in alphabetical order, but by root. And there are many online lexicons with morphological metadata as listed in other comments here. So you're right, machine learning is not necessary here (but that's not to say it couldn't be done). This is mainly just for fun and learning.

Now, if you were to add all dialects of Arabic, then you may have a use case for ML...

From my experience any rules based approach in linguistics may solve the problem at first but it won't scale for too long (it has been tried before, most of old NLP was very rules based I would say) due to many intricacies of languages, corner cases and non-obvious exceptions. It could work — and it fact it does — for simple scenarios, but these are not very real world ones IMHO. I believe a neural model of such cases, over time, will yield better results. I am very skeptical these days about doing computer linguistics without considering some level of AI (so to speak).

Reply on news.ycombinator.com