A review of free sparse sequence taggers for NLP

fnl.es

8 points·fnl··3 comments

Did you test on newspaper text? 3% error rate looks like it, but the caveat would be that

(i) you want to tag other texts, and

(ii) a license for the Penn Treebank (i.e. the standard training set for English newspaper text) sets you back by about $3k,

plus another couple thousand if you want a commercial license for the Stanford tools (although GPLv2 means you can use it in a SaaS without getting one)

I ran the systems over the CoNLL 2000 PoS data (see training time), consisting of roughly 10k sentences, split into 8k training and 2k testing sentences (by the organizers of the ST).

However, this review is not about "who has the best accuracy", as I believe that is mostly feature-dependent, and you can find entire tomes in the literature and Web discussing this point back and forth (to little objective avail, unless part of a shared task with unseen data, IMO).

My interest was in training times, ease of defining features, model flexibility, and token throughput. And on these points, the systems I looked into have very large and important differences.

By the way, regarding your other points (although I think they are a bit off topic):

As for the cost of corpora: That number is extremely variable, and in many cases (outside of newswire, in particular) you might have your own training data. And then there is non-english NLP, too. Last but not least, tagging text is not the only thing you can apply a CRF (or any graphical model) to...

Regarding paying for a Stanford (or any other software) license: That is precisely what the text linked to explains - that you do not need to pay for such stuff - not only are there free tools around, some of them are far better than the commercially restricted ones.

Reply on news.ycombinator.com