Hacker News Reader

It's a fun exercise for a single tweet feed, but it is unfortunately not very useful when applied to larger scale learning, since you start hitting the model's limitations of a fixed number of topics and, its potentially long runtime, and the fact that it's not an online method (although there are online variants of LDA).

View on HN

Would you like to point us to some progress on how to overcome these limitations with LDA?

The gensim [0] package has a nice implementation of online LDA that can handle massive streaming datasets. If you want to avoid specifying the number of topics, you can use HDP-LDA. David Blei (inventor of LDA) has a reference implementation on his website [1], along with many other variants of LDA.

[0] http://radimrehurek.com/gensim/

[1] http://www.cs.princeton.edu/~blei/topicmodeling.html

Also check out Vowpal Wabbit [0]. Although this is a tool that provides several online versions of classification, regression, etc, there's an online LDA module included as well. It takes advantage of the "hashing trick", meaning that you can build an LDA model with a fixed memory footprint. I've had a lot of success using this tool.

[0] https://github.com/JohnLangford/vowpal_wabbit

Reply on news.ycombinator.com