Hacker News Reader

Comment by stevenbedrick | Hacker News Reader

+1 to this... I've had very good luck using SVD/LSA-ish approaches for building recommendation engines.

HOWEVER- the big catch with naïve (i.e., simple to implement and understand) approaches to LSA are that every time you add a new datum, you have to recompute the SVD of your data- which is time consuming, and raises some interesting logistical challenges when done in the context of a web app (i.e., how you persist your matrices, etc.). There are more sophisticated LSA approaches that get around some of these problems, but they're more complex to implement.

Nothing insurmountable, but kind of a pain nonetheless, and w.r.t. the system described in the parent post, I'm not sure how well they'd integrate with GAE- can you use NumPy/GSL/etc. on GAE? Or would you be stuck using some sort of Java numerical computing library?

View on HN

I'd love any resources you could point to that were helpful to you when builder your recommendation engines (also, do you have links to the actual engines?). I've been going through a tutorial to understand how SVD works [1], but it would be nice to find something that will help me deal with implementation details. I thought I might just use Gensim [2] for now, but the authors of that tool themselves say that Gensim is more of a framework to learn about these things rather than a production-ready tool.

[1] http://www.miislita.com/information-retrieval-tutorial/svd-l... [2] http://nlp.fi.muni.cz/projekty/gensim/index.html

Sorry for the delay, I just now saw that somebody had replied. Regarding links to the systems- the only one that was ever publicly accessible was for a Facebook app that I wrote some years back that was let researchers post links to their publications on their profiles, and also featured a recommendation system to find other users of the app that had "similar" publications. It's unfortunately fallen into a pretty sorry state of disrepair, as I haven't had any time to devote to maintenance in more than two years and Facebook's APIs have changed quite a bit since then. I described the underlying system a little bit in a conference paper: http://view.ncbi.nlm.nih.gov/pubmed/18999247

The other systems I've used the approach for have all been along either bibliometric/bibliographic lines, or have been relating to content-based image retrieval. It's a pretty robust approach, but can take a little bit of tuning to get just right- coming up with a good evaluation strategy is important to getting the most out of it, I've found.

As far as references that were useful:

Ilya Grigorik has a very accessible getting-your-feet-wet tutorial on his site: http://www.igvita.com/2007/01/15/svd-recommendation-system-i...

It might be a little dated w.r.t. specific libraries or APIs, but the basic technique is there. For a more comprehensive look at the SVD-IR approach, take a look at:

Berry et al. Using Linear Algebra for Intelligent Information Retrieval. SIAM Review (1995) vol. 37 (4) pp. 573-595

The SVD approach falls in the same family as Latent Semantic Analysis, which is a whole black art unto itself- I'd actually suggest going back to the early papers by Landauer, Dumais, etc. if you're really interested- those guys did a great job writing up what was at the time really novel stuff.

My contact info should be in my profile, drop me an email if you have any more questions (or to let me know what you end up doing!).

heya binomial, don't let gensim's disclaimer deter you! it's been used by many people, both academically and commercially, and it's fairly mature.

having said that, i'm always happy to hear about success and failure stories from users. so if there's something you find broken/missing in gensim, just let me know.

NumPy is not available on the Google App Engine: http://code.google.com/p/googleappengine/issues/detail?id=19... Apparently they don't want to vet C libraries?

Some searches suggested: http://www.picloud.com/

Reply on news.ycombinator.com