Hacker News Reader

Comment by boolean | Hacker News Reader

This was useful. In my case I have a million articles that I want to group related ones together. Similar to Google News. I'm guessing I can use one of the algorithms (Cosine Similarity) to calculate the similarity of every two article and group the close numbers together. Any recommendations on how I should go about it? I'm trying to find Python libraries that can make this easier.

View on HN

Seconded gtani -- read up on LSA and SVD to do dimensionality reduction will get you pretty far in semantic analysis, after which you can do an additional step of analysis with the reduced dimensions to better suit your needs.

I don't know the python libraries off the top of my head, but the Python implementation of R (RPy) is decent and R is heavily used in most circles to have great literature written for it.

Not entirely clear what you're asking, if it's cluster by topic matter, or pick out specific named/physical entities or maybe sentiment analysis.

Two good first steps to look into, depending on your needs, are Bayesian classifiers and SVD (reduction of high dimensionality, the application to text processing was patented as Latent Semantic Indexing/Analysis, LSI or LSA, by IBM, I don't knnow if that's lapsed).

To stick with good first step approaches, look at ngrams and nwords.

Basically, you need a reasonable feature to match similarity on. N-words are pretty easy to construct, a 2-gram would be every pair of words used in a document.

Tf-idf is a good metric with that kind of feature, because it handles well the bias of frequent words like "the"

Reply on news.ycombinator.com