Topic mining with LDA and Kmeans and interactive clustering in Python | Hacker News Reader

Topic mining with LDA and Kmeans and interactive clustering in Python

ahmedbesbes.com

81 pointsahmedbesbes21 comments

Hey Ahmed, nice write up!

One thought I had was that I would be careful using KMeans on tSNE transformed data. From my understanding, while tSNE preserves local relationships between data point, it (heavily) distorts long-distance relationships. What this means is that (if I get this right), you have almost arbitrary clustering results if your k is chosen such that it is not equal to the number of local clusters (a thing you don't know in advance). So clustering on distances between data points in tSNE coordinates seems to be a risky way to draw conclusions. I am not saying this is wrong, but if what I am saying is correct (please point out if I got something wrong), then your analysis may perhaps work better if you did KMeans on say the final 5-dimensional PCA space or something like that, and leave tsne to visualize the data (and perhaps label the data points by the clustering on PCAs?). Do you agree / does that make sense?

He fits and transforms Kmeans on the output of TfIdfVectorizer and not tSNE. tSNE is only used for 2D visualisation.

kmeans = kmeans_model.fit(vz)

kmeans_clusters = kmeans.predict(vz)

You're right. In that case, I would worry about the high dimensionality of that array. Does MiniBatchKMeans do some internal dimensionality reduction?

I'm still waiting to see interesting topic modeling results on non-news data. The news has this tendency to follow a relatively low amount of topics, making it easy to discover meaningful word distributions, but reviews, messages, and comments written by "normal humans" almost never seem to.

> interesting topic modeling results on non-news data.

this is easy enough to do, I'm wondering what exactly is your definition of "interesting" ?

Interesting in that it provides some real insight - confirming that "crash" and "frozen", etc are similar in app reviews is not interesting. Interesting topics are more precise: people talking about the video being frozen or not playing is a singular topic, but given that video is used in so many other reviews, would not be detected as a topic using LDA.

I have done countless LDA on tweets and it worked great.

here is an example of taking tweets that have 'diabetes' and correlating topics with counties. https://www.linkedin.com/in/karl-dailey-02557b65/treasury/po...

The original topics were actually better, but I was asked to adjust the language to wipe out common language across all tweets. You can still see interesting things going on though. North East: Charity, Hospitals, Research. South: Koolaid, sweat tea. etc.

I had also done topic modeling on customer surveys for Comcast (I cant show them), but the topics identified 3 key features that lead to low customer satisfaction.

I have also used LDA on grocery purchases (for mere fun), and it worked out really great as well.

But what real insights do the topics give? The only LDA results I've seen are "fairly obvious" or "not understandable". Seeing a "car, road, light, drive, trip, ..." topic is not insightful - this is a topic that is obvious to most humans.

A more interesting topic would be one that is understandable but was not obvious, even to the people intimately involved in data's subject matter. These are harder to discover, but they do exist - and I have never seen LDA surface them.

I see quantifiable and visualizible insights. Its also good for generating features, and I can also create interaction features with important n-grams.

If it cant give you what you are looking for, maybe you are asking a fish to climb a tree

The purpose of LDA/topic visualisations isn't to do something humans can't do. It's to do it extremely rapidly on large amounts of data.

positive reviews: http://pastebin.com/enEZzKmC

negative reviews: http://pastebin.com/vkGn7ZLH

Amazon Echo, roughly ~34k reviews. https://en.wikipedia.org/wiki/Non-negative_matrix_factorizat...

It's surprisingly useful for finding interesting trends in reviews.

"20 newsgroups" is a pretty ubiquitous human-generated data set for testing LDA and other NLP techniques. I've run LDA on it myself and it recovers topics fairly well.

The thing is, reviews, messages, comments, etc, in general, do tend to revolve around some central topic(s). For example, this comment right now is about LDA. It's not totally random.

blekko did some work with LDA against our 2,000 "slashtags", almost all of which were non-news-oriented. Sorry that this is not a free link:

http://ieeexplore.ieee.org/document/7050791/

I did a project around this (AutoTagger): search google for all the subjects in the library of congress, spider the first 10 pages, strip off anything not body text, build a vocabulary with frequency, merge vocabularies to get a subject vocabulary, reject words common across all vocabularies.

Then when presented with a new article on any subject create the vocabulary and frequencies for that article, strip out the words stripped previously and then do a match against all subjects.

Very easy to parallelize, very good results (surprisingly good, for such a simple algorithm in fact).

There are a lot of interesting applications of topic modelling (including more interesting + powerful techniques than LDA) in recommender systems - worth looking into if you're curious.

In a previous life we used LDA (together with some other techniques) to find clusters of similar Android applications (based on their descriptions). We then looked for apps within a cluster that used vastly different permission-gated APIs to identify malware and grayware. It worked pretty well.

See http://dl.acm.org/citation.cfm?id=2568276

Good stuff ... if you're interested in this stuff, you might be interested in Richard Socher's Stanford NLP class - http://cs224d.stanford.edu/

I have a dumb question about Bokeh ... if I use RStudio I can create some markdown, export to PDF, send around to team/clients. Typically I can't do anything interactive without hosting on a shiny server or referencing some server-side magic.

If I use Bokeh, is it as simple as creating a Jupyter notebook, saving to HTML, and then anyone can see all the D3 magic? Or are there any drawbacks like file size, compatibility, browsers not letting you open due to security or whatnot? If any pointers, tutorials, examples anyone has would love to hear them, look me up if it's not deemed on-topic.

If you only need basic filtering and brushing for interactivity there's a new library called crosstalk that can do that. Some htmlwidgets like Plotly also support some interactivity without the need for a server.

http://rstudio.github.io/crosstalk/

Edit: Note that you can knit rMarkdown to HTML as well, which is what enables most of this stuff.

hmmh, thanks, might check that one out.

for next project maybe I'll try both ways, with ggplot and rmarkdown to HTML, and with Bokeh in python.

The ggplot will just knit to PNGs, would be interesting if ggplot could output D3 with intelligent rollovers, maybe eventually some scriptable animations.

One could use ggplotly but I don't want to post anything to, or host anything on a server, has to work 100% offline.

The R version of plotly has been fully offline ever since they open sourced it. Alternatively you could also try ggvis, but it's not as polished as ggplot yet.

thank you! I did not know this, having started with the online version...will give it a shot.

Reply on news.ycombinator.com