Show HN: Using LDA to suggest GitHub repositories based on what you have starred

That is a clever way to make use of a search API like GitHub's. The principled way to do this, though, is to run LDA over all descriptions on GitHub, then use that similarity index to find similar repositories. You could run LDA over code, too.

I'll note that there is a cold start problem with this implementation: using LDA on such a small set of short documents will often lead to uninformative topics with words that are too-specific. You need a big corpus to capture e.g. synonym relationships.

I've worked with NLP a bit before, but haven't worked with LDA and have only read the wikipedia article and gensim documentation. One thing I don't understand is why you only generate a single topic for each user, and then query the top n (5) terms. From what I understand of LDA, its usefulness is in partitioning text into k separate topics based on how often words are used in similar contexts. In my mind, this is more or less analogous (please tell me if this is wrong) to finding k centroids for a vector representation of text after training a word2vec mapping (in an appropriately low dimension given the document size) on that text. However, if you are only finding a single topic, you are only using one centroid, so your search will be the n tokens that are closest to the centroid. I'm pretty sure that the tokens (from the text) closest to the centroid of a word2vec mapping trained on a text will mostly consist of high-frequency words and semi-stop words (by this I mean words used in varied contexts because of their use in language, but not filtered by the stop word check).

Then if someone has many different topical interests, LDA might over-represent whichever topic has the plurality of text dedicated to it. For example if my starred repos are something like 30% Fortran, 30% Javascript, 40% Java, I believe your algorithm will mostly contain Java terms as queries. This seems to run counter to the goal of using LDA, which would be (to my understanding) to identify these latent topics and give relevant queries for each one / combining them.

I think a good way to address this would be to implement some way to change the default number of topics. One approach may be to use a trained (perhaps on github instances itself) word2vec instance to determine the "spread" of the incoming tokens: you could construct cliques based on pairwise distance between vectors and do something with that (let k be the number of cliques, or the number of cliques of size greater than m, etc.).

A different approach might be to precompute the vector average of each github repo. Then you could perform richer comparisons directly to documents (e.g. compare each clique's centroid to the repos) without directly querying github for tons of repos.

The New York Times recommender system uses a hybrid approach (Content Based + Collaborative Filtering) called Collaborative Topic Modeling on top of LDA[4]. It would be interesting to try that.

[1]: https://github-recs.appspot.com/

[2]: https://medium.com/towards-data-science/recommending-github-...

[3]: https://github.com/jbochi/facts

[4]: https://open.blogs.nytimes.com/2015/08/11/building-the-next-...

For anyone else who didn't know :D

It'd be interesting to know what topic terms it produces for each of my repos. It looks like it's taking all the repo descriptions, producing a topic model over that corpus with a single topic (`LdaModel(num_topics=1)`), and retrieving the top N terms for that topic. Those topic terms will be the most frequent words from the topic, so I think this will end up producing the most frequent words from the cleaned token set.

I'd be curious to see what happens if you could run LDA over the full dataset, produce multiple topics, and suggest repos based on those topics. This would be a pretty fun extension to the project!

If you're just running LDA over the repo description (and not looking into the content of any file, e.g. README), might http://ghtorrent.org/ be able to provide this?

Alternatively, maybe you want to include text from the README files -- could you use the Google Data snapshot of GitHub https://cloud.google.com/bigquery/public-data/github and do analysis like this: https://blog.exploratory.io/clustering-r-packages-based-on-g...

Or, it might be interesting to try producing a vector representation per repo by taking the description (and readme?), and doing something like: produce word vectors for each word, and sum the word vectors. https://spacy.io/ is a nice-to-use library that could help here.

Once you have a vector representation for each repo, using a distance metric cosine similarity could find related repos. Or (depending on the dataset size / performance) an approximation like spill trees or LSH forest.

Looking forward to seeing where this goes next!

I am so curious about your implementation, for instance, what sort of preprocessing did you have to carry out? I had written a script sometime back to analyze Paul Graham's essays (link: https://github.com/futureUnsure/pg-essay-lda), and had to remove date and times because they appeared a lot and distorted the top topics. I'm wondering if you had to do something similar for text that described code?

Also, did you write an LDA library yourself or did you leverage an existing library?

I apologize in advance if my questions sound naive/stupid, am just a noob...