Topic Suggestions for Millions of Repositories

Thanks for bluemonday, we use it in our GFM sanitisation. Certainly saved time on having to write our own from scratch and potentially getting it wrong. :)

We just implemented topic suggestions as well on our Q&A data. The literature in this field is actually quite solid. Here are a few key insights from our research: 1) Using recall@5 as your test metric gives you the benefit of being able to compare your results against the academic literature. They often (but not always) use recall@5. 2) We read the articles that introduce the following systems: TagCombine (was state of the art in 2013), NetTagCombine, EnTagRec, TagMulRec, fastText's supervised topic recommendation, and a few others. Unfortunately other than fastText not many of these have OSS libraries available out of the box, but that's ok as long as you're willing to use the underlying methods which are available off the shelf. 3) Our approach: we tested a few of the above systems, as well as mix-and-matched our own systems with tf-idf, topic modeling, multilabel classification, l-lda, fasttext, and others. For each attempt we calculated (a) recall@5 against our test set (we used k-means cross validation) and (b) how long it took to complete the training step. 4) In the end, we decided that we got the best combination of recall@5, training time, and engineer efficiency by using ONLY the fastText library. In the end we spent about 1 week trying other methods before we tried fastText. fastText took us about 3 hours to get first results, and then we tweaked the parameters for another two days before we found the right combination of learning rate, epoch, n-gram #, etc. (read the docs :) Our current recall@5 for v1 of this feature is a little above 0.5, which we think is good enough to provide a solid user experience.

Caveat: Unlike Github we actually do have user-generated labels to start with, so our modeling problem has one-step less complexity than theirs but are otherwise the same.

(TL;DR If you have gold standard data to start with and no time to read academic articles, try fastText supervised learning. If you have time to read academic articles, start with a lit search.)

I have also been researching this heavily recently. I am more concerned with Keyphrase Extraction rather than Topics. Topics I view as being more generalised across documents where as Keyphrase Extraction is concerned with the phrases or words that best represent each document which better suits my use case.

Moz have a good write up of their Keyphrase Extract pipeline here. https://moz.com/devblog/machine-learning-approach-to-keyword...

I would be happy to discuss this with you more if you would like.