Hacker News Reader

You indicated that you used the Pushshift.io datasets, but how did you compute Jaccard Similarity on a dataset of 38M?

I didn't use pushshift, sorry. The data was collected from bigquery, stored locally into CSV files, and then I just wrote a node.js script to compute similarities.

Did you simply collect "user has posted to X, Y, and Z subreddits", or did you look at frequency too?

I didn't look into frequency. Is there a version of jaccard similarity that accounts for frequencies?

There's a weighted variant: https://en.wikipedia.org/wiki/Jaccard_index#Weighted_Jaccard...

The reason I asked the question is because back in 2016 I had a similar (now out of date) approach to finding related subreddits at scale using Jaccard similarity: https://minimaxir.com/2016/06/reddit-related-subreddits/

There, I only built a user edge if a given user commented on 5 distinct threads in a subreddit, since a lot of subreddit interaction was due to brigading.

Check out Graphlab Create's recommender toolkit, pretty fast for sets of that size

https://turi.com/products/create/docs/graphlab.toolkits.reco...

+1 for this recommendation, but it's called turicreate now and can be found here: https://github.com/apple/turicreate

Reply on news.ycombinator.com