I didn't try Doc2Vec. I wanted a hosted solution because I wouldn't have been able to compute all this locally (more than 100,000 posts).
If you tried it, did you have great results with? I may use it in future projects.
If you tried it, did you have great results with? I may use it in future projects.
Overall my approach is less automated than what I have seen in your codebase so it’s likely a bigger investment. I am happy to share more.
The hit rate is low. I've only tried to get embeddings for stories with a score greater than 100. SQL Query "SELECT count(*) FROM story WHERE score > 100;" gives me 155,228 stories and the corpus size is 108,477 stories.
108,477/ 155,228 = 0,6988236658
The main problems were 404 links and posts that weren't articles (such as tweets).