Our Hybrid lda2vec Algorithm | Hacker News Reader

Our Hybrid lda2vec Algorithm

multithreaded.stitchfix.com

100 pointsharaball4 comments

So this is basically a repost of the Slideshare on this topic (https://news.ycombinator.com/item?id=10919744 < previous HN thread.) without the pretty landing page around it. It does look like some 5000 new LOCs went into it since then, so that 's positive.

I also like the honesty of this report, mentioning different methods that are similar (even another project called lda2vec) and the disclaimer that this is probably not for everyone. The Python code does make it more accessible however, so I could see myself at least reusing concepts that are implemented here. On the whole this looks like an interesting algorithm for the stated purposes.

> Maybe the folks behind Hacker News want to evolve the site by encouraging more job hiring posts, or maybe they’d like to devote special site features to housing or more security features. That’s the kind of analysis we need to effectively steer our companies: more insight, less black box.

I guess that was written in gest, but I do hope some kind of reddit or SO-style siloing isn't on the cards to 'make HN grow'!

I rather hope that HN grows in the collaborative filtering direction instead. Instead of most interesting topics never making it high enough up the first page to actually gain any traction, it would be nice if collaborative filtering was used to make it so people can easily get more of what they like and easily find something completely different. Of course I've blogged and petitioned for this kind of approach in the past ;) http://williamedwardscoder.tumblr.com/post/15581427232/self-...

Then HN can stay the HN we all like, as many of us like slightly different things, and collaborative filtering can evolve with each of us letting us find like-minded others to share our filter-bubble with :)

I like the phrase "collaborative filtering" and it fits what we have in mind for HN nicely. (The term we use internally is "story review game", because we want to make a game of it.) But I think there's value in having HN's front page be the same for everybody, even though that guarantees that it's optimal for nobody, and produces a daily crop of "why is this on HN" protest comments.

A clarification of the "job postings" mechanic: only YC startups can make job postings, and they start at #6 and undergo constant rank decay (this behavior has been stated by the mods multiple times). The fact that % job postings has increased implies that % increase in YC startups > % increase in HN submissions over time, which is not unexpected.

If HN were profit-optimizing and let anyone put ads, it would break the site completely as the density of ads makes it more difficult to read content (something many many blogs like Forbes fail to realize).

Reply on news.ycombinator.com