Comment by gaberivera | Hacker News Reader

Comment by gaberivera | Hacker News Reader

Parent Full thread

Hi there, I wanted to flag a new feature we added this week: under the headlines on Techmeme we include links to "Forums" like Reddit, Ars OpenForum, Slashdot, etc. when we find threads discussing the news.

Included in the communities we check is Hacker News. In fact, currently the top story cluster links to Hacker News four times, and Hacker News appears a total of seven times on the page. (This will change of course as Techmeme evolves throughout the day.)

So this is a good way to sample the views this community has on the big tech stories of the moment, and often a handy way to see where they differ from other communities.

Ironically, we're using Reddit's search API to find all the subreddits linking to news on Techmeme. I'm pleased Reddit hasn't turned off that particular API!

This is a really cool feature! Sometimes I enjoy reading discussion in forums, moreso than individual tweets.

I'd like to ask: how do you scrape news articles and the forums that have discussion on them? Do you use RSS feeds or do you use a custom web scraper? How do you group them together when they are related (i.e., clusters)? Have you considered making it open source or do you prefer to keep the secret "sauce" secret?

I regularly use Techmeme and I've always wondered about these things.

Huge topic, but briefly, we use a combination of RSS, scrapers, and official APIs to obtain article/post text. As far as clustering, it's a combination of text analysis, link analysis, and editorial input. I don't expect us to open source much of our code unless we find a new source of revenue!

That's great, thanks!

I don't think reddit has turned off any APIs, that's not happening until next month.

why waste dev hours on api's that will be basically closed in a month?

Nicely done Gabe!

Is it really morally correct to include Reddit at all? They've shown they don't care about the community which powers them. Why give them your traffic?

Reply on news.ycombinator.com