If I were to leave a search engine to index these links, then I would have a very useful vertical search engine for anything and everything that interests people like us.
I don't know how this is done, other wise would have done so.
Any ideas why this hasn't been done yet?
What I am dreaming of is something like Google bot with the Hacker News links as the seed to start indexing... Also displaying the results with snippets and rating the results.
What say?
1. Take all URL's that have made it to the homepage.
2. Weigh them by the points they've been assigned.
3. Only go out say n deep (maybe three) by following all the links.
4. Let this be your corpus of HN relevancy and rank the results returned by the weighted points assigned to the original link with a decay factor so that by the time you hit the last n deep links it has decayed appropriately.
I have 204mb of HN scraped by user. I don't think this would be useful to you, but if anyone else is interested or is looking for a fun project that is related to linguistics that's half started you can contact me at noah@noahc.net