Hacker News Reader

Though you could end up with a load of URLs, titles, and a summary text, crawling google won't really get you to the important data in building a search engine. No follow-on links, only a subset of the text, no referrals, etc.

Love the 'paradox' of the infinite spider loop, hadn't considered that before.

View on HN

Most spiders implement a maximum depth in their search though. So you'd really just get an inefficient crawl, not an infinite one.

Of course, using Google as a seed list more than crawling only google. Not sure why I didn't think of that.

Crawling groups would be interesting. What other archives of usenet are there?

also, google had no problem crawling groups when it was still called dejanews.

Reply on news.ycombinator.com