Hacker News Reader

Comment by brk | Hacker News Reader

Yes, but...

For the most part "Google" is a condensed version of the web. In theory, you could "spider" a single site (google.com) and build your own search database without having to go out and crawl the web at large.

It's an odd paradox, but I think one search site crawling another search site is not a good idea. And there is probably an infinite spider loop hiding in that process somewhere.

View on HN

And there is probably an infinite spider loop hiding in that process somewhere.

And then, this strange loop attains self awareness... (GEB reference)

Not like anyone writing a spider has to obey the robots.txt file.

Sure, and their IP will be blocked by Google some time around the 10th request.

Next time you're looking at a PC infected with malware have a look at the network traffic using a sniffer, chances are pretty good that you'll see searches to google for the weirdest of terms. Apparently this is to get around the limitation that you mention. I'm assuming the results of such searches will be 'mailed home' through some kind of dead-drop.

Sure, but responding to the parent, that's why people don't crawl Google, not the Robots.txt.

And incidentally, some of those searches are to find forms that it can stuff links into. My sites are constantly getting hit by botnets searching for Drupal comment forms. Luckily it uses a quirky URL format that's easy for mod_security to block.

I smell a business opportunity here! I could develop a Digital Robots.txt Management platform that will be built into all server operating systems and would enforce that no spiders are allowed to disregards robots.txt. I'm a genius! Oh, wait...

Though you could end up with a load of URLs, titles, and a summary text, crawling google won't really get you to the important data in building a search engine. No follow-on links, only a subset of the text, no referrals, etc.

Love the 'paradox' of the infinite spider loop, hadn't considered that before.

Most spiders implement a maximum depth in their search though. So you'd really just get an inefficient crawl, not an infinite one.

Of course, using Google as a seed list more than crawling only google. Not sure why I didn't think of that.

Crawling groups would be interesting. What other archives of usenet are there?

also, google had no problem crawling groups when it was still called dejanews.

Actually a lot of search sites reguarlly pull down large searchs result sets from eachother to do comparisions to their own, though of course this isn't tecnically 'crawling' in the follow every link sense.

Reply on news.ycombinator.com