Hacker News Reader

Comment by tibbon | Hacker News Reader

Err, if I was writing a scraper today, I'd just ignore the robots.txt

I dunno if Geocities had a similar robots.txt, but it didn't stop several groups from archiving it (which was the right thing to do in either scenario).

View on HN

Well, I've worked for organizations that had active defenses against crawlers... Make too many HTTP request and hour and ~poof~ a deny directive goes in the .htaccess file, or if they really like to play a rough game they'll firewall you.

I know delicious had active defenses because I ran afoul of them.

And really, slowing your connections down to rates that are measured in baud would be the most fun.

True, there are plenty of relatively cheap proxy services made for exactly this reason though that will roll your IP on each request.

It's only a matter of time before some sysadmin, or automated log analyzer sees you and stops it. You'd be better served to start a distributed scraping and archiving mission going, but, I'm not sure how to start that.

I'm not sure if JS can request pages and send them to you (XSF protection), but it might be worth writing a small script to request URLs from your server, spider them and send the content back.

Then, put it on a website, and tell people "by staying on this page, you are donating bandwidth and helping archive delicious".

It's so no-hassle that I bet you could get a huge following.

I guess this might actually be a great use of 80Legs. Has anyone used them in the past, and if so, did it actually work?

This would be possible with Flash.

Ah, yes! Come to think of it, JS probably couldn't get around the same-origin restrictions of browsers...

Reply on news.ycombinator.com