Hacker News Reader

Those millions of people want google to scrape and harvest, in the hope that they will rank higher etc etc.

If an unknown person tries to scrape, he/she will promptly get banned by those very same people (Google wouldn't like someone scraping their stuff either).

Different players different rules, I guess.

View on HN

Which large website did you get banned from for scraping? I did some scrapping and never got banned... perhaps my scraping's rate was not too fast.

If you try to mass scrape almost any major site (millions of pages of content) they'll block you.

For example, if you went one by one through Stack Overflow and sucked out every question and answer, your scraper bot would get banned (unless you're doing one request per minute, in which case you'll never finish).

Or if you tried to scrape Twitter.

Reply on news.ycombinator.com