Hacker News Reader

Comment by niels | Hacker News Reader

This will only be a good approach if you are going to scrape a small amount of pages. The problem is using synchronous requests, as this blocks the crawler until a request has finished. Using asynchronous requests such as supported by twisted (and scrapy) will allow you to crawl a lot faster using the same resources.

View on HN

This can actually sometimes be a feature. It makes it far less likely to have your IP banned. Its also a far more polite way to crawl someones site.

I agree, and for a 101 web scraping tutorial keeping it simple is nice.

You could crawl a lot of different sites one page at a time. When I wrote a large distributed download system, I would use pycurl's bandwidth throttle and also store a 5 minute average of bandwidth per domain that would prevent other downloaders from saturating a domain.

I would argue that the proper implementation provides real rate limiting, both in terms of max requests per second and also max concurrent requests. Limiting to one concurrent request is likely to be extremely slow for any significant amount of data, and a couple concurrent requests is not impolite. Obviously I'm not saying you should effectively DoS the site you're scraping, but there's a balance and 1 concurrent request is almost definitely the wrong place to set it.

I've heard this from countless people who have read the post. It's definitely made me want to look into Scrapy.

Reply on news.ycombinator.com