Hacker News Reader

> Crawl at off-peak traffic times. If a news service has most of its users present between 9 am and 10 pm – then it might be good to crawl around 11 pm or in the wee hours of the morning.

How do you know this if it is not your website?

Also, the internet has no time zone.

View on HN

For sites where there is a peak usage time, it's probably obvious what that peak usage time is. A news service (their example) presumably primarily serves a country or a region - then off-peak traffic times are likely at night.

The Internet has no time zone, but its human users all do.

If your scraping a popular website Google Trends should be a pretty good proxy

Reply on news.ycombinator.com