Hacker News Reader

So GoogleBot can't scrape more than 10 NYT articles per month, right?

Googlebot doesn't maintain a nytimes.com session, so it can crawl an unlimited number of articles same as you could if you tell your browser to not accept cookies from nytimes.com

Reply on news.ycombinator.com