Hacker News Reader

Indeed. There are only 86K seconds in a day, or 31.5 million seconds in a year. Even if you assume that Google refreshes a page only a couple of times per year (which is absurd in terms of their freshness), it still means that you cannot have more than a few million pages in the index without accepting on average multiple crawls per second from the bots.

View on HN

Well, the necessity of scraping that much content would depend on how often it is being updated. There must be only a few thousand active pages on Stack Overflow on any given day, so executing a million-plus page views per day seems a bit overkill.

Google tends to reduce its crawl frequency for content that changes less frequently. Perhaps in SO's case, it struggles to identify recently changed content without crawling the entire site.

SO gets well over 10k completely new posts (Q&As) on a typical day.

A few thousand older posts are edited a day.

Throwing comments into the mix more than doubles the number of "new things", and of course every question, answer, comment, or edit is displayed on multiple pages (generally, on the owning user's page).

tl;dr - there's a lot more page churn than you might expect.

Reply on news.ycombinator.com