Well, the necessity of scraping that much content would depend on how often it is being updated. There must be only a few thousand active pages on Stack Overflow on any given day, so executing a million-plus page views per day seems a bit overkill.

Google tends to reduce its crawl frequency for content that changes less frequently. Perhaps in SO's case, it struggles to identify recently changed content without crawling the entire site.