The Science of Crawl, Part 2: Content Freshness

Nice post.

I only skimmed the post, but I believe you are assuming the utility of all pages to be equal. By "utility" I mean the value of information contained on the page relative to what your business is trying to achieve (not the organization's utility as you define it in the blog post). However, in practice, aren't information certain pages of much greater value to the business than others? For example, finding a new front page article on the NYT website could be more valuable than detecting 50 new Hacker News submissions. However, the NYT page would exhibit less divergence than the HN page.