Google Crawls My Site 90% Faster - Speed is Important
kadavy.net
kadavy.net
For reference, when some sites I was tracking spiked up over a few hundred MS crawl time I grew alarmed and resolved those issues.
Also, another good way to measure things is to look at 'Site Performance' in the Labs section of Webmaster Tools (right below Diagnostics used in this post). You'll get a graph that represents your site relative to the Internet along with improvement suggestions.
Also - while the conclusion seems accurate: faster load time, better ranking in perhaps 1% of cases is something Google has talked about - I think your reasoning is off.
You wrote: .If a site can be crawled faster - and requires less resources to index, doesn’t it stand to reason that it will be rewarded with higher search rankings?
It's not resources, it is a matter of user experience. The faster your site loads, the happier Google's searchers who clicked over to you are:
Speeding up websites is important - not just to site owners, but to all Internet users. Faster sites create happy users and we've seen in our internal studies that when a site responds slowly, visitors spend less time there
See http://googlewebmastercentral.blogspot.com/2010/04/using-sit...
What I did to make the speed improvements - check out the post that I referenced (twice) in the post: http://www.kadavy.net/blog/posts/wordpress-optimization-drea...
Unless they're doing it wrong, and I very much doubt they are given it's central to their business, it's purely a user experience issue.
Even so, faster page load time also means higher AdSense CTR, and more money - though I guess you could make an indirect user experience case for that.
You make a fair point about the AdSense. I would guess they care more about a better user experience, but it's certainly possible. Everything they do can be viewed as a means to keep people searching and using the web, and clicking on more ads.
But I have a cron job that tracks the loading times of random pages on our sites (from offsite, every 10 minutes). I get very consistent loading times (nowhere near that variance).
(Edit: they also suggest GZIP resources that are in fact already gzipped, at least they are according to the HTTP headers)
I noticed that when one of my sites went from 300ms response times to 30ms, the crawler started indexing more pages per day, and would index deeper, which meant more of my pages in their index. The result was a healthy boost in organic search traffic due to more long tail search matches.
What's your hypothesis for why the crawler started indexing more pages per day when your response time improved? That seems like evidence of my theory, but I'm a designer by training so maybe there's a more technical explanation I'm missing.
My hypothesis is that there's a single crawler that was looking at my site, and it makes a best effort to crawl as much as it can. I have more pages on my site than can be crawled in a single day at 300ms/page, but at 30ms/page, it can be done in 3-4 hours. I don't know enough about their architecture to make any further guesses than to say that it probably just grabs as much as it can within the time it's focused on my site before something else on the queue becomes a higher priority.