Tell HN: Increase your crawl rate in Google Webmaster Tools

I found that only a small percentage of my site was getting included in the Google index even after submitting a sitemap, however after putting the crawl rate upto the maximum (i.e. allowing Google to hit my server hard) my entire site was pulled in.

In theory changing the rate should have no impact on how much of your site is crawled, but in practice it does.

I'm guessing Google allocates crawling your site a block of wall time and then stops crawling after that period, hence increasing the rate Google can crawl at allows it to reach more of your site.

The end result for my startup CoderStack was dramatic, search traffic increased by an order of magnitude (roughly 10x).