For anyone looking to diy something similar without bothering with the hassle of running a crawler, check out http://commoncrawl.org