CommonCrawl: an open repository of web crawl data that is universally accessible

Just a pointer, the code for CommonCrawl Project is available on Github https://github.com/commoncrawl/commoncrawl

thread on HN from when common crawl was announced, interesting info there: http://news.ycombinator.com/item?id=3209690

If you into said things then maybe http://yacy.net/ (p2p crawler and search) will be useful to you as well.

The latest data available is from 2010-09-25, which seems to be too old to be useful for most things.

It would be great to hear more about the tools they are using to crawl and potentially open it up to more people who want to contribute computing resources.

This one may be also interesting for open data devs: http://scraperwiki.com/

I hear a lot of people are crunching on CommonCrawl's data. It'll be interesting the type of stuff people come up with!

This looks really nice!