Just a pointer, the code for CommonCrawl Project is available on Github https://github.com/commoncrawl/commoncrawl