Show HN: Google's Got a Secret

Common Crawl is great and super useful for research. It's where I got all those 18 million robots.txt files that informed my analysis. As far as using it for a search engine though, it's been proposed before and hasn't gone anywhere as far as I know. There are three major limitations that prevent it from being used to start up a search engine.

First, Common Crawl doesn't execute javascript or emulate a browser like Google does, so any page with dynamic content or interactivity wouldn't be crawled as fully as it would be by Googlebot.

Second, it suffers many of the same limitations in permissions as any other bot that isn't Google or Bing, so it wouldn't have access to important websites like facebook or linkedin.

Third, Common Crawl is on a monthly cadence. That's useful for research but not as a basis for a search engine by itself. It's outdated within a few hours of it finishing it's crawl.

Some people have suggested using Common Crawl as a starting point and then expanding from there with your own directed crawling of sites you deem important. I think it's not a good starting point for a search engine for the above reasons and also people have been suggesting probably since this 2007 and yet here we are, with not even a minor search engine based on common crawl that I am aware of. If you are gonna run your own crawler at the end of the day, might as well do the whole thing rather than try and combine two separate outputs.

So, Common Crawl is great, and I am very glad it exists and has been very helpful to my research, but I do no think it will not increase competition in the search engine market because it hasn't for the past decade. I think the governments needs to regulate Google's crawlers for us to see competition in the search engine market.