Even though Common Crawl exists, if you wanted to get the data locally to have a bit of a poke around, it's practically impossible.

The next search engine will never be made by a few kids playing around in the shed.

The latest Common Crawl is roughly 120TB, enough to fit on 48 LTO-6 tapes. LTO-6 is the perfect middle ground expense-wise if you did want to play with the entire dataset at home. The tapes cost ~$30 (AUD) each, a cheap loader from eBay can be had for around ~$1000-4000 (like a Dell PowerVault or Fujitsu Eternus).

Your looking at around $8k just to have your data close to your computer. Either that, or you are going to just run everything on AWS/Google Cloud/Azure. Two of these are your competitors.

No one is going to copy 48 tapes for you, either :)

120TB would take around three months to download - if my ISP didn't cut me off first for probably being the heaviest consumer user of bandwidth.

https://www.chatnoir.eu/ has a search engine that is built from the CommonCrawl data running on Elasticsearch, and it runs on 130 nodes [0].

I would love to be able to run my own search engine, but unless there are a number of breakthrough algorithms, I don't see how it can be easily achieved.

I did ask some guys from the Internet Archive if they would be happy to copy some data for me onto some tapes. I wonder if there is a service for this?

[0] https://webis.de/downloads/publications/papers/bevendorff_20...