Hacker News Reader

I think it is more complex than that. Common crawl does not index the whole web every month. So even if you use common crawl and just index it every month, which you could do pretty cheaply admittedly, I don't think that would lead to a good search index.

Running an index is an extremely profitable business, from multiple points of view (you can literally earn money, but also run ads, you get information you can sell, you can buy mindshare). Everybody is looking for indexes beyond Google and Bing, but there are none. If it really is as easy as indexing common crawl, then I think we'd have more indexes.

This reads like the “I could build Dropbox in a weekend”.

Haha yes, but my argument is not about individuals, but about "tech" companies that externalize everything and do not develop tech internally.

It's just files, how hard could it be?

No that hard, took me 4 weekends to build a private search engine with Common Crawl, Wikipedia and HN as a link authority source. Takes about a week to crunch the data on an old Lenovo workstation with 256gb ram and some storage.

For news-only there's https://littleberg.com