Hacker News Reader

I’m guessing that it’s impractical to self-host something big enough to be useful.

I don't believe that is a good enough limitation in this day and age of docker and kubernetes.

You take the search engine docker image and keep scaling the number of nodes that are running it in the cluster till your search result time is fast enough for you.

I am sure this already exists even as I type it, but this sounds like a great job for a distributed system running a federated search protocol.

The hard part is already solved, you don't even have to crawl the web to build the index. There is already a periodically refreshed index of the web that you can download: commoncrawl.org

Now someone just needs to configure, Apache Lucene as a proper docker image that can consume this index.

Reply on news.ycombinator.com