Hacker News Reader

Comment by gdcbe | Hacker News Reader

Pretty sure that only Google and Microsoft have the money and resources to crawl the entire internet. Or perhaps the only that can AND are willing to.

Correct me if I’m wrong though, but I’m pretty certain that all other search engines in the same category use one of these as their backend. Eg I’m pretty certain that counts for duckduckgo as well.

View on HN

> Pretty sure that only Google and Microsoft have the money and resources to crawl the entire internet. Or perhaps the only that can AND are willing to.

Money and resources and a dominant-enough position so that your crawlers are not blocked by websites.

Unfortunately.

There's definitely gatekeeping on websites but having done a bunch of crawler work I can say you'd be surprised how rarely a site will outright block you if you just do the right things: have an identifiable user agent with a working URL that explains what your crawler does, respect robots.txt, implement polite crawling. As for actually being able to crawl the whole thing though, yeah it's stupidly expensive :-/

Brave Search has its own independent index too - https://brave.com/brave-search-beta/

I think Yandex have their own index and some others too like Marginalia, but the latter couldn't be called "in the same category" as the other three.

Yandex also suffered a security breach and their source code is available[0] although utilizing it in any way is ethically and legally dubious (at best).

[0] - https://arstechnica.com/information-technology/2023/01/massi...

What about Common Crawl?

https://commoncrawl.org/

Mwmbl has its own index but it's orders of magnitude smaller than commercial search engines.

Reply on news.ycombinator.com