Hacker News Reader

Comment by ec109685 | Hacker News Reader

How do new search engines like Perplexity and Kagi avoid getting rate limited or hitting captchas as they build and maintain their indexes?

Sites have to make exceptions for Google, but likely wouldn’t care enough to allow other search engines in.

View on HN

Kagi's internal index seems to focus on sites that wouldn't necessarily block spiders.

https://help.kagi.com/kagi/search-details/search-sources.htm...

I've crawled small web sites (even hosted on major providers) to archive them and never hit a wall.

I’m fairly sure Kagi reuses Google and Bing search results, then de-craps and applies your customized preferences to them. I’m not sure it even runs its own spider.

Kagi sends “anonymized API calls to traditional search indexes like Google, Yandex, Mojeek and Brave, specialized search engines like Marginalia, and sources of vertical information like Wolfram Alpha, Apple, Wikipedia, Open Meteo, Yelp, TripAdvisor and other APIs,” and also maintains its own “web index (internal name - Teclis) and news index (internal name - TinyGem)” [1].

[1] https://help.kagi.com/kagi/search-details/search-sources.htm...

Umm, So Google and MS can leave Kagi high and dry if it eats into their revenue?

they can, on the other hand, kagi launched recently something that would improve the crawling process

Thanks for correcting me on this.

My take (as a subscriber) is still that the majority of information I'm seeing is assimilated and distilled from the more mainstream sources, but I'm possibly underestimating how much comes from their own indexes.

I will admit that for news I tend to use the !gn bang to jump straight to Google News as it tends to be a more complete feed than Kagi's news tab, so wasn't thinking about TinyGem. But I'd also forgotten completely about Kagi's "Small Web" initiative for indexing non-commercial content that's not interlinked enough for other engines to prioritize it. I believe that's what Teclis (and possibly TinyGem as well) concentrates on.

IIRC, the few times I tried routing searches into that index I didn't find much, so I set it aside, but that's probably because I'm usually searching for more widely-consumed content.

This does not seem to be correct. I asked Kagi "does kagi build its own index". From the first result (on help.kagi.com):

> Kagi Search includes anonymized requests to traditional search indexes including Brave, as well our own non-commercial index (Teclis), news index (TinyGem), and an AI for instant answers. Teclis and TinyGem are a result of our crawl through millions of domains, focusing primarily on non-commercial, high-quality content.

[Edit] I guess it is possible that Google or Bing are hiding under "traditional search indexes including Brave".

I believe Kagi just pays to use Google's and Microsoft's indexes.

Google sells their index??

(Microsoft famously sells theirs to DuckDuckGo, right?)

Bing index has always been licensable , Yahoo was using it too, and they have other customers .

I don’t think Google has a publicly known licensing deal ,

They sell API access to Google searching.

Microsoft sells Bing to a ton of search engines, it has been the go to for everyone who wanted to run a search engine.

Ecosia have also been relying on Bing, but recently announced that they've now added Google as well.

I don’t think that’s correct.

Reply on news.ycombinator.com