Hacker News Reader

Comment by pseudocomposer | Hacker News Reader

I’m my second year into Kagi and loving it. I actually just upgraded to get Kagi Assistant (basically, cloud access to every LLM out there). But the search alone is worth every penny, and it’s built/operated fully in-house as far as I know.

https://kagi.com

View on HN

They are highly dependent on outside search engines. Someone from Kagi gave an explanation on HN of their search costs and why they can't go lower, and calling the Google API on many (most?) search queries was a major driver of their costs.

It's great that they are developing their own index, but I'm skeptical that it makes up more than a tiny fraction of what they can get from Google/Bing. DDG has been making similar claims for years but are still heavily reliant on Bing.

This isn't to knock on upstart search engines. I think that Google Search has declined massively over the past 5-10 years and I rarely use it. More competition is sorely needed, but we should be be clear eyed about the landscape.

Being dependent as a fall-back seems like that's the right solution: Provide your own index, but vector in the default if you don't get it right.

That way users get tailored search without losing scale.

As much as I like kagi and wish it success, it's not a search engine from scratch. Kagi uses other search engines (Google and Bing) wraps them and does a light reranking

No they do not, they have their own indexes.

https://help.kagi.com/kagi/search-details/search-sources.htm...

That page includes the text "Our search results also include anonymized API calls to all major search result providers worldwide".

They source results from lots of places including Google. One way that you can confirm this is to search for something that only appears in a recent Reddit post. Google has done a deal with Reddit that they're the only company allowed to index Reddit since the summer.

DuckDuckGo gets no answers if you specify only results from the last week: https://duckduckgo.com/?q=caleb+williams+site%253Areddit.com...

Kagi is fine if you do the same: https://kagi.com/search?q=caleb+williams+site%3Areddit.com&d...

edit: I don't think this is a bad thing for Kagi. I'm a very happy subscriber, and it's nice for me that I still get results from Reddit. They're very useful!

Note that due to adversarial interoperability, search engines other than Google can scrape Reddit if they try hard enough. A rotating residential proxy subscription, while pricey, likely still costs orders of magnitude less than what Google paid. The same goes for Stack Overflow. You can also DIY by getting a handful of SIM cards. CGNAT, usually a scourge, works in your favour for this application since Reddit can't tell the difference between you loading 10000 pages and 10000 people on your ISP loading one page each (depending on the ISP)

Kagi is also building their own index at the background, and mixes these indexes as you search.

When I search Kagi for "Hacker News", results start with this fine text:

    65 relevant results in 1.09s. 47% unique Kagi results.

So, other indexes are fillers for Kagi's own index. They can't target their bots to places, because they don't have the users' search history. They can only organically grow and process what they indexed.

How is it possible that a search for "Hacker News" produces only 65 results? There are thousands of pages out there with that exact phrase on it (including many sub-pages of this site).

The first result is almost assuredly the right one, but either they're ruling out a lot of pages as not-what-you-meant, or their index is really small.

It's another feature of Kagi. They know they have thousands of results, but they provide you a single page of most relevant results. If you want to see more because you exhausted the page, there's a "more results" button at the bottom.

Kagi reduces mental load by default, and this is a good thing.

That's an interesting positive spin on what is clearly a cost-cutting feature. I'm on the unlimited search tier, so it hasn't really been a big deal, but it's worth noting that clicking the "more results" button charges your account for an additional search too. Or at least it used to.

I'm also on the unlimited search tier, and honestly never needed to reach for the more results button even once. Kagi always delivered.

Having a low number of results has a net benefit of lower cognitive load for me, so I like how Kagi returns less results, not more.

But "more results" is counted towards a new search, and I didn't know that. Thanks for pointing out.

Light reranking is a huge understatement, given the amount of preferences they let you manually set for ranking.

Key differentiator: Kagi still properly responds to the negation sign and quotes in your search terms. This is why I pay for it. The signal-to-noise ratio is way higher than with other engines.

I loved kagi, but their cost is prohibitive for me.

The only way there is a chance for me to afford Kagi might be to buy "search credit" without a subscription and without minimum consumption. And then it would only be good if they allowed more than 1000 domain rules and showed more results (when available)

The search credit model sounds great. I would probably also pay Youtube credit. I use it too rarely that paying their monthly rate makes sense. The user experience with ads sucks, so I further try to reduce my usage. At least that's good for the environment and I can do more useful things.

Something like this? https://athenut.com/

If $10 is too much to afford, the problem is not really Kagi. You need to improve your economic situation urgently, because that is starvation level poverty. Seeing that you're posting on a message board like this, I assume you have at least average talent and capabilities to have a paid occupation.

Reply on news.ycombinator.com