Hacker News Reader

Comment by freediver | Hacker News Reader

This is very creative. Thing to note is that this is not free (or at least it is likely not to be, OpenAI is charging for APIs). We are talking about ~1c per search looking at current GPT pricing (2c per ~750 words in/out). It is also indiscriminate deployed in this way, triggering for every search, even those that chatgpt will show suboptimal responses for, like 'starbucks near me', navigational queries like 'twitter' or anything else chatgpt will be bad in the context of search (a lot of things!). And it is non-trivial to predict which searches GPT will be good for (especially within typical result latency requirements of modern search engines).

We are doing some experiments with this at Kagi, and the main trick is to manage cost, possibly through on-demand triggering mechanism (which also can help manage accuracy). One thing to keep in mind is that this is likely going to get better/faster/cheaper in the future.

View on HN

For search, abandon Google, not hope. All ye should enter here: https://kagi.com

// as a kagi user, i'd

(a) imagine this as a lens at the simplest; just pick text-davinci-003, tokens 3000 to leave room for prompt, temp 0.7, freq 1 - 1.5, presence 0.5, and instead of best of 3 or 5 show repeated calls as if 3 - 5 unique results

(b) imagine a richer implementation that summarizes other articles on first SERP, then collates and summarizes those (with the compare/contrast structured synthesis GPT-3 does well when POVs differ), and shows the final rollup summary above the individual summaries, in the right hand column

// would also be OK connecting it to my OpenAI token so I'm paying, not you. having done the math, it's nominal cost if not the default mode.

Hey wow, it's the Kagi guy. Offtopic, sorry: I love Kagi. I would say 85% of my searches I stick with it, 15% I tuck tail and head to that one big search engine. Pretty phenomenal IMO, and I have no doubt it will get better. Keep it up!

The true killer apps will come when StabilityAI or another group releases an open source equivalent of GPT-3.5.

Compare Dall-E to the creative explosion that arose from Stable Diffusion.

Nobody is going to build the next Google atop OpenAI APIs except for OpenAI themselves. An open source model and pretrained weights will open the playing field for everyone to compete with Google.

Stable diffusion for davinci003 is coming

I can't wait!

Stable diffusion can run on regular PCs. GPT-3.5 on the hand can't without doing a lot of swap between disk and vram making it sl.

I forget where I read this recently but a compelling point made was that this is fun as a nerd toy but to deploy it to something general like Google or Gmail would be overwhelmingly expensive.

Reminds me of the, “so expensive that only the five richest kings of Europe will own them.” joke from The Simpsons. Eventually it’ll be ridiculously cheap and easy to include anywhere.

I remember estimates of the worth of each search to Google being 10 cents on average. Google does run Bert on many if not all of its searches anyway? And let’s not forget that searching the entire internet is not cheap either! Google holds the entire index in memory and every search hit goes through thousands of machines to return your results. In other words, running chatGPT might not exactly be a problem for google if it decides to do so.

Google’s search of “the entire internet” is lacking more and more every year.

It’s probably not true at all anymore. It’s probably “the sliver of the Internet we prefer you interacted with.”

Any time I search these days I’m amazed at how you can exhaust the search in 1-2 pages before you get to “related hits.”

Yeah, it's clearly lacking. Either de-indexing many pages or just blocking them from the results, even when I use double quotes.

Then I go to Yandex or something and voila, it pops right up. I'm not sure I care enough to pay for Kagi but there's something very wrong with Google (and DDG, and Bing, etc)

It’s definitely deindexing pages. It’s been years since I’ve been able to locate one of my old HN rants via Google Search when it used to be the first result. Before all I had to use was my username + a politician’s name that I’ve only ever used once. After looking up the comment with Algolia and adding more keywords directly from the post, Google just gives up

Genuinely curious, can you give both your name and the politician name in double quotes individually? Google has stopped requiring all terms to be present in search, this is a way of forcing it..

If I put the two words in quotes, the only result is this unrelated thread: https://news.ycombinator.com/item?id=28162412 - I never replied in that thread, so it completely ignored my username despite the quotes but did contain one instance of the other search term.

Just double checked Algolia and I actually have three comments that fit the query. Two posted nine years ago and one posted ten years ago. It's been at least three or four years since I was last able to use Google to find my comment.

Edit: Turns out if quote the entire first line containing the name, Google finds the comment. It seems they're only purging parts of their index

Using double quotes used to do this...but for semi-obscure topics I notice Google search doesn't give a crap about the quotes, it'll latch onto one or neither of the terms.

I'd estimate that running a large language model like GPT-3.5 or Lambda is currently 100x-1000x both more expensive and slower to run inference through than a language model like BERT.

So deploying it at Google scale is not viable (yet).

There is also the question of incentives. If an LLM model can return exactly what the user asked for, where do you put ads? Before the answer, after the answer or inside the answer? Any of those is a bad outcome for the user wanting just the answer.

We already witnessed the failure to monetize Alexa with ads - in a purely question answering setting, users will not tolerate anything but the answer itself. Thus, the business model for this needs to be paid AI/search, and Google would be facing innovator's dillema. If I was writing a book about Google, I would love to witness the VP level meetings at Google at this moment.

I can easily see how to monetize it in a minute - whenever the answer involves a recommendation that has monetary value (a product or service) you get the option of choosing what product label to recommend and there’s the monetization angle.

Problem with this is a) product or service recommendations are a tiny fraction of queries and b) gpt3 and any llm would be pretty bad at doing this, because they would exclude any recent product/services (they are trained on data from 1-2 years ago) and c) if the output of llm is changed in a way to replace llm’s recommendation with a paid one, why bother with having a llm in the first place?

The question is: how does Google monetize chatGPT results? If the answer is right there on the page, what’s the incentive for anyone to click on an ad?

If ads were truly useful we'd click on them anyway. You may be looking for the answer to how to run a VM inside of ChatGPT[0] but the AI also knows you'd like a new pair of headphones so… two birds with one stone!

[0] https://news.ycombinator.com/item?id=33847479

  > the AI also knows you'd like a new pair of headphone

I understand how Google knows that I'm currently searching for a new pair of headphones. But how did _you_ know?!?

If the entire internet is composed of bot written SEO spam, I think you’re spot on.

If it’s possible, please create a family plan or regional pricing. I love Kagi but if I were to buy it for my entire family, the bill would be $600/year - more than a month of rent.

In the country where I live (Vietnam) $600 would be 3-4 months wages for many people.

Any company that doesn't do regional pricing is only interested in doing business with rich countries. Which sucks but is understandable. I wish they were more honest about it though.

> Any company that doesn't do regional pricing is only interested in doing business with rich countries. Which sucks but is understandable. I wish they were more honest about it though.

This may be true if your cost per client/customer/etc is either negligible (such as with digital goods delivery), or dependent on their country (eg. retail).

Here, the bulk of their cost is computing resources, and they (according to their profile page) don't even make enough to cover it with the current price. I don't think this cost would go down with the customer location.

Yeah, it sucks a lot that people in rich countries can afford things people in other countries cannot, but that's kinda what "rich country" means.

I’m curious about their economics. I would assume there’s a huge upfront fixed cost but the marginal unit cost is tiny. They may not be breaking even or run at even, but that (I expect) is because they’re struggling to overcome that first hump of the fixed cost, but at the margin they’re doing well. I would be flabbergasted if my monthly is going to pay for my search transactions rather than the ability to execute any search at all. In that model you can in fact offer regional pricing as long as there’s a critical mass sufficient to justify the localization and regional edge deployments. You can go even further and let larger (in dollar space) regions subsidize smaller regions while they grow critical mass. Some companies I’ve worked at felt offering their service globally was worth the depressed margins even if they never made it make economic sense simply because it was useful enough - but also it creates a solid global brand. Anyway Kagi is great and I wish them the best of fortunes.

Kagi is not having own index and is a proxy. They pay to use indexes of actual search engines and this is not cheap that they process and display to users.

> Any company that doesn't do regional pricing is only interested in doing business with rich countries. Which sucks but is understandable. I wish they were more honest about it though.

Kagi loses money on their paid customers. If you have a Kagi account you can see how much you cost them. It makes no sense to offer regional pricing unless there is also a way to serve those customers in a cheaper way.

Will you have a family plan? Yes, in the future we will also offer the family plan. It will allow for adding multiple accounts under the same billing. And it will include powerful options for kids (Kagi for Kids) including control over the acceptable content.

I saw they have a Team plan, doesn’t that work for you?

> it is non-trivial to predict which searches GPT will be good for

Since pricing is roughly proportional to response length (and maybe prompt length?), it seems like ChatGPT could use itself to determine if the search is a good fit or not. Give it a prompt like "I want to know if you are likely to generate more immediately useful and actionable results than my web search. The query I am searching is <query>. Should I run you on this query?"

I tried running that on some sample queries to see its output, and compiled its responses in this table (note the last one is incorrect in my opinion):

    | Query                              | Yes or No |
    |------------------------------------|-----------|
    | "starbucks near me"                | No        |
    | "twitter"                          | No        |
    | "python requests get json"         | Yes       |
    | "peach and mango cocktail recipe"  | Yes       |
    | "hn"                               | No        |
    | "image of versaille"               | Yes       |

Btw, incredibly, ChatGPT actually made that table for me when I said "Please compile all the queries I just asked you about into an ascii table, where the first column is my query, and the second column is your Yes or No answer." The table it printed was correct but formatted as an HTML table, so I asked it to put it in a code block and I got almost exactly what I pasted above (just had to remove two extraneous spaces that made the lines not line up)

Yep, this is one of the amazing things about it. But using ChatGPT to determine whether a query is good for ChatGPT answer does not address the second point I made, one about typical latency requirements (it makes it worse).

Normally you want your search results in about 500ms. Using ChatGPT to first figure out if the query is good will take 1-3 seconds then using it again another 1-3 seconds. So we are talking about results in about 4 seconds. Plus it is not 1c per query now it is more likely 1.5c per query as you introduced an extra prompt.

It's awesome to hear that Kagi is experimenting with this. Even if it's some extra paid opt-in feature, I'd give it a chance.

I don't know why people are always so up in arms about cost: if Google switched to a paid rather than ads model and charged 1 credit (1c) per search you made, I'd still use it.

People always raise such a ruckus: how dare Youtube offer premium, etc. People get too addicted to free services and become self-entitled.

Results for the same query can be cached, making the Starbucks near me type queries bypass GPT3 other than the first time across all users.

Reply on news.ycombinator.com