CEO of Google Says It Has No Solution for Its AI Providing Incorrect Info | Hacker News Reader

CEO of Google Says It Has No Solution for Its AI Providing Incorrect Info

futurism.com

27 pointsColinWright32 comments

Open articleView on HN

So.. turn it off. It's not going to increase revenue or even maintain it, instead it will undermine core brand value.

Someone will get hurt and sue

Wait until Monday and see how the stock reacts

I predict that the stock will not react at all on Monday.

probably a bit of a "that's the joke" question, but is that due to the market being closed Monday?

or is that it didn't adjust on Friday and this isn't significant enough to adjust the next time the market is open (Tuesday)?

Yes, I was making a joke about Monday being a trading holiday.

Alphabet is quoted on some non-US markets, so it is tradeable today.

This is why GOOG didn't release a dedicated LLM product earlier despite building the core models for it almost a decade ago (eg. BERT).

This is also why MS and other companies are leveraging OpenAI, as they can offload liability onto them.

Emily Bender is correct about how LLMs are just "extruded text machines".

Google should have released a disableable feature leveraging Llms for whoever wanted to use it not shove it in their search results. Perhaps Pichai needs to step down and allow a more capable CEO take over.

> Perhaps Pichai needs to step down and allow a more capable CEO take over.

At a Google scale company (and any public company for that matter), CEO's job is not to drive product decisions. That comes down to product leadership.

The CEO's job is longer term strategy (as in these are the types of categories we need to build features for, these are the the kinds of customers we need to target, these are the core financial metrics we need to hit).

> Google should have released a disableable feature leveraging Llms for whoever wanted to use it not shove it in their search results

The PM that owned this feature definitely has UX A/B tested it and found that the majority of users just don't care.

For example, I don't see auto-generated LLM results when searching on mobile which is the majority of Google Search impressions.

I mean, I assume that once the fad dies away a bit, they will. _Wouldn’t exactly be the first time, for Google._

However, today, the markets, which can remain irrational longer than anyone can remain solvent, would punish Google for this; it has to go along pretending this is a real thing for a bit longer.

A LLM provides a statistical approximation that sometimes appears intelligent --- and sometimes not.

The question is, do the results justify the cost?

Instead of it being used as an "oracle" / "encyclopedia" it should beused as a glorified reference / helper system, with varieties optimized in different domains (say Java coding, history, and so on) as part of a bigger system with built-in guards

And according to Ilya, when this statistical approximation gets crazy good, its equivalent to AGI. AFAIK! We need Ilya himself to confirm again.

Should have weathered the storm and realized LLM craze is at its peak, and will die down. We are at Juicero juicing phase. Instead, they decided to kill their brand value.

Most of the blame that LLMs get is because they are used wrong.

LLM is a good information transformation engine. But if I supply it with messy or wrong information in the context, then I’ll end up with hallucinations.

This is why most of the AI systems tend to produce junk. LLM part is good, but the underlying system just provides it with noisy and meaningless data.

I believe that over time OpenAI will eat a considerate amount of Googles Search users. Search as we have known it for 25 years is becoming irrelevant because of LLMs and I don't think Google will be the one to counteract OpenAI. No search users no ad income. LMMs has totally removed the use for ads for revenue. Its amazing how well Microsoft and OpenAI has created a brand new way to interact with the Internet.

I actually believe that Google will not be around as we know it in 10-15 years. And I do think OpenAI and Microsoft is playing a major part of that.

[dupe]

More discussion: https://news.ycombinator.com/item?id=40468436 (Actual article https://news.ycombinator.com/item?id=40415846)

https://news.ycombinator.com/item?id=40448074

https://news.ycombinator.com/item?id=40461580

I think they are actually being too timid.

ChatGPT is already replacing search in many professions starting with SWE. IMO they should embrace it 100% and make Gemini a full 50% of the search page or a new button. Just making it a sad widget gives the signal its not the place to be to get AI ar all, not to mentio the subpar results.

Maybe sergey and larry are not taking the reins now. Sundar was a good peacetime CEO but maybe Google need some deep reorg of their product which only an original foubder can have the credential to lead

Not meant as a slight or troll, is gemini anywhere near ChatGPT yet? My understanding from limited experience is that they function very similarly but Gemini is wrong much more often, meaning Gemini power users are at a significant disadvantage relative to if they used ChatGPT instead.

I know the benchmarks say they’re vaguely similar, but benchmarks said the same of Llama II, and it was almost useless compared to ChatGPT. (so I don’t quite trust benchmarks, more interested in how these tools perform in real use cases)

It all depends on the benchmark and the use case.

If we are talking about the ability of models to follow instructions and carry out concrete tasks (as in products or inside RAG systems), then Gemini Pro 1.5 is currently on the eighth place in our benchmark.

Academic benchmarks, HF Leaderboards or LMSYS Chat arena will have different numbers.

> It all depends on the benchmark and the use case.

That's why I have my own set of simple benchmarks that I'm not going to publish. Everybody can easily prepare such a set - in my case they are various programming tasks that should generate determined output. It is not easy to automatically qualify the quality of code, but at the very least you can filter the results by invalid outputs. With reasonably high number of tasks and their complexity, this can be a fair estimator - provided that it's never published publicly.

Yep, exactly!

My approach is similar - closed source benchmarks with prompts and tests from real LLM-driven products (mostly around boring business automation and enterprise workflows).

Although it would be neat to upgrade the setup to work on the synthetic data. This will at least make benchmarks shareable publicly (not just the results)

I only use gpt4 but to be fair ive read that some people prefer gemini for coding tasks.

Ive used gemini only for vision tasks on document and it was absolute CATASTROPHY (hallucinating the content instead of actually reading it)

The woke part is also extra unhinged on gemini (hating white people is fine but hating other is racist etc...)

Thats not the point of my message tho, the problems could be corrected later. Its more about the product vision

Unless they split the screen into two, it makes no sense at all. These are two very different activities: when I search for "how to implement self-balancing binary search trees in C" it means one thing, when I search for "german history wiki" I mean something completely different and I don't need hallucinating chatbot standing on my way.

The way for Google to beat OpenAI is to recognize AI chatbot usage is going to make a dent in web search usage and further develop Gemini to make it jump from the current GPT-3.5 to 4.0. When combined with a huge context window, they could quickly make people switch and own both web search and AI search. With the current strategy, they just make people migrate away from their web search.

I dont disagree but i think they are currently SUPER LUCKY that open ai isnt going harder on making chatgpt available for more of the general public.

There are still 60 70% people who havent tried it like my parents. They still have a window to stay relevant one way or the other

> when I search for "how to implement self-balancing binary search trees in C" it means one thing

I mean, even then you’d surely prefer to be pointed to an article by an actual human than getting whatever an LLM spits out?

My problem with this is that clearly a big portion of AI Overview's issue is bad training. That is a problem with a straightforward solution. A lot of the publicized cases of bad information actually do have a source on the web, indicating they are not hallucinations.

Wait but when did he say that?

BSaaS?

Judging by how much crap there already is on the web these days, there would seem to be a market for it.

I think it is less of a service than a disservice (I certainly didn't ask, want, or expect misinformation in my search engine results), so BSaaDS would be more appropriate.

Wow, AI sure is turning out great! /s

Reply on news.ycombinator.com