Hacker News Reader

Comment by light_hue_1 | Hacker News Reader

What is Google doing?

Gemini is terrible. It's way worse than even GPT 3. Never mind 3.5 or Claude. It's basically useless. Even the simplest things like trivial code transformations don't work. Gemini goes rogue all the time and starts to do things it shouldn't.

I get the feeling that in desperation people at Google are hacking the metrics to make their model look good. While in reality it's just junk.

No model, and I've tried a lot of them, has such a massive gap between good benchmark performance and horrible real world performance.

View on HN

Honestly this sounds like you tried one unrepresentative query at launch and nothing else. I’m using Gemini regularly now and it’s not bad. Via the API I’ve had even more success. Much better than 3.5, and competitive with 4 in the things I’ve used it for, plus a huge context window.

I agree with you. For business applications, context window is more important than any extra reasoning ability. I can load up nearly all business documents, regulations and rules and ask it questions and the responses are accurate and better that models with small context window.

> and the responses are accurate

If you knew this ahead of time then what value did it provide you? Put another way, once the counter party realizes that you've pushed this responsibility onto an LLM, aren't you worried that they could take advantage of this fact to produce intentionally misleading query results?

In an adversarial world what long term value could possibly exist here?

>> If you knew this ahead of time then what value did it provide you?

I think I should have clarified that the responses were accurate when we bench-marked them against RAG applications built on other LLMs. If the results had been poor or below a set threshold, the RAG application built on Gemini Pro would have been shelved.

>> Put another way, once the counter party realizes that you've pushed this responsibility onto an LLM, aren't you worried that they could take advantage of this fact to produce intentionally misleading query results?

This is an excellent point but inapplicable in our use case as this was only meant for internal users, who otherwise would be opening word and PDF documents and searching through them or relying on their own memory.

Reply on news.ycombinator.com