Hacker News Reader

Comment by rafaelero | Hacker News Reader

Does this guy really believe that GPT-4's answers are of the same quality as GPT-3.5? That's ridiculous.

I agree that GPT4 is better, but the amount of times it still hallucinates is still way too high to be useful for any real work. LLMs at present are super impressive, but have very very low trust.(This is true for all models, Claude, Opus, etc.)

More than 60% of programmers are already using it and finding it useful. Boston Consulting Group conducted a trial and concluded that it increases quality of their consultants work while decreasing the time it took to finish it. So no, you are not correct that hallucination makes it impossible to use.

> More than 60% of programmers are already using it

Firstly, there is absolutely no way anyone could know this. Secondly, no they aren't.

It's what surveys are showing. Why would they lie?

Hallucination, in my opinion, is mostly down to bad prompting.

Just as one can say, in response to a perfectly correct answer, "Explain what's wrong with your answer" and it will find something, one can either create a vacuum it feels compelled to fill with hallucinations or ask in a way it stays grounded.

It's pretty dependant on the questions you ask. Obviously when you push them to their limits, GPT 4 goes a lot further. But what about for your average queries though? I don't have the data to make a proper claim about that.

> It's pretty dependant on the questions you ask.

That’s the description of a house of cards rather than some revolutionary new technology.

That's why we have Lmsys for. And spoiler alert, it performs better than 3.5 no matter the complexity of the question.

And are you sure people ask the chatbot arena the same caliber of questions they ask a single model? Wouldn't you tend to ask more difficult questions to find where one fails but the other doesn't? I'm not convinced that the average query will get a significantly better result from GPT 4. I only know that is true for more difficult queries.

Do you have access to ChatGPT? You can try it yourself (you can query GPT-4o and GPT-3.5 afterwards). The difference is large enough to be noticeable at least 80% of the time in my experience.

Recently I got free 30 minute trial access to GPT4 (and was prompted to upgrade if I liked it).

I used the opportunity to send some queries which GPT-3.5 was bad at, and while GPT4 was better and closer to the correct answers, it still failed to produce them. When I pointed out the problems, it got even closer, but still ended up insisting that its (wrong) answer was correct in the end.

So, yeah, GPT4 is much better than GPT3.5, but it's got a long way to go before it becomes real AI.

Currently, I still use it for 3 cases:

1. When exploring something completely new. It saves time reading and learning because you can ask questions with "layman words" and get to learn the specific terms used in the domain quickly.

2. When google search is swamped by SEO spam or google insists on ignoring some of the keywords

3. When I need some boilerplate code that has been written a thousand times. It saves time which would otherwise get spent on concatenating different code segments from 5 different Stackoverflow threads and reading the docs afterwards to ensure it's correct. Now I just copy/paste from ChatGPT and read the docs afterwards to fix any places where it messed up.

You're doing it wrong if you are not feeding Chat the documentation first.

It's not as great a gpt4O, but with gemini and 1M tokens (2M now also avaiable) you can dump in a small library of "writing javascript for X" books before asking it to produce code for X. It dramatically improves output.

ChatGPT also can be front loaded with documents, but the context is much smaller (32K in the web app, I believe?)

> So, yeah, GPT4 is much better than GPT3.5, but it's got a long way to go before it becomes real AI.

I don't disagree. Just wanted to highlight the poor intuition of the guy doing the video.

Reply on news.ycombinator.com