Hacker News Reader

Comment by yokoprime | Hacker News Reader

It does give an impression of diminishing returns on this family of models. The output presented in samples and quoted in benchmarks is very impressive, but compared with 4o it seems to be an incremental update with no new capabilites of note. This does, however, come at a massive increase in cost (both in terms of compute, but also monetary). Also, this update did not benefit from being launched in the wake of Claude 3.7 and DeepSeek, which both had more headlining improvements compared with what we got yesterday from OpenAI

View on HN

Why would you not expect diminishing returns? Surely that’s the default position of all AI non-experts?

I thought we were supposed to see AI managing costs reductions by now. Instead I just see chatbots and frantic PMs looking to prove their utility.

I don't know why people would downvote you.

ChatGPT was released in 2022! It doesn't feel like that, but it's been out for a long time and we've only seen marginal improvements since, and the wider public has simply not seen ANY improvement.

It's obvious that the technology has hit a brick wall and the farce which is to spend double the tokens to first come up with a plan and call that "reasoning" has not moved the needle either.

I build systems with GenAI at work daily in a FAANG, I use LLMs in the real world, not in benchmarks. There hasn't been any improvement since ChatGPT first release and equivalent models. We haven't even bothered upgrading to newer models because our evals show they don't perform better at all.

That's obviously untrue... Yes it hasn't hit the usable threshold for human replacement for many tasks, the majority of tasks, but that doesn't mean it hasn't improved massively for many others. Your evals seem to just be not fine enough, probably you're just waiting for agi. I do evals on a ton of tasks and my API/chatgpt usage has shot up drastically. It's now irreplaceable tool for me.

> It's obvious that the technology has hit a brick wall and the farce which is to spend double the tokens to first come up with a plan and call that "reasoning" has not moved the needle either.

If nothing else, that technique has cut down drastically on hallucinations.

"Hallucinations" (ie a chatbot blatantly lying) have always struck me as a skill issue with bad prompting. Has this changed recently?

to a skilled user of a model, the model won't just make shit up.

Chatbots will of course answer unanswerable questions because they're still software. But why are you paying attention to software when you have the whole internet available to you? Are you dumb? You must be if you aren't on wikipedia right now. It's empowering to admit this. Say it with me: "i am so dumb wikipedia has no draw to me". If you can say this with a straight face, you're now equipped with everything you need to be a venture capitalist. You are now an employee of Y Combinator. Congratulations.

Sometimes you have to admit the questions you're asking are unlikely to be answered by the core training documents and you'll get garbled responses. confabulations. Adjust your queries accordingly. This is the answer to 99% of issues product engineers have with llms.

If you're regularly hitting random bullshit you're prompting it wrong. Models will only yield results if they get prompts they're already familiar with. Find a better model or ask better questions.

Of course, none of this is news to people who actually, regularly talk to other humans. This is just normal behavior. Hey maybe if you hit the software more it'll respond kindly! Too bad you can't abuse a model.

A sufficiently skilled person can use a warped and bent slide rule to design a jet engine.

But that doesn't mean the warped slide rule and a super computer capable of finite element analysis are equally useful or powerful.

You can easily crush a person with a super computer. But what are you going to do with a slide rule—lightly tap someone?

You are lacking in imagination.

(Alas, Dall-E also lacks, so I couldn't generate a picture of deadly slide rule kung fu. At least none that wasn't unintentionally hilarious.)

I also work with genAI daily. to say there's no improvement between gpt3 and gpt4 and gpt4o is massively incorrect. the improvement in the latter two was measurable and obvious. Just the speed of answer alone is improved by an order of magnitude.

The boundaries of LLMs’ capabilities are really weird and unpredictable. I was doing a very basic info extraction task a few days ago and none of Gemini 2.0 Flash, Llama 3.3 70B, or GPT-4o could do it reliably without going off the rails. With the same prompt, I switched to the open-weights Gemma 2 27B - released last spring - and it nailed it.

Is this something that structured propmpts a la DSPy are designed to address?

GPT 2 has been out for more than 6 years.

Reply on news.ycombinator.com