The GPT era is already ending | Hacker News Reader

The GPT era is already ending

theatlantic.com

55 pointsbergie28 comments

Please read the article before posting comments, or at least read a summary. The article is saying that GPT-4o style models are reaching their peak, and are being replaced by o1 style models. The article does not make value judgements on the usefulness of existing AI or business viability of AI companies.

I'd suggest that the GPT-4 models are reaching their peak, and that OpenAI is pushing into the o1 reasoning in an attempt to differentiate and regain the mindshare they're losing.

Claude is regarded as better at many things, Gemini at some others, Gemini has massive context windows, Flash runs faster and cheaper, Bedrock runs cheaper, Gemma and Llama are open source and doing pretty well for many use-cases.

If it weren't for mindshare, I'm not sure OpenAI's models would be thought of as anything special at this point. Now it does sound like o1 is a step up, but it also takes a long time and costs a lot, making it quite niche. It's not something you drop in and get better results from.

The author is clearly better informed than the average journalist. However, I think it's strange how the article discusses o1 as though it's some new mystery class of model that no one understands. I mean yes, the exact training recipe isn't known, but it's pretty clear that it is still a GPT model that outputs some chain-of-thought material inside <thinking>...</thinking> tags, which is filtered from the user's view.

> does not make value judgements

So we are not allowed to? The Hacker News gatekeeping instinct is particularly hilarious to me.

You're welcome to make value judgements. The statement you're responding to is a statement about what the article does, not what you should/should not do.

I started skimming about 1/3 through this article. Looks to be just a fluff piece about how cool the old AI models were and how they pale in comparison with what's in the works, with about 2 to 5 lines of shallow 'criticism' thrown in as an alibi?

Ten minutes and a teeny bit of mental real estate I will never get back.

https://archive.ph/xUJMG

> Although you can prompt such large language models to construct a different answer, those programs do not (and cannot) on their own look backward and evaluate what they’ve written for errors.

Given that the next token is always predicted based on everything that both the user and the model have typed so far, this seems like a false statement.

Practically, I've more than once seen an LLM go "actually, it seems like there's a contradiction in what I just said, let me try again". And has the author even heard about chain of thought reasoning?

It doesn't seem so hard to believe to me that quite interesting results can come out of a simple loop of writing down various statements, evaluating their logical soundness in whatever way (which can be formal derivation rules, statistical approaches etc.), and to repeat that various times.

I think this is both true and false in some senses. The whole point of the transformer and why it was so impactful is that it gave the model the ability to introspect on output in this way, and therefore achieve the sorts of results you've seen (that I've also seen), and in general a much more consistent output, and therefore one more likely to be factual, than previous types of generative model.

However, the way in which the author is correct is that there is still functionally one stream, one "thought process", and the introspection is very limited and well defined based on how the model is constructed. If you ask a human a hard question they can take longer thinking about it, and that's not something an LLM can do at all. o1 is one attempt to add further streams of thought as a way to address that limitation, so that the LLM can continue to think and weigh up options through multiple rounds of generation.

You have to think of the next token prediction as the immediate next 'thought'. We have trained LLMs to be forced to immediately say the right thing.

This is akin to how real humans almost immediately will formulate an answer but then think before responding.. Except imagine if someone had a probe that looked into your mind and made you commit to the first thing you thought.

Companies like anthropic literally train their model on the introspected thoughts of the earlier model (constitutional AI). The model is capable of 'thought' if the system doesn't force its hand

I certainly agree on "very limited". (Transformers are linear-time complexity per token, as far as I understand, so from computational theory principles, there are things these things trivially will not be able to answer, as they just "can't take longer thinking about it" as you point out.)

But "very limited" isn't "not at all", and I think the distinction matters, since the former can often be scaled more easily than the latter. And at the meta-level, one possible output token can always be "let me ramble on this a bit more before I provide my final answer".

In any case, I think "they can't look back at their output", as presented in the article, is just the wrong way to look at it; even "they can (currently) not take longer time to look at their output" is also wrong, since they can just produce more output, which then in turn lets them "think more".

Not sure if I got the gist of the article right, but are they trying to say that chain of thought prompting will lead us to AGI / be a substantial breakthrough? Are CoT techniques different to what o1 is doing?? not sure if I'm missing the technical details or if the technical details just aren't there.

The "GPT Era" ended with OpenAI resting on its junky models while Anthropic runs rings around it, but sure, place a puff piece in the Atlantic; at least it's disclosed sponsored content?

And presented in audio narration at the head of the written article: “Produced by ElevenLabs and News Over Audio (Noa) using AI narration. Listen to more stories on the Noa app.”

The article wasn't against the use of AI if that was supposed to seem ironic..

I like AIs with a personality; I like them to shoot from the hip. 4o does this better than o1.

o1 however is often better for coding and for puzzle-solving, which are not the vast majority of uses of LLMs.

o1 is so much more expensive than 4o that it makes zero sense for it to be a general replacement. This will never change because o1 will always use more tokens than 4o.

> I like AIs with a personality

You are confusing training artifacts for "personality."

> could be better for coding and for puzzle-solving, which are not the vast majority of uses of LLMs.

To see a product fail to evolve and merely stratify itself gives a solid hint as to what it's likely future is going to be.

That's not what I mean here by personality. I mean that for everyday chats, I like AIs to freely express its own internal beliefs about something without having to think them through. It should know when to override what I said, to not be a mere robot, and this is where 4o shines.

4o versions have progressively become better at instruction following. I don't think their peak has been reached yet.

AIs do not have "internal beliefs" any more than a textbook does, or a calculator program. ChatGPT is generally spectacularly bad at "overriding what you said"; its output in response to a programming question will be shaped like an answer, accepting the terms as given, without anything resembling a spontaneous attempt to detect an XY problem. Properties like that require "thinking it through", and LLMs don't even encode a model of such thought - at best, they encode, in some abstract, undebuggable way, the meaning of words in context. They have no will or agency. They are not "following" your instruction, nor could they in principle "refuse" to do so. They are merely generating what seems to "come next" logically.

I do agree that these models are lacking significant capabilities to be considered even in the ballpark of humans, and it's unclear if their architecture, scaled up, ever will be, but this strikes me as a very imprecise distinction:

> They are merely generating what seems to "come next" logically.

So are you and I: We're predicting what we'll do next.

By your defined standards, humans aren't much different. This is why your standards don't matter.

With a whimper too, not the anticipated bang.

Insane cope. Emily Bender and Gary Marcus still trying to push "stochastic parrot", the day after o1 causes what was one of the last remaining credible LLM reasoning skeptics (Chollet) to admit defeat.

Push what?

Anti AI grift and FUD and more or less awful takes, cashing in on their credentials to the detriment of their respective institutions.

Yeah they are definitely making a lot of money doing this compared to being on the other side.

It ended because its a glorified search engine now. All of the more powerful functionality was limited or removed

My guess is to sell it to governments and anyone else willing to pay for it.

Source/citations/examples?

Reply on news.ycombinator.com