Hacker News Reader

Rumors said that GPT4.5 is an order of magnitude larger. Around 12 trillion parameters total (compared to GPT4's 1.2 trillion). It's almost certainly MoE as well, just a scaled up version. That would explain the cost. OpenAI also said that this is what they originally developed as "Omni" - the model supposed to succeed GPT4 but which fell behind expectations. So they renamed it 4.5 and shoehorned it in to remain in the news among all those competitor releases.

This is all excellent detail. Wondering if there's any good suggestions for further reading on the inside baseball of what happened with GPT 4.5?

Well, it's not...it gets most details wrong.

Can you elaborate?

GPT-4 was rumored to be 1.8T params...not 1.2

And the successor model was called "Orion", not "Omni".

Appreciate the corrections, but I'm still a bit puzzled. Are they wrong about 4.5 having 12 trillion parameters, it originally intending to be Orion (not omni), or an expected successor to GPT 4? And do you have any related reading that speaks to any of this?

GPT-4 was 1.3T. 221B active. 2 experts active. 16 experts total.

That sounds correct except for that total parameter count. 110B per expert at 16 experts puts you just shy of 1.8T. Are you suggesting there are ca. 30B shared params between experts?

GPT-4 was rumored to be 1.8T params...not 1.2

And the successor model was called "Orion", not "Omni".

the gpt-4o ("omni") is probably a distilled 4.5; hence why not much quality difference

4o has been out since May last year, while omni (now rechristened as 4.5) only finished training in October/November.

4.5 was called Orion, not Omni.

You're thinking of "Orion" not "Omni" (GPT 4o stands for "Omni" since it's natively multimodal with image and audio input/output tokens)

How does this compare with Grok 3's parameter count? I know Grok 3 was trained on a larger cluster (100k-200k) but GPT 4.5 used distributed training.

>The price is a bit of a mystery

I think it at least is somewhat analogous to what happened with pricing on previous models. GPT 4, despite being less capable than 4o, is an order of magnitude more expensive, and comparably expensive to o1. It seems like once the model is out, the price is the price, and the performance gains emerge but they emerge attached to new minified variations of previous models.

I don't think th October 2023 training cut-off means the model finished pre-training a year ago. All of OpenAI's models share that same cut-off date.

One theory is that they're worried about the increasing tide of LLM-generated slop that's been posted online since that date. I don't know if I buy that or not - other model providers (such as Anthropic, Gemini) don't seem worried about that.

Releasing it was probably a mistake. In context what the model is could have been understood, but they haven’t really presented that context. Also it would be lost on a general audience.

The general public will naturally expect it to be the next big thing. Wasn’t that the point of releasing it? To seem like progress is being made? To try to make that point with a model that doesn’t deliver is a misstep.

If I were Sam Altman, I’d be pulling this back before it goes on general release, saying something like it was experimental and after user feedback the costs weren’t worth it and they’re working on something else as a replacement. Then o3 or whatever they actually are working on instead can be the “replacement” even if it’s much later.

or just say it was too good and thus too dangerous to release...

I sort of believed this but also 4.5 coming out last year would absolutely have been a big deal compared to what was out there at the time? I just dont understand why they would not launch it then.

> slew of strong non-reasoning models (Gemini 2.0 Flash, Grok 3, Sonnet 3.7)

Sonnet 3.7 is actually reasoning model.

It's my understanding that reasoning in Sonnet 3.7 is optional and configurable.

I might be wrong but I couldn't find a source that indicates that the "base" model also implements reasoning.

From limited experimentation: Sonnet 3.7 has “extended thinking” as an option, although the UI, at least in the app, leaves something to be desired. It also has a beta feature called “Analysis” that seems to work by having the model output JavaScript code as part of its response that is then run and feeds back into the answer. Both of these abilities are visible — users can see the chain of thought and the analysis code.

It seems, based again on limited experimentation doing sort-of-real work, that analysis works quite well and extended thinking is so-so. Whereas DeepSeek R1 seems to be willing and perhaps even encouraged to second-guess itself (maybe this is a superpower of the “wait” token”), Sonnet 3.7 doesn’t seem to second-guess itself as much as it should. It will happily extended-think, generate a wrong answer, and then give a better answer after being asked a question that it really should have thought of itself.

(I’m not complaining. I’ve been a happy user of 3.7 for a whole day! But I think there’s plenty of room for improvement.)

so is grok3

> GPT 4.5 also has a knowledge cutoff date of 10-2023.

I've heard the theory that's intentional, because after roughly that point the internet training data became full of AI slop