Hacker News Reader

Comment by sss111 | Hacker News Reader

So far, it seems like this is the hierarchy

o1 > GPT-4o > o3-mini > o1-mini > GPT-4o-mini

o3 mini system card: https://cdn.openai.com/o3-mini-system-card.pdf

I think OpenAI really needs to rethink its product naming, especially now that they have a portfolio where there's no such clear hierarchy, but they have a place along different axis (speed, cost, reasoning, capabilities, etc).

Your summary attempt e.g. also misses o3-mini vs o3-mini-high. Lots of trade-ofs.

Can't wait for the eventual rename to GPT Core, GPT Plus, GPT Pro, and GPT Pro Max models!

I can see it now:

> Unlock our industry leading reasoning features by upgrading to the GPT 4 Pro Max plan.

Had the same problem while trying to decide which Roborock device to get. There's the S series, Saros series, Q Series and the Qrevo. And from the Qrevo, there's Qrevo Curv, Edge, Slim, Master, MaxV, Plus, Pro, S and without anything. The S Series had S8, S8+, S8 Pro Ultra, S8 Max Ultra, S8 MaxV Ultra. It was so confusing.

I ordered the wrong xbox on amazon once. Wanted the series X, got the one X instead

Which one did you pick?

Oh, I'll probably wait for GPT 4 Pro Max v2 NG (improved)

I think I'll wait for the GTI model myself.

ngl I'd find that easier to follow lol

OpenAI chatGPT Pro Max XS Core, not to be confused with ChatGPT Max S Pro Net Core X, or ChatGPT Pro Max XS Professional CoPilot Edition.

Careful what you wish for. Next thing you know they're going to have names like Betsy and be full of unique quirky behavior to help remind us that they're different people.

It's like AWS SKU naming (`c5d.metal`, `p5.48xlarge`, etc.), except non-technical consumers are expected to understand it.

Have you seen Azure VM SKU naming? It's.. impressive.

And it doesn’t even line up with the actual instances you’ll be offered. At one point I was using some random Nvidia A10 node that was supposed to be similar to Standard_NV36adms_A10_v5, but was an NC series for some reason with slightly different letters…

Those are not names but hashes used to look up the specs.

I was thinking we might treat model names analogously, but their specs can be moving targets.

Did they even think about what happens when they get to o4? We’re going to have GPT-4o and o4

They’ll call it GPT-XP. But first we need gpt-o3.11 for workgroups.

They're strongly tied to Microsoft, so confusing branding is to be expected.

It needs to be clowned on here:

- Xbox, Xbox 360, Xbox One, Xbox One S/X, Xbox Series S/X

- Windows 3.1...98, 2000, ME, XP, Vista, 7, 8, 10

I guess it's better than headphones names (QC35, WH-1000XM3, M50x, HD560s).

I can't wait for Project Unify which just devolves into a brand new p3-mini type naming convention. It's pretty much identical to the o3-mini, except the API is changed just enough to be completely incompatible and it crashes on any query using a word with more than two syllables. Fix coming soon, for 4 years so far.

On the bright side the app now has curved edges!

One of my favorite parodies: https://www.youtube.com/watch?v=EUXnJraKM3k

Flashbacks of the .NET zoo. At least they reigned that in.

They can still do models o3o, oo3 and 3oo. Mini-o3o-high, not to be confused with mini-O3o-high (the first o is capital).

They should just start encoding the model ID in trinary using o, O, and 0.

Model 00oOo is better than Model 0OoO0!

You’re thinking too small. What about o10, O1o, o3-m1n1?

Yeah I tried my best :(

I think they could've borrowed a page out of Apple's book, even mountain names would be better. Plus Sonoma, Ventura, and Yosemite are cool names.

Yeah their naming scheme is super confusing, I honestly confuse them all the time.

at least if i ran the company you'd know that

ChatGPTasdhjf-final-final-use_this_one.pt > ChatGPTasdhjf-final.pt > ChatGPTasdhjf.pt > ChatGPTasd.pt> ChatGPT.pt

Did you just ls my /workspace dir? Lol

If this is the hierarchy, why does 4o score so much higher than o1 on LLM Arena?

Worrisome for OpenAI that Gemini's mini/flash reasoning model outscores both o1 and 4o handily.

Is it possible people are voting for speed of responsiveness too?

I suspect people on LLM Arena don't ask complex questions too often, and reasoning models seem to perform worse than simple models when the goal is just casual conversation or retrieving embedded knowledge. Reasoning models probably 'overthink' in such cases. And slower, too.

The LLM Arena deletes your prompt when you restart so what's the point in trying to write a complicated prompt and testing an exhaustive number of pairs?

It's easy to pin this on the users, but that website is hostile to putting in any effort.

This is something I've noticed a lot actually. A lot of AI projects just give you an input field and call it a day. Expecting the user to do the heavy lifting.

o1 on LLM Arena often times out (network error) while thinking. But they still allow you to vote and they make it seem as if your vote is registered.

What about "o1 Pro mode". Is that just o1 but with more reasoning time, like this new o3-mini's different amount of reasoning options?

I have been paying $200 per month for 01-pro mode and I am very disappointed right now because they have completely replaced the model today. It used to think for 1-5 minutes and deliver an unbelievably useful one-shot answer. Now, it only thinks for 7 seconds just like the 03-mini model and I can't tell the difference in the answers. I hope this is just a day 1 implementation bug but I suspect they have just decided to throw the $200 per month customers under the bus so that they'd have more capacity to launch the 03 model for everybody. I can't tell the difference between the models now and it is definitely not because the free 03 model delivers the quality that 01-pro-mode had! I'm so disappointed!

This is why I use the Azure-hosted versions (disclosure: I’m an MS FTE, but I use all sorts of 3rd party models for my own projects) - I _know_ which version is behind each endpoint and when they will be replaced (you can also pin versions within a support window that varies according to model), so I don’t have to rework all my prompts and throw work away at the drop of a hat.

They have been doing this since day 1. I was a subscriber in the first few months (then occasionally every few months to check out the progress) and so many people complained that OpenAI fiddled with the models daily without saying anything to anyone.

It was always attributed to variability but we all know it's not.

o1-pro is a different model than o1.

Are you sure? Do you have any source for that? In this article[0] that was discussed here on HN this week, they say (claim):

> In fact, the O1 model used in OpenAI's ChatGPT Plus subscription for $20/month is basically the same model as the one used in the O1-Pro model featured in their new ChatGPT Pro subscription for 10x the price ($200/month, which raised plenty of eyebrows in the developer community); the main difference is that O1-Pro thinks for a lot longer before responding, generating vastly more COT logic tokens, and consuming a far larger amount of inference compute for every response.

Granted "basically" is pulling a lot of weight there, but that was the first time I'd seen anyone speculate either way.

[0] https://youtubetranscriptoptimizer.com/blog/05_the_short_cas...

I don't think this is true

Is o1-pro not the o1 equivalent of o3-mini-high?

For non-stem perhaps.

For math/coding problems, o3 mini is tied if not better than o1.

I spent some time yesterday struggling with o3-mini-high trying to do a trigonometry problem, we went round and round and could not solve it. o1 solved it in one shot.

That seems very bad. What's the point of a new model that's worse than 4o? I guess it's cheaper in the API and a bit better at coding - but, this doesn't seem compelling.

With DeepSeek I heard OpenAI saying the plan was to move releases on models that were meaningfully better than the competition. Seems like what we're getting is the scheduled releases that are worse than the current versions.

It's quite a bit better than coding --- they hint that it can tie o1's performance for coding, which already benchmarks higher than 4o. And it's significantly cheaper, and presumably faster. I believe API costs account for the vast majority of COGS at most today's AI startups, so they would be very motivated to switch to a cheaper model that has similar performance.

Right. For large-volume requests that use reasoning this will be quite useful. I have a task that requires the LLM to convert thousands of free-text statements into SQL select statements, and o3-mini-high is able to get many of the more complicated ones that GPT-4o and Sonnet 3.5 failed at. So I will be switching this task to either o3-mini or DeepSeek-R1.

I actually switched back from o1-preview to GPT-4o due to tooling integration and web search. I find that more often than not, the ability of GPT-4o to use these tools outweighs o1's improved accuracy.

How would the DeepSeek fit into this?

Or can it not compare? I don't know much about this stuff, but I've heard recently many people talk about DeepSeek and how unexpected it was.

Deepseek V3 is equivalent to 4o. Deepseek R1 is equivalent to o1 (if not better)

I think someone should just build an AI model comparing website at this point. Include all benchmarks and pricing

I had resubscribed to use o1 2 weeks ago and haven't even logged in this week because of R1.

One thing I notice that is huge is being able to see the chain of thought lets me see when my prompt was lacking and the model is a bit confused on what I want.

If I was anymore impressed with R1 I would probably start getting accused of being a CCP shill or wumao lol.

With that said, I think it is very hard to compare models for your own use case. I do suspect there is a shiny new toy bias with all this too.

Poor Sonnet 3.5. I have neglected it so much lately I actually don't know if I have a subscription or not right now.

I do expect an Anthropic reasoning model though to blow everything else away.

R1 servers seem to be down or busy a lot lately.

It’s an amazing model but was so much faster before the hype

The servers being constantly down is the only reason I haven’t cancelled my ChatGPT subscription

Me too actually. I wish I could pay to get priority. I know there are 3rd party providers but I want a chat interface and not fiddle with setting my own.

The thought-stream is very important to me as well.

This one is good: https://artificialanalysis.ai/

Looks like this only compares commercial models, and not the ones I can download and actually run locally.

https://livebench.ai/#/

My experience is as follows:

- "Reason" toggle just got enabled for me as a free tier user of ChatGPT's webchat. Apparently this is o3-mini - I have Copilot Pro (offered to me for free), which apparently has o1 too (as well as Sonnet, etc.)

From my experience DeepSeek R1 (webchat) is more expressive, more creative and its writing style is leagues better than OpenAI's models, however it under-performs Sonnet when changing code ("code completion").

Comparison screenshots for prompt "In C++, is a reference to "const C" a "const reference to C"?": https://imgur.com/a/c-is-reference-to-const-c-const-referenc...

tl;dr keep using Claude for code and DeepSeek webchat for technical questions

no the reasoning models should not directly be compared with the normal models: they often take 10 times as long to answer which only makes sense for difficult questions

I really wish they would open up the reasoning effort toggle on o1 API. o1 Pro Mode is still the best overall model I have used for many complex tasks.

Have you tried the o1-pro mode model today, because now it sucks!

You cannot compare GPT-4o and o*(-mini) because GPT-4o is not a reasoning model.

Sure you can. "Reasoning" is ultimately an implementation detail, and the only thing that matters for capabilities is results, not process.

By "reasoning" I meant the fact that o*(-mini) does "chain-of-thought", in other words, it prompts itself to "reason" before responding to you, whereas GPT-4o(-mini) just directly responds to your prompt. Thus, it is not appropriate to compare o*(-mini) and GPT-4o(-mini) unless you implement "chain-of-thought" for GPT-4o(-mini) and compare that with o*(-mini). See also: https://docs.anthropic.com/en/docs/build-with-claude/prompt-...

That's like saying you can't compare a sedan to a truck.

Sure you can.

Even though one is more appropriate for certain tasks than the other.

It is a nuanced point but what is better, a sedan or a truck? I think we are still at that stage of the conversation so it doesn't make much sense.

I do think it is a good metaphor for how all this shakes out though in time.

Yes you use the models for the same things, and one is better than the other for said thing. The reasoning process is an implementation detail that does not concern anybody when evaluating the models, esp since "open"ai does not expose it. I just want llms to to task X which is usually "write a function in Y language that does W, taking these Z stuff into account", and for that i have found no reason to switch away from sonnet yet.

Why can't you ask both questions (on a variety of topics etc), and grade the answers vs an ideal answer?

Ends before means.

If 4o answered better than o3, would you still use 03 for your task just because you were told it can "reason"?

The point is that you cannot make a general statement that “o1 is better than 4o.”

Yes, but because you need to say exactly what one is better than the other for. Not because o1 spends a bunch of tokens for "reasoning" you cannot even see.

If you would like to see the CoT process visualized, try the “Improve prompt” feature in Anthropic console. Also check out https://github.com/getAsterisk/deepclaude

o-whatever are doing the same thing as any LLM, it's merely that they've been tuned into using a chain of thought to break out of their complexity class (from pattern matching TC0 to pseudo-UTM). But any foundation model with a bit of instruction tuning is going to be able to do this.

OpenAI needs a new branding scheme.

The Llama folk know how. Good old 90s version scheme.

ChatGPT Series X O one

yeah, you can def tell they are partnered with Microsoft.

Reply on news.ycombinator.com