Hacker News Reader

Comment by croemer | Hacker News Reader

The programming task they gave o3-mini high (creating Python server that allows chatting with OpenAI API and run some code in terminal) didn't seem very hard? Strange choice of example for something that's claimed to be a big step forwards.

YT timestamped link: https://www.youtube.com/watch?v=SKBG1sqdyIU&t=768s (thanks for the fixed link @photonboom)

Updated: I gave the task to Claude 3.5 Sonnet and it worked first shot: https://claude.site/artifacts/36cecd49-0e0b-4a8c-befa-faa5aa...

View on HN

Looks like quite shoddy code though. Like, the procedure for running a shell command is pure side-effect procedural code, neither returning the exit code of the command nor its output. Like the incomplete stackoverflow answer it probably was trained from. It might do one job at a time, but once this stuff gets integrated into one coherent thing, one needs to rewrite lots of it, to actually be composable.

Though, of course one can argue, that lots of human written code is not much different from this.

Which code is shoddy? The Claude or o3-mini one? If you mean Claude, then have you checked the o3-mini one is better?

Youtube is currently blocking my VPN, can't watch it.

It's good that it works since if you ask GPT-4o to use the openai sdk it will often produce invalid and out of date code.

But they did use a prompt that included a full example of how to call their latest model and API!

I would say they didn’t need to demo anything, because if you are gonna use the output code live on a demo it may make compile errors and then look stupid trying to fix it live

If it was a safe bet problem, then they should have said that. To me it looks like they faked excitement for something not exciting which lowers credibility of the whole presentation.

They actually did that the last time when they showed the apps integration. First try in Xcode didn't work.

Yeah I think that time it was ok because they were demoing the app function, but for this they are demoing the model smarts

Models are predictable at 0 temperatures. They might have tested the output beforehand.

Models in practice haven't been deterministic at 0 temperature, although nobody knows exactly why. Either hardware or software bugs.

We know exactly why, it is because floating point operations aren't associative but the GPU scheduler assumes they are, and the scheduler isn't deterministic. Running the model strictly hurts performance so they don't do that.

Cool, thanks a lot for the explanation. Makes sense.

Sonnet isn't a "mini" sized model. Try it with Haiku.

How mini is o3-mini compared to Sonnet and why does it matter whether it's mini or not? Isn't the point of the demo to show what's now possible that wasn't before?

4o is cheaper than o1 mini so mini doesn't mean much for costs.

What? Is this what this is? Either this is a complete joke or we're missing something.

I've been doing similar stuff in Claude for months and it's not that impressive when you see how limited they really are when going non boilerplate.

heres the right timestamp: https://www.youtube.com/watch?v=SKBG1sqdyIU&t=768s

Yeah I agree that wasn't particularly mind blowing to me and seems fairly in line with what existing SOTA models can do. Especially since they did it in steps. Maybe I'm missing something.

Reply on news.ycombinator.com