osti

I heard they are already proficient at assembly languages.

A lot of it was beyond me, but this was all the branch names for all the stuff it tried, most of it unsuccessful of course. About 10x perf improvement came from architectural changes, and then 2x from micro optimizations.

https://pastebin.com/eac0SAYg

> propose, implement, measure, keep the wins

Pretty much what I did to let Codex with gpt5.4xhigh improve my fairly complex CUDA kernel which resulted in 20x throughput improvement.

This was published two months ago. Even though it was at a time that open source models are publishing comparable swe bench scores.

Can't they write a script to solve rubik cubes?

True, but I think for local models, we are mostly considering personal usage.

Yeah... I would definitely call 2t/s unusable. For simple chats, I'd want at least 15 t/s. For agentic coding (which this model is advertised for), I'd want good prefill performance as well.

Oh i’m fully aware of that lol

I think this one is only about 600GB VRAM usage, so it could fit on two mac studios with 512GB vram each. That would have costed (albeit no longer available) something like less than 20k.

Maybe open source == communism

I can't vouch for whether or not it can beat human experts though because I'm no CUDA expert myself. The original CUDA code were human written and I first let codex adapt it to my specific use case. Then I basically let codex generate ideas and try the ideas out itself (I think it's a bit like Karpathy's autoresearch, except I was still doing manual prompting). And that was enough to get me 20x improvement.

I suspect when people said AI wrote non performant CUDA kernels it was beginning-mid last year and it's definitely vastly improved since back then. And the agent's ability to iteratively improve really impressed me.

I was using codex cli with 5.4xhigh. So it was able to iteratively improve from simple prompts on my part (can you give some architectural ideas to improve the performance? And once it does, I just say can you implement and benchmark it).

I think it was a bit like Karpathy's autoresearch, except I was doing manual promoting... Though I feel I could definitely be removed from that equation.

For me it was able to try out different architectures for perf improvement, then once it's settled on some good architectures, it can do lower level optimizations on them by profiling the code etc.

Yup I've mentioned this in another thread, I got gpt 5.4xhigh to improve the throughout of a very complex non typical CUDA kernel by 20x. This was through a combination of architecture changes and then do low level optimizations, it did the profiling all by itself. I was extremely impressed.

I had a very complex cuda kernel and codex cli managed to improve the throughout 20x.

But is arc-agi really that useful though? Nowadays it seems to me that it's just another benchmark that needs to be specifically trained for. Maybe the Chinese models just didn't focus on it as much.

That is ture, but the revenue of the artisanal stuff is probably only a very low percentage of the overall market, which would imply a lot of software engineers would have to exit the field. Which is what we here don't want to see.

Doesn't the chat version of chatgpt or gemini also have interleaved tool calls, so do those also count as with harnesses?

Seems like the high compute parallel thinking models weren't even needed, both the normal 5.4 and gemini 3.1 pro solved it. Somehow Gemini 3 deepthink couldn't solve it.

Actually yeah, I shouldn't have said that regex is a microbenchmark, it's indeed an important one.

There's a fine line between making ppl civilized and fascism-like level of control. And I believe Japan errs on the other side too much with their ridiculous number of such rules in all areas of life. Even though I recently visited Japan, I can't really speak to how happy they are, but the stereotype is that they are not the happiest ppl out there. I believe their obedience to all such societal rules has a role in it.

[flagged]

More like oppressed people by all those bs rules.

I took a look, it's not bad but it seems to contain too many micro benchmarks like regex or primes. Geekbench at least has clang which is a subscore that I always look at.

Not true. Geekbench, especially single threaded benchmark, is probably the best we got, it has a bunch of workloads, unlike many other benchmarks like cinebench for example. And they publish all the results on their website, so you can dig into each individual workload and find the ones that apply to you.

And like the other poster mentioned, it correlates well with SPEC, so it's basically a easily accessible SPEC. These days the only benchmark I use to quickly judge some CPU is geekbench.

It's only that one number that is for sonnet.

Their company is called Anthropic after all.

ByteDance never really open sourced their models though. But I agree, they will only open source when it doesn't really matter.

That's what I found with some of these LLM models as well. For example I still like to test those models with algorithm problems, and sometimes when they can't actually solve the problem, they will start to hardcode the test cases into the algorithm itself.. Even DeepSeek was doing this at some point, and some of the most recent ones still do this.

So now you don't want capitalism?