Quite a big improvement in coding benchmarks, doesn’t seem like progress is plateauing as some people predicted.
By which I mean, I don't find these latest models really have huge cognitive gaps. There's few problems I throw at them that they can't solve.
And it feels to me like the gap now isn't model performance, it's the agenetic harnesses they're running in.
It’s incredibly trivial to find stuff outside their capabilities. In fact most stuff I want AI to do it just can’t, and the stuff it can isn’t interesting to me.
Whether it's genuine loss of capability or just measurement noise is typically unclear.
I wonder what caused such a large regression in this benchmark