What’s special about it is that it required no handholding; that is new.
My impression is that the base models have not improved dramatically in the last 6 months and incremental improvements in those models is becoming extremely expensive.
Tooling has improved, and the models have. The combo is pretty powerful.
https://aider.chat/docs/leaderboards/ will give you a flavor of the last six months of improvements. Francois Cholet (ARC AGI: https://arcprize.org/leaderboard) has gone from “No current architecture will ever beat ARC” to “o3 has beaten ARC and now we have designed ARC 2”.
At the same time, we have the first really useful 1mm token context model available with reasonably good skills across the context window (Gemini Pro 2.5), and that opens up a different category of work altogether. Reasoning models got launched to the world in the last six months, another significant dimension of improvement.
TLDR: Massive, massive increase in quality for coding models. And o3 is to my mind over the line people had in mind for generally intelligent in, say, 2018 — o3 alone is a huge improvement launched in the last six months. You can now tell o3 something like: “research the X library and architect a custom extension to that library that interfaces with my weird garage door opener; after writing the architecture implement the extension in (node/python/go) and come back in 20 minutes with something that almost certainly compiles and likely largely interfaces properly, leaving touch-up work to be done.
What I haven't seen is any LLM model consistently being able to fully implement new features or make refactors in a large existing code base (100k+ LOC, which are the code bases that most businesses have). These code bases typically require making changes across multiple layers (front end, API, service/business logic layer, data access layer, and the associated tests, even infrastructure changes). LLMs seem to ignore the conventions of the existing code and try to do their own thing, resulting in a mess.
Dumping it at Claude 3.7 with no instructions will 100% get random rewriting - very annoying.