I’m listening to my own experience. Just today I gave it another fair shot. GitHub Copilot agent mode with GPT-4.1. Still unimpressed.
This is a really insightful look at why people perceive the usefulness of these models differently. It is fair to both sides without being dismissive as one side just not “getting it” or how we should be “so far” past debate:
https://ferd.ca/the-gap-through-which-we-praise-the-machine....
https://alexgaynor.net/2025/jun/20/serialize-some-der/ - using Claude Code to compose and have a PR accepted into llvm that implements a compiler optimization (more of my notes here: https://simonwillison.net/2025/Jun/30/llvm/ )
https://lucumr.pocoo.org/2025/6/21/my-first-ai-library/ - Claude Code for writing and shipping a full open source library that handles sloppy (hah) invalid XML
Examples from the past two weeks, both from expert software engineers.
And both of them heavily caveat that experience:
> This only works if you have the capacity to review what it produces, of course. (And by “of course”, I mean probably many people will ignore this, even though it’s essential to get meaningful, consistent, long-term value out of these systems.)
> To be clear: this isn't an endorsement of using models for serious Open Source libraries...Treat it as a curious side project which says more about what's possible today than what's necessarily advisable.
It does nobody any good to oversell this shit.
I linked to those precisely because they aren't over-selling things. They're extremely competent engineers using LLMs to produce work that they would not have produced otherwise.