Co-founder of Nimbalyst (https://nimbalyst.com), an open-source visual workspace for building with coding agents like Claude Code, Codex, OpenCode. Previously founded Evergage and sold it to Salesforce. karl@nimbalyst.com
I agree with this. Why get locked in at the harness layer. I want my workspace to be independent of my coding agents and models so I can use multiple and switch as I wish
This is cool! We have a similar approach in Nimbalyst... agent makes extensions that are human editable and agent editable ... includes Jupyter, markdown, csv, excalidraw, code, pdfs, browser. Its so nice not to have to app switch all the time. MIT as well. Maybe we should join forces?
From their abstract: "Our findings reveal that despite continuous development activity and growing codebase complexity of the agent harnesses, there is no statistically significant improvement in SWE-bench benchmark score (i.e., resolve rates of bugs) across releases for a given fixed LLM version. Worse, later agent harness versions consume nearly double the computational tokens and tool calls without corresponding quality gains. We explain this paradox from two angles: at the project level, we identify the development patterns (e.g., feature additions, fix-heavy releases, scattered small changes) correlating with quality fluctuations, while at the architecture level, we localize regressions to specific high-risk architectural components. Our findings call for a new practice of quality assurance in the development of agent harnesses."
Exactly. Expertise is frequently required to steer, correct, rework, know what is good and what isn't. This happens 500 times a day on both the micro level and the architecture level.
I have Max subscription to both Claude Code and Codex. I use both fulltime throughout the day. I have them check each other's work, and route tasks they are best at to the right one.
The checking each other's work is so helpful.
New features, mockups, I use Claude Code. Deep fact-checking, I use Codex.
My harness matters at least as much as the model. I use the best harness and then invest a ton in it
The article is correct to emphasize the importance of definition and that this can be a bottleneck. But it is incorrect to show the layered documentation and development lines taking just as long as they did pre-coding-agents.
Our team is coding much much faster at high quality then we were before coding agents. As this commenter says, if you empower individuals and small teams to define, document and they set up good practices and harnesses, they can accelerate dramatically.
Well said. It is so much better for when the human can see what the LLM has changed and iterate on it with the LLM, making their own changes. Markdown is better for that.
I kind of felt the same way reading the article! It felt so unusual to encounter someone who is both smart and humble and willing to admit they were learning. And I was happy to encounter it and sad that I was so surprised by it.
Congratulations! I think that a lot of the value will be in the judgement of the maintainer about the marginal next feature (and saying no to all those other features) ... if software is a stream then the value is in what gets into the stream. That is an AI resistant value and if you can provide it for your project.
What an excellent article by a smart, humble, still-learning person!
Favorite quote:" There are a whole bunch of reasons I’m not scared that my career as a software engineer is over now that computers can write their own code, partly because these things are amplifiers of existing experience. If you know what you’re doing, you can run so much faster with them. [...]
I’m constantly reminded as I work with these tools how hard the thing that we do is. Producing software is a ferociously difficult thing to do. And you could give me all of the AI tools in the world and what we’re trying to achieve here is still really difficult. [...]"
Consider Nimbalyst, its a free visual workspace for Claude Code and Codex that has visual editing of markdown, mockups, diagrams, code with your agents with WYSYWIG diffs as well as task management and kanban session management tied into your agents. Its got a files/plan/editing mode and an agent/sessions mode.
I'm concerned about these laws and their implications for privacy, but as a parent, I'm not sure what you mean to say parents should parent. How? What should the parent do? How would you recommend a parent protect a 13 year old who spends their time in their room and out with their friends on their phones?
Thanks for sharing. I appreciated getting different perspectives. I come down on ultimately this is lying and I don't see how lying to oneself is going to be helpful.
But to push the analogy a bit. If you are rowing on a lake with motorboats, it is a totally different experience. Noisy, constant wake. We are part of an ecosystem, not isolated.
Growing up, the lakes in New England were filled with sailboats. There were sailing races. Now, its entirely pontoon boats. Not a sailboat to be found.
From my experience in some large tech firms, you could easily cut 20% of the workers and not see much impact. There is so much bloat, process-people, meetings-people, etc... Even if the cuts aren't from AI, execs will use AI as a reason to make these cuts.
Thank you for sharing this. I found some good articles in what you shared. The long lists of places to post are not that helpful. I've poured through 100 of them in the past and only the top 20 make a difference, you might want to update the list to prioritize. I tend to point Claude Code or Codex at these lists, have them evaluate the scores of the sites and give me a priority list.
This is not my experience either. If you put the work in upfront to plan the feature, write the test cases, and then loop until they pass... you can build a lot of high quality software quickly. The difference between a junior engineer using it and a great architect using it is significant. I think of it as an amplifier.