Does code cleanliness affect coding agents? A controlled minimal-pair study
arxiv.org
arxiv.org
Worse, they don't control for breaking the application's tests.
> Pass rate scores the agent’s final state against the hidden tests we wrote for each task. We do not check whether the agent broke unrelated tests already present in the repository, and a cleaner-side and messier-side solution that both pass the hidden test may still differ on tests they were not graded on.
Any conclusions with respect to token consumption seems pretty meaningless if we're not controlling for the quality of the final output.
Essentially it's an absolutely worthless study then. This AI fatigue got boring a long time ago, this is just painful now.
What we didn't do (stupid oversight on my part) was to ensure that there are no regressions in the remaining tests in the repo (unrelated to the current task). In practice, when using Sonnet 4.6 IRL, I don't see a lot of regressions because often the agent runs the test before calling it done. But it could have gone either way. We don't know.
That's not to say the study is good, but I can respect their decision because the tests passing isn't necessarily correlated with the effectiveness of the agent.
That's pretty much what we did. We start with a repository, and
- (vibeclean pipeline) make changes to clean analyzer issues (some of them involve moving big code chunks around to reduce cognitive complexity), and tweak the existing test sets when needed.
- (slopify pipeline) inverse, add analyzer issues and increase complexity, add dead code etc.
In both cases, we ensure that the test coverage remains the same, and the tests are passing, before we start our experiments.
The most useful discussion would be if we all read the paper and critique its methodology or results.
though people who complain that llms aren't that great strike me as the type to have messy code bases
You can get the LLM to run a script which checks for all of these and also enforce them by running the same script as a pre-commit hook. Setting this up religiously in every code base I work on has been what's given me the most mileage with agentic coding.
I wrote down a more detailed post of the various linters I use here:
https://www.balajeerc.info/Use-Deterministic-Guardrails-for-...
I have legacy endpoints that are no longer used in practice, there for historical reasons, intertwined with existing code etc. They might be marked obsolete, services implementing it are not - agent greps those, builds off of them - produces half legacy garbage.
Linters only handle trivial cases most of us already solved.
// LEGACY CODE, per docs/legacy_rules.md §14, §19Most of the time. Except for when it forgets to do it.
I think it’s funny that the solution is to use something that is not LLM driven to enforce it.
Also - pre commit hooks aren’t enforced, people will not set them up. You have to run this stuff in CI (which is incredibly annoying given that machines are writing the code in the first place)
Putting structural code checks in a precommit hook is arguably better than pulling it into the harness, as it will enforce those constraints no matter whether an agent or human is making the commit.
> what if your harness hook malfunctions?
That’s a bug in the harness and should be enforced
Unless the agent, or the human don't enable the precommit hooks in the first place.
Adding an auto runner of the unit tests is just... Boring?
The reason is quite obvious if you have dealt with such a huge code basis in production with thousands of developers contributing for decades to it coming from different vendors and countries.
Code has a meaning attached to it. And paradoxically being able to cleanly cut out such dead code raises my suspicion. There is a reason why such code exists in there often times and since almost always stakeholders give a damn about documentation, and developers traditionally have a hard time writing even JavaDocs, JSdocs, whatever and not to mention maintaining them.
In earlier times CPU time was precious and comments were deliberately left off due to space and processing considerations.
So why is this all important?
Because until you cannot find the one guy who uses this code for a good reason, I would never kill it. Good reasons in these cases are almost always so called application owner, an app admin and hosts, who serves according to ITIL specs as deployment and production person.
Dead code can be actually quite lively under the right circumstances. And since sometimes people have to be very creative to serve regulation requirements and compliance, sometimes release pressure or missing tools can make such code an important script or deployment tool or fix for a reboot or whatever.
Believe me, dead code isn't. What you can do is, watch it at least over a period of two years.
Here is why.
Most processes have yearly deadlines. Many fall on the 1.1. of each year, while others somewhere at the end of the year. Some processes need to be served once a year due to compliance to laws.
And why two years then? As I said, human beings. Maybe that one time a guy had an exception running for it or there is a maintainer, who uses a different method - his own dead code so to say - because these scripts are rarely shared and maintainer's best kept secrets. The lesser a company knows, the more important these last line folks feel and they are blackboxes when something is or isn't working. (I hated this, this was not way of working and I changed it. It is not their company and a keeper is a Red Flag for me.)
But rarely the same person will serve the process two times in a row. Vacation times vary as well as positional changes.
Hence the two years period and even then, there are smarter, more easier ways than to use linters.
Hint: it is the frontend first paradigm.
How do I know? Because I invented it. Proof: Huge international Bank. dbCORE.
You’ve never had an agent completely lose the plot and forget/confuse its instructions due to the context filling up?
I guess it depends on what you mean with "better" but almost all the agent-built projects I do with zero regards to code quality, design and architecture ends up with every single agent needing 10+ minutes to do even the easy changes, while the ones where I focused on those things together with the agent, large changes can take 10+ minutes but everything else is solved faster.
I don't have empirical evidence of this yet, I guess I should put together some sort of test to confirm/disconfirm this.
Refactor the Python code to make it more Pythonic, e.g. fewer classes/singletons, especially if it will provide a speedup. The Python code **MUST** follow code organization standards expected of popular open-source Python packages code without causing any benchmark performance regressions.
The Rust codebase in `/src` has become bloated with several files >1k LoC. Refactor the Rust codebase to fit code organization standards expected of popular open-source Rust code without causing any benchmark performance regressions.
In terms of benchmark performance it generally improves after the refactor which I suspect is coincidental (especially in Rust where it shouldn't matter due to compiling) but I'm not complaining.
(Can we not play language police? It's boring and doesn't lead to interesting discussion.)
Also, as evident from the different tones in these two prompts, my prompts are often freeform.
“Write perfect code, make no mistakes”
I use this one in my Ralph Harness all the time, it’s a classic!
It’s not that it can’t do that, it’s just that you haven’t told it to!
It’s not about making mistakes, it’s about telling it not to make mistakes!
I've noticed a thing where by default, agentic coding tools are reluctant to remove code. Even when you tell them to. It will bend over backwards to keep old code around, to add complexity for allowing that code to still be called, etc. Super annoying if you are basically just prototyping. You basically end up with a lot of dead code, which than confuses things when you try to add to it. But once you know this, you can just ask it to get rid of the legacy stuff.
Keeping the code base clean, actually stimulates AIs to do the right thing. If there are lots of tests, it will add more when creating new functionality. If there's documentation, it will update that without needing to be prompted as well.
As code harnesses improve, a lot of this is probably being built in as well. Which means even less experienced prompters can get decent results.
I have my code styling rules in my CLAUDE.md already anyway
Well... that is quite different than just saying "refactor the codebase".
This feels like a terrible approach, sufficient to condemn the entire study.
Apparently half of the "minimal pairs" in this work were constructed in this way. I simply am not going to trust any conclusion that requires assuming these AI "cleaned" repos are in any way representative of actually-good codebases.
I'm not a huge fan of their methodology for the AI-degraded cases either (ideally one would set up the mirror pairs by taking some real repositories and rewinding history a month or so and then having a succession of independent agents reimplement each bit of feature work and bugfixes over that period of time), but it's at least a coarse approximation whereas I just don't trust the cleanup methodology to resemble anything real in the first place.
In fact, there are some papers (that we cited) which create a set of tasks doing exactly this, and it is non-trivial [1].
Using LLMs to rewrite code to remove these violations is a rather accepted practice. Sonar's existing one-shot LLM based approach [1] (in production since 1+ year), and a recent agentic approach [2] to do the same work rather well to do this.
[1] https://www.sonarsource.com/solutions/ai/ai-codefix/
[2] https://www.sonarsource.com/products/sonarqube/remediation-a...
The minimal-pair design is honestly one of the strengths, as it tries to isolate the cleanliness from other factors (e.g. architecture, dependencies, tests) which is more rigorous than comparing repos. However, using LLM-generated "sloppified" code (rather than some kind of mechanical or human-guided) is a bit questionable.
I'd say the biggest critique which others have correctly mentioned here is the authors' choice to not check the full test suite. The claim of "behavioral equivalence" is only as good as the tests and coverage.
This hypothesis is compelling for two reasons: 1) it makes sense (garbage in -> garbage out as it were) since LLMs will mimic what they see in the codebase; 2) it matches what many engineers feel somewhat intuitively over the last year or two of using these models. Greenfield is almost always easier than joining a busy codebase. The mess comes in complex integration, and maintaining a system for legacy purposes, etc.
So I personally at a minimum will want to talk “in code” about what code does.
Based on that, it should be a pretty unsurprising conclusion as long as the code quality metrics you are using are reasonable; as long as the quality metric is good (within the context of coding agents), then this is the result we'd expect to see.
1. clean structure means the AI can predictably work because i want deterministic output.
2. basic things like following SRP leads naturally to layered code which makes testing new features and fixing bugs super simple.
3. reviewing code is simpler, because i see ai agents as "servants" i command to do something and thus an overseer is still required
All of these things have historically been the job of engineers, because it helps other people contribute to the code.
Now it helps other people and other agents contribute to the code.
If only we cared so much for junior developers.
You can reduce a lot of the token use for traversal by giving your agent access to some form of LSP in addition to hierarchical direction with your AGENTS.md (or equivalent) for monorepos - but a spread-out codebase is always going to end up requiring some form of traversal to solve each task.
And that traversal isn't just token use - its repeated round trip latency (LLM (queue time -> prefill -> decode -> output) -> Agent (parsing -> tool call -> tool response) -> back to LLM) for EACH step (well, some can be done in parallel, but in practice its mostly sequential) - slowing down the task considerably.
Locality and structure are key when it comes to efficient use of agents. The context window is always bounded and attention across it is inconsistent.
At work we had largely moved off of some old hardware but kept partial support for it in the code because normally that would be zero cost. But the constant overhead of bullying, bribing, and threatening the LLMs to leave that code alone was way more work than just excising it and reimplementing it if needed.
I'm not sure if you would call it an objective or subjective measure, a fixed model would be consistent which would provide an objective base for comparison, but other models would be different, so it would be subjective in relation to the model itself.
With coding agents, agents can produce code quicker. The same trade-off still applies.. but, the time it takes an LLM coding agent to write well organised code is still going to be quicker than the time it takes me to write scrappy code.
You take on a lot of tech debt. Then you need to do the same work you would do with any legacy app: finding where the brittle points are, what needs better testing, which leads to breaking apart the big ball of mud into cleaner components.
https://github.com/bjcoombs/ai-native-toolkit/tree/main#what...
In AI labs, they can run a task repeatedly for extended periods, eventually marking it as completed in benchmarks. But in the real world, completing tasks quickly and well remains important and valuable.
Have you even seen one ever?
Also, not checking if it breaks unrelated tests is wild, good software is written by modifying the "just right" amount of code to get your result.
If you break _unrelated_ tests than you most likely didn't changed the right amount of code.
Idk I call bs.
I actually don't see a connection between the mechanism used to create the code and the code messiness. Things like code repetition, incorrect level of abstractions, tests testing only tests themselves, using too smart optimizations for things that don't matter, .. These all can happen in both person- and machine -authored code.
I would be surprised if a professional software developer has never seen at least some aspects of messy codebase in most any large project. The difference can be whether this messiness ever managed, or just piled on.