[flagged]
If the abstraction that the code uses is "right", there will be hardly any edge cases, and something to break three layers deep.
Even though I am clearly an AI-hater, for this very specific problem I don't see the root cause in these AI models, but in the programmers who don't care about code quality and thus brutally reject code that is not of exceptional quality.
Is there a typo here? If they don't care about code why would they reject code based on quality?
Indeed an accidental omission by me:
programmers who don't care about code quality and thus don't brutally reject code that is not of exceptional quality.
This is what tests are for. Humans famously write crap code. They read it and assume they know what's going on, but actually they don't. Then they modify a line of code that looks like it should work, and it breaks 10 things. Tests are there to catch when it breaks so you can go back and fix it.
Agents are supposed to run tests as part of their coding loops, modifying the code until the tests pass. Of course reward hacking means the AI might modify the test to 'just pass' to get around this. So the tests need to be protected from the AI (in their own repo, a commit/merge filter, or whatever you want) and curated by humans. Initial creation by the AI based on user stories, but test modifications go through a PR process and are scrutinized. You should have many kinds of tests (unit, integration, end-to-end, regression, etc), and you can have different levels of scrutiny (maybe the AI can modify unit tests on the fly, and in PRs you only look at the test modifications to ensure they're sane). You can also have a different agent with a different prompt do a pre-review to focus only on looking for reward hacks.
And tests are inherently imperfect, they may not test the perfect layer, so they break when they shouldn't, and they certainly don't capture every premise.
I'm on board with the tactics you suggest, but they are only incrementally helpful. What we really need is AI that removes duplicative code and unnecessary tests.
I will imagine that in the future this will be tackled with a heavy driven approach and tight regulation of what the agent can and cannot touch. So frequent small PRs over big ones. Limit folder access to only those that need changing. Let it build the project. If it doesn't build, no PR submissions allowed. If a single test fails, no PR submissions allowed. And the tests will likely be the first if not the main focus in LLM PRs.
I use the term "LLM" and not "AI" because I notice that people have started attributing LLM related issues (like ripping off copyrighted material, excessive usage of natural resources, etc) to AI in general which is damaging for the future of AI.
I think you have that backwards.
The resource and copyright concerns stem from any of these "AI" technologies which require a training phase. Which, to my knowledge, is all of them.
LLMs are just the main targets because they are the most used. Diffusion models have the same concerns.
Now I first discuss with an AI Agent or ChatGPT to write a thorough spec before handing it off to an agent to code it. I don’t read every line. Instead, I thoroughly test the outcome.
Bugs that the AI agent would write, I would have also wrote. Example is unexpected data that doesn’t match expectations. Can’t fault the AI for those bugs.
I also find that the AI writes more bug free code than I did. It handles cases that I wouldn’t have thought of. It used best practices more often than I did.
Maybe I was a bad dev before LLMs but I find myself producing better quality applications much quicker.
I don't understand, how can you not fault AI for generating code that can't handle unexpected data gracefully? Expectations should be defined, input validated, and anything that's unexpected should be rejected. Resilience against poorly formatted or otherwise nonsensical input is a pretty basic requirement.
I hope I severely misunderstood what you meant to say because we can't be having serious discussions about how amazing this technology is if we're silently dropping the standards to make it happen.
I don't understand, how can you not fault AI for generating code that can't handle unexpected data gracefully?
This is likely the future.
That being said: "I used to spend most of my time writing code, fixing syntax, thinking through how to structure the code, looking up documentation on how to use a library.".
If you are spending a lot of time fixing syntax, have you looked into linters? If you are spending too much time thinking about how to structure the code, how about spending some days coming up with some general conventions or simply use existing ones.
If you are getting so much productivity from LLMs, it is worth checking if you were simply unproductive relative to your average dev in the first place. If that's the case, you might want to think, what is going to happen to your productivity gains when everyone else jumps on the LLM train. LLMs might be covering for your unproductivity at the code level, but you might still be dropping the ball in non-code areas. That's the higher level pattern I would be thinking about.
I’ve always been more of a product/business person who saw code as a way to get to the end goal.
That elite coder who hates talking to business people and who cares more about the code than the business? Not me. I’m the opposite.
Hence, LLMs have been far better for me in terms of productivity.
That’s what code always is. A description on how the computer can help someone faster to the end goal. Devs care a little more about the description, because end goals change and rewriting the whole thing from scratch is costly and time-consuming.
> That elite coder who hates talking to business people and who cares more about the code than the business? Not me. I’m the opposite.
I believe that coder exists only in your imagination. All the good ones I know are great communicators. Clarity of thought is essential to writing good code.
I believe that coder exists only in your imagination. All the good ones I know are great communicators. Clarity of thought is essential to writing good code.
These coders may exist, but they are in my experience not that common. Most coders do care about the business or customers part, but think very differently about these aspects than business people, and thus come to very different conclusions how to handle these topics.
In my experience, it's rather exactly these programmers who are often in conflict with business people
- because they care about such topics
- because they come to different conclusions than the business people, and
- because these programmers care so much about these business-related topics, they are very vocal and sometimes confrontative with their opinions.
In other words: coders who barely care about these business-related aspects are often much easier to handle for business-minded people.
> I believe that coder exists only in your imagination. All the good ones I know are great communicators. Clarity of thought is essential to writing good code.
Clarity of thought does not make you a good communicator with respect to communicating with business people. People, for example, say about me that I am really good at communicating to people who are in deep love of research, but when I present arguments of similar clarity to business people, my often somewhat abstract considerations typically go over their heads.