I wonder the same. The answer I usually get from people who do manage is that they don't look at the code – or at least not in detail.
Personally, I always end up tweaking something the agent produced. I wonder if I should let go of that control...
Personally, I always end up tweaking something the agent produced. I wonder if I should let go of that control...
It definitely has fewer bugs than a senior developer, but it really hinges on getting the plan right. 20 minutes of planning and 20 of implementation sounds about right for my workflow as well, just make sure you have GPT as a reviewer. It's very nitpicky and finds lots of bugs.
I guess you actually review and actively participate in making the plan, you just don't review the code afterwards?
Could you share some more details on the specifics of your workflow? (What models/harnesses? do you use the same or different context windows? How exactly do you run the review, and how do you pass along and act upon the information from the review?) Also, how big are the changes you usually implement with one plan/develop/review cycle?
The changes aren't usually very big, basically what you'd put in one ticket. If I need to make large changes, I do them in self-contained stages, if that's possible, otherwise I will tell the LLM to add specific tests in the plan, and I will test thoroughly after.
First, that this is challenging to scale across large orgs. Even if your plans produce high quality code, that isn’t true for everyone. I’m definitely struggling with slop code being collectively mailed to me for review my our 1,000 engineers that were told to use their AI subscription all at once.
I feel like we should be taking “prompt engineering” more seriously. And when people mail me code to review, it should also include the agentic workflow and plan. So that when code isn’t up to quality, and can have a discussion about the prompts used to generate it.
My second thought is related to your senior engineer comment. This isn’t surprising, because in most engineering orgs, seniority is completely unrelated to code quality. In fact, many orgs incentive the opposite: “senior” devs that push out buggy code quickly and push accountability downhill to the junior devs.
Judging by how they struggle to communicate generally, I can't imagine their prompts are doing much heavy lifting.
About senior engineers, I guess that depends on the org you have experience with. My experience doesn't match yours.
I know a lot of engineers who skip the last part. They're over confident in their original plan. They're over confident the agent actually fulfilled the plan.
The answer won't be the same for all software, but you're assuming it will be.