I recommend you look at tools like Aider or Codebuff... sure they need to call some LLM at some point (could be your own, could be external), but the key thing that they are doing complex modifications of source code using things like treesitter -> i.e. you don't rely directly on the LLM modifying code, but the LLM using trees to modify the code. See in Aider's sourcecode: https://github.com/Aider-AI/aider/tree/main/aider/queries

Simple copy-pasting of "here's my prompt, give me code" was always doomed from the start to be perfect every time, and DEFINITELY won't work for an agent. We need to start thinking about how to use these LLMs in smarter ways (like the above mentioned tools)