Few things give me more dread than reviewing the mediocre code written by an overconfident LLM, but arguing in a PR with an overconfident LLM that its review comments are wrong is up there.
On the other hand the amount of flip flopping they go through is unreal. I’ve witnessed numerous instances where either the cursor bugbot or Claude has found a bug and recommended a reasonable fix. The fix has been implemented and then the LLM has argued the case against the fix and requested the code be reverted. Out of curiosity to see what happens I’ve reverted the code just to be told the exact same recommendation as in the first pass.
I can foresee this becoming a circus for less experienced devs so I turned off the auto code reviews and stuck them in request only mode with a GH action so that I can retain some semblance of sanity and prevent the pr comment history from becoming cluttered with overly verbose comments from an agent.
The dread is explaining this to someone less experienced, because it’s not helpful to just say to use your gut. So I end up highlighting the comments that are legit and pointing out the ones that aren’t to show how I’m approaching them.
It turns out that this is a waste of time, nobody learns anything from it (because they’re using an LLM to write the code anyway) and it’s better to just disable the integration and maybe just run a review thing locally if you care. I would say that all of this has made my responsibility as a mentor much more difficult.
Relative quality is better but the absolute quality is not. I only care about absolute quality.
Bugbot is now a valuable part of our SD process. If you have genuine examples to show that we are just being delusional or haven’t hit a roadblock, I would love to know.