If I go “find issues in this code” it will hallucinate some, but if I say “can you check the recent change, there might be some things that introduced regressions, maybe?” Then it will be more cautious.
Also especially fable but opus too can talk back and advise you against going into a direction it thinks unwise.
And I’ve had much more success in clearing out why I think that is a better approach or asking it to clarify itself, as if if I tell it my assumption, sometimes it self corrects and starts doing what I needed in the first place, it was just coming at it from a different direction before. For example assuming I don’t care about cost and providing “the best solution” or trying to make something reusable where what I needed something quick (or vice versa)
It really is best to think of it as a gradient plane where it might get stuck in local minima, or you can prime it to “teeter on the edge” and able to flow into different directions.
If you had multiple people look at your PRs multiple times on different days results would be very similar.
It’s not perfect but usually it works pretty well, and I’ve had the model come back to me with oh actually the test passed, the bug doesn’t work exist
As a bonus, you’ve now got a test that can detect that bug if it comes up again.
The "keep improving" the code base prompt have been tried and it never works. The LLM has no consciousness of where to stop and where to draw the lines of reasonableness.
For a normal review loops you can ask the model to return with nothing found if nothing is found and not invent things and it will do a better job of exiting without anything found.
typically this means there is some ambiguity in the specification, and the model flips between alternative interpretations
Like when you do recursive programming, have you tried providing more/better stop conditions? If you literally just say "Continue until there are no more issues" then it'll do just that, but if you scope it better, like "Only mention issues related to X, Y or that leads to Z" and so on, you'll get less noise and more focus on issues that actually matter (to you).
Tell it something like:
Before doing any commits or producing a summary for the user, you must run a verification sub-agent.
Its goal is to adversarially and critically check your supposed findings to look out for false positives and hallucinations.
Doing so with a separate sub-agent with relatively clean context (but with all the relevant details of the problem space that appear to be facts) should improve our confidence in the findings.
Try to classify each found issue as either SERIOUS, CRITICAL or NITPICK, discard nitpicks, we only care about impactful issues.
I've largely found the same in regards to generating code - the initial pass will often have bugs that the model itself can find but only when run as a separate sub-agent without the confidence poisoning in its own previous output.
Anyway it will never match your judgemend completely unless you upload your brain dump into model.
Not entirely true IME. Eventually the bug hunt will end with general design advices that may not be suitable to your use case and that you can skip.
I've had the same experience, but whenever I've reviewed what it finds it's basically right. It's pedantic, and a lot of the problems aren't things I really care about, but they definitely are real problems.
I'm not sure you can blame the AI for always finding problems if a) you asked it to, and b) there are problems to find.
(The fixed prices are just temporary discounts)
Would you like it to stop when there's still flaws in the code?