Have you tried the reasoning models? I find that o1 really is able to find logic bugs, even tricky ones!
Have you tried the reasoning models? I find that o1 really is able to find logic bugs, even tricky ones!
my guess is for some applications they can, but even reasoning models (o3-mini-high, grok3, sonnet 3.7, o1, deepseek, etc.) often fail to fix logic bugs. note that this isn't necessarily a form validation logic bug I'm referring to but, say, a pretty in-the-weeds tool for cleaning and pre-processing data for ML purposes. my guess is basic business-logic-y type stuff is much more doable.
i haven't really found a good way around cases where it either just adds printfs or loops through the same, non-working fixes repeatedly. they keep getting better, they're just not yet below my epsilon for unreliability.
I think "often" is the key word here. To be clear, they often fail for me to! But they also often work.
The real complicated "bugs" often come from unclear requirements and the hard part is clearing up the requirements. It's more about design than logic errors in the code. And LLMs suck hard at this.