You know how everyone thinks agents are bad at the thing they're good at and good at the thing they're bad at? It turns out I must be bad at testing, because I thought they do a reasonable job.
assert(CONSTANT_CONFIG == valueOfConfig)
assert(prompt.includes("repo url"))I've seen in multiple projects things like assertTrue(true).
I'm sure the agent is better in testing than average enterprise developer.
I think they show that additional prompting for testing approaches have wide differences in error rate (worst has twice the error rate of the best) but actually no extra prompting is pretty fine and most custom prompts are worse than no prompt.