> they don't control for breaking the application's tests.
Essentially it's an absolutely worthless study then. This AI fatigue got boring a long time ago, this is just painful now.
Essentially it's an absolutely worthless study then. This AI fatigue got boring a long time ago, this is just painful now.
What we didn't do (stupid oversight on my part) was to ensure that there are no regressions in the remaining tests in the repo (unrelated to the current task). In practice, when using Sonnet 4.6 IRL, I don't see a lot of regressions because often the agent runs the test before calling it done. But it could have gone either way. We don't know.