I had a similar experience in search and found even holdouts can be overfit to. IE through brute force, it may not see the holdout, but if you gate a change on holdout acceptance it will land on a solution that’s overfit to it by somewhat random chance.

The other problem is that holdouts / data inaccessible to the agent isn’t easy to do in most coding agents. It’s not as simple as splitting training data 80% and giving some to the agent and hiding 20%. The agent can figure out where its data came from and find ways to reconstruct / cheat the holdout data.

All the ways of doing this seem annoying: ie having a second project that accepts / rejects changes.

I opted to just build my own harness for these things to avoid overfitting.

https://softwaredoug.com/blog/2026/05/17/autoresearching-a-b...