* The scope is constrained and clearly defined
* You articulate an exhaustive and objective set of success criteria
* You provide a deterministic tool for the AI to evaluate its output against those criteria and receive feedback (compiler errors, unit tests, regression tests, etc.)
You're essentially writing a comprehensive spec for the AI to follow. The time suck was always from making the spec and the QA bulletproof.
One shot benchmarks that have exhaustive prompts are impressive, but benchmarks that emulate real software development (iterative changes over time with changing requirements) have close-to-zero pass rates.
But there are cases where there are no changes, or only backwards-compatible changes. Take the core WebAssembly specification, for example. Extremely detailed, formally verified, comes with a large test suite. Someone else has done the hard work of specifying the behavior of every possible edge case. A WASM runtime implementer can point their agent to that spec, plus some prior art. The remaining manual work is in constructing a test harness and collecting a corpus of test WASM binaries that exercises decent coverage.