I thought it pretty clear that the code was generated by the same agent that received the testing prompt, so there were no constraints on the code structure, and the testing strategy was known at the time the structure was generated.
> For the first eval, I tried giving agents the zstd RFC (plus errata) and telling them to implement a complete zstd decoder (agents are stuck in a container without internet access). The tests were not given to agents. For something with the surface area of zstd, it's not really reasonable to expect that the tests cover every possible case. For example, even though zstd is a fairly well-tested piece of software, I once found a data corruption bug in zstd. The test suite isn't intended to find extreme corner cases that might be lurking for years and is instead intended to check various cases that can "easily" be derived from the RFC that should work.
So basically, the agents were given the zstd RFC, told to implement a decoder, and also told to use a particular testing strategy (or not, for the control runs).
It's a bold strategy, cotton.
I don't think that's a very effective test of an agents ability.
It just looks like a lot of effort to go to in a way that doesn't answer the question the title poses.
I'd love to see something that could guide how to set an agent up for success rather than this naive approach which only really answers the questions that most of the audience here probably don't have.