> How well do current models do against prompt injection? Not so great. A recent paper found human red-teamers achieve near-100% attack success rates against frontier models5. But, these same LLMs score near-perfectly on standard prompt injection benchmarks! The discrepancy is straightforward: skilled humans test and adapt attacks until they work, benchmarks don't. Static benchmarks measure attacks models have already learned to catch.

I know it's likely more complex than what I am about to say, and maybe this quote is addressing it - I have a suite of ~100 tests for my agent to detect regression in task efficiency, but they're mostly designed for its compliance. E.g., if I say never do something, and then it does the thing, that is a failed test.

I struggled early on very badly designing this - because it seems no matter what, if the agent knows it's being tested, it virtually never fails. I could not get it to at first, but when I'd actually work with it, it would break rules constantly and often.

So the only way I could successfully test it was to design what looked a "normal" workflow task that could potentially involve doing things that it was not supposed to do. At first I would have it log every tool call, but it figured out fairly quickly why I wanted it to do that - perfect compliance again. So now I just have a secondary context that looks at all tool calls and measures it against the benchmark tests, and scores it. This ended up catching the behavior pretty well.

It was so weird though, colleagues and I would joke about "what if we make it think it's always being evaluated" and a few did end up doing that, and claim some success.