1Password's AI patching benchmark is misleading

Good example of how to push back on a study: recheck their data, publish your own, and include the cases where your work failed. The freenginx section is worth the read on its own.

In my experience the newest crop of models make patches that are quite good, especially if you have a revalidation loop versus a bug trigger.

If two grader models disagree on 37% of the same patches, there's no actual methodology. It's just a coin flip.

Kind of insane that anyone would think AI can only patch things correctly 26% of the time

This adds much needed context to a compelx issue.

calling it an AI failure when the human maintainer made the same mistake says more about the difficulty of the problem than the model

love the idea of generating a human baseline/control to try to evaluate agentic patching systems

Really exciting to see such important work being aided by AI given the scale of the threat.

When people question the efficacy of using AI for security work, they're often asking the wrong questions. (I blame the initial marketing hype around Claude Mythos for much of the unhelpful discourse.) And I don't just mean they're asking their AI the wrong questions, either.

Two problems that often arise when using LLM-based vulnerability assessment methodologies for a codebase are false positives and severity inflation.

As far as I can tell, false positives occur when the underlying language model pattern-matches a snippet of code with vulnerable, risky, or error-prone samples in its training data and/or context window. (Many SKILL.md files you find on GitHub are piles of Markdown describing anti-patterns, and occasionally TOML files describing roles the agent can assume when interpreting that Markdown.) It might be gesturing towards a real problem, but in practice, the signal-to-noise ratio can be extremely low without validation steps or expert feedback.

I don't quite understand the source of the usual severity inflation observed with LLM output, but I also haven't seen the training data, so it could just be a maladaptive behavior from the source material.

Regardless, the two problems are a symptom of the same systemic flaw: Insufficient rigor.

If you decide the work is done once you have "a result", you're doing yourself and your audience a disservice. It's imperative to have explicit mechanisms to ensure that the result is: a) real, b) relevant to the threat model of the application in scope, and c) measured appropriately in terms of both severity and difficulty. If you build the right mechanisms, you solve both symptoms in one fell swoop.

The post-patch-validation skill mentioned in the blog post was created as one of several mechanisms to ensure that the final output (i.e., what humans review) is actually worth anyone's time. We have other interesting AI experiments that I plan to open-source in the coming months (or in early 2027 at the latest).

Happy to answer any questions and hear any feedback anyone has on this skill.