1Password Research: AI Vulnerability Patches Succeed Just 26% of the Time

The appeal of AI-generated vulnerability patches is easy to understand. Security teams face a permanent backlog of CVEs, remediation timelines are under pressure, and frontier AI models can produce code fixes in seconds. The question is whether those fixes actually work. New research from 1Password suggests the answer, at least for now, is: less often than you might hope.

The Study

1Password’s security research division, Off-by-1 Labs, published a paper this month titled “Frontier Models’ Vulnerability Patches are Often F.L.A.W.E.D.” — where the acronym expands to Fix-Like Artifacts With Embedded Defects. The study evaluated two frontier AI models, ChatGPT 5.5 and Claude Opus 4.8, against six recently disclosed vulnerabilities in open-source software, generating 6,080 patches in total.

The six CVEs covered a meaningful spread of vulnerability types: CVE-2026-31431 (dubbed “Copy Fail”), CVE-2026-34197 (an ActiveMQ remote code execution flaw), CVE-2026-8512, CVE-2026-45185 (EXIM RCE), CVE-2026-22738 (a Spring AI SpEL injection vulnerability), and an RCE in the Gemini CLI. These are not obscure or trivially simple bugs — they represent the kind of vulnerabilities that active security teams are routinely working to remediate.

Each generated patch was evaluated across three outcome categories: did it fully fix the vulnerability without changing application behaviour, did it fix the vulnerability but alter how the application behaved, or did it fail outright — either not fixing the flaw or introducing a new one?

What the Numbers Show

The headline finding is that only 26% of patches fell into the clean success category: vulnerability fixed, behaviour unchanged, no new issues introduced.

A further 20.1% fixed the target vulnerability but changed application behaviour in ways that could introduce problems downstream. Whether that constitutes success or failure depends heavily on context — a behavioural change in a security-sensitive code path could itself become a vulnerability, even if the original flaw is gone.

That leaves 53.9% of patches that failed in one way or another: either the original vulnerability persisted, a new vulnerability appeared, or both.

There is a further finding that sits underneath those headline numbers. A significant proportion of patches that appeared successful on first evaluation contained what the researchers describe as “subtler fragile weaknesses” — issues that only emerged under more rigorous analysis. This suggests the 26% fully-clean figure may itself be optimistic, depending on how thorough the downstream review process is.

Why Patches Fail

The failure modes documented in the study are not random. AI models consistently struggle with patches that require understanding context beyond the immediate code being changed. A fix that closes one code path may leave an equivalent path open. A patch that addresses the specific input validation bypass in the CVE description may miss that the same bypass is achievable through a different entry point in the same function.

The SpEL injection vulnerability in Spring AI is a useful example of this category. SpEL (Spring Expression Language) injection can manifest through multiple expression evaluation contexts. A patch that correctly sanitises input in one context may not account for the others, leaving the application technically patched against the specific CVE but still exploitable via a variant technique.

AI models also tend to produce patches that are syntactically correct but semantically narrow. They fix what they can see in the prompt context. That limitation is fundamental, not accidental — it reflects how language models process and generate code. It is not a problem that scaling or better prompting reliably resolves.

The Human Review Question

1Password’s recommendation, and notably Anthropic’s own publicly stated position on AI-assisted patching, is that domain experts should remain mandatory final reviewers. The framing matters here. This is not a provisional recommendation pending model improvement. It is a statement about the current state of what frontier models can reliably do.

The 26% clean success rate means that deploying AI patches without review creates a situation where roughly three in four patches deployed to production are incomplete, behaviour-altering, or actively harmful. For a vulnerability remediation workflow, that failure rate is not acceptable. Patching is one of the highest-confidence actions a security team takes — the assumption is that after patching, the vulnerability is closed. This research shows that assumption does not hold when AI is generating the patch autonomously.

What This Means for Security Teams

The practical implication is not that AI should not be used in vulnerability remediation workflows. It clearly has a role: generating candidate patches, exploring fix options quickly, and handling straightforward cases where the fix is well-understood and the codebase context is limited. The implication is that the review gate cannot be automated away.

Teams using AI assistants for remediation should enforce three checks before deploying any AI-generated patch. First, automated testing that validates the specific vulnerability is actually closed — not just that the code compiles. Second, review for behavioural change, not just correctness. Third, sign-off from someone with working knowledge of the affected codebase, not just the specific file being patched. That combination will catch most of the 74% that the models miss. None of it is new security practice — the finding is that existing standards need to be held, not relaxed, when AI enters the workflow.

The full research paper is available via 1Password’s Off-by-1 Labs publication. The methodology, including the specific prompt structures used and the evaluation criteria for each outcome category, is documented in the PDF.