Published
- 6 min read
By Allan D - Editor, AI Security Wire
Gating AI-Written Security Patches: Threat Models and Regression Checks
Trail of Bits argues that as AI accelerates bug discovery, the industry has to turn findings into reliable fixes just as quickly. Our reading is that agent-written patches move risk into the patch itself: a fix can close one hole and break supported behavior somewhere else. In October, Trail of Bits published a concrete pattern for controlling that risk, and it translates into controls any team can adopt.
What Trail of Bits measured
Trail of Bits wrote the plugins through Patch the Planet, a joint initiative with OpenAI in which it says it has co-authored hundreds of security patches for widely used open-source projects. Its October post describes a review of every patch a Trail of Bits engineer authored and maintainers merged upstream. An agent looked for candidate regressions, confirmed each with a proof-of-concept (PoC) run before and after the change, and passed the results to an engineer.
The review confirmed 21 regressions among more than 130 merged PRs: 16 functional, one performance and four build or test. Trail of Bits states that none broke a security protection. Its regression definition is a new defect introduced by a change that breaks previously correct and supported behavior along a path a real user or attacker can reach.
The review was also noisy. The 21 confirmed regressions came out of 58 candidates in a single run, roughly 36% precision, and each additional run produces more candidates to sort by hand. Trail of Bits drew two conclusions: patches need a standard way to be written, and regressions need a standard way to be checked, with written rules for what counts as one.
Control 1: write down the threat model for the agent
Both plugins read project knowledge from files in the repository. Trail of Bits names two:
THREAT_MODEL.mdtells agents what counts as a vulnerability or a regression. It should define the assets whose compromise matters, trust boundaries, access controls, supported configurations, what is and is not a vulnerability, and the evidence required for a valid bug report.PATCHING_GUIDELINES.md(orAGENTS.md) tells the agent how the project wants fixes written: how to build and test, how big a fix should be, where it should go, how to present changes and what a regression test must show.
The regression-hunt plugin passes the threat model into every step, so each agent has to check its conclusion against the file. Trail of Bits points to published examples from GPAC, liboqs and validator.js.
Our reading: an agent given no definition of supported behavior or scope has to guess one, and a written file makes that definition reviewable by a human. Trail of Bits does not make this claim in those words.
Control 2: require a live proof for every claim
Trail of Bits’ plugins insist on a live PoC against a built, running instance of the project, reached through the interfaces a real user would use. If a step cannot produce a live PoC, its result is inconclusive. This applies when reproducing a bug, proving a fix and confirming a regression.
For the patching plugin, the proof takes a specific shape, the three-arm proof:
- The original code must fail the reproduction.
- The patched code must pass it.
- A complete revert of the patch must fail it again.
The third arm is the useful one. Trail of Bits says bringing the bug back by removing the patch is stronger evidence that the patch, rather than an unrelated environment or test change, actually fixed the bug.
Control 3: replay and adjudicate the evidence
The regression-hunt workflow has four stages, per the post:
- Enumerate candidate regressions by tracing the changed code through its callers to its real interfaces.
- Reproduce each candidate on a live deployment.
- Replay the PoC on a fresh-context agent.
- Judge all prior results to decide whether the proof meets the definition of a regression.
The burden of proof sits with the agent in the adjudication step. It has to show the suspected regression is not a false positive, an incomplete fix, a latent bug or a pre-existing bug. The plugin reports a regression only if every gate passes. Strict input and output schemas, covering the claim, the code revision, the required check, raw evidence and open questions, let each step verify the previous step’s work rather than trust its summary. Replaying the PoC in a fresh context is the source’s design; keeping the reviewing agent separate from the agent that wrote the patch is our recommendation, not a claim from Trail of Bits.
Control 4: patch the invariant, then run the regression pass
The patching-bugs plugin aims to fix the underlying, violated security invariant rather than the one reported bug. That helps the fix cover variants of the same bug class. It also mandates running regression-hunt on every patch, so agent-written fixes go through the same pipeline.
What the numbers do and do not show
In preliminary evaluations on the 21 confirmed regressions, Trail of Bits reports that regression-hunt produced 25% fewer false-positive findings than a baseline Codex agent prompted to “review the PR for regressions” (95% confidence interval 14% to 38%). The strict evidence rules also left some known regressions classified as borderline, where the plugin saw a possible regression but did not gather enough evidence to confirm it. Treat this as a tradeoff: fewer false positives, at the cost of confidently reporting fewer known regressions. Every figure is Trail of Bits’ own evaluation on its own patches, against a simple baseline prompt, and the post gives no absolute false-positive rate or recall figure. No independent replication found as of Nov 20 2026 in the Trail of Bits post, the Patch the Planet page or the plugin README. Trail of Bits describes these as preliminary results and says it is building a further evaluation for patching-bugs.
Adoption checklist for your repository
These are our recommendations, drawn from the Trail of Bits post, not Trail of Bits’ own checklist.
- Add a
THREAT_MODEL.mdthat states assets, trust boundaries and what is out of scope. - Add
PATCHING_GUIDELINES.mdorAGENTS.mdcovering build, test, fix size and regression-test expectations. - Require a live PoC for any agent-reported bug, fix or regression, and treat anything without one as inconclusive.
- Require the before, after and revert runs on security fixes.
- Replay regression evidence in a fresh context. We also recommend keeping that check separate from the agent that wrote the patch.
- Keep a human reviewer on the merge. Trail of Bits’ workflow ends with an engineer reviewing the results.
- Feed missed regressions back as benchmark tasks, as Trail of Bits does with its agent trajectories.
References
Frequently Asked Questions
- What did Trail of Bits find when it reviewed its own AI-assisted patches?
- Trail of Bits reviewed every patch authored by a Trail of Bits engineer and merged upstream by maintainers. The review confirmed 21 regressions across more than 130 merged PRs: 16 functional, one performance and four build or test. Trail of Bits states that none broke a security protection.
- What is a three-arm proof?
- The same reproduction runs against three versions of the code. The original code must fail, the patched code must pass, and a complete revert of the patch must fail again. Reverting and seeing the bug return shows the patch, not an unrelated environment or test change, fixed the bug.
- What should a THREAT_MODEL.md contain for patching agents?
- Trail of Bits says it should define the assets whose compromise matters, trust boundaries, access controls, supported configurations, what is and is not a vulnerability, and the evidence required for a valid bug report. The regression-hunt plugin passes it to every step so each agent checks its conclusions against it.
- Does regression-hunt remove false positives completely?
- No. In Trail of Bits' preliminary evaluation it produced 25% fewer false-positive findings than a baseline Codex agent prompted to review the PR for regressions (95% confidence interval 14% to 38%). The stricter evidence rules also left some known regressions classified as borderline, so fewer false positives came with a tradeoff in how many known regressions it confidently reported.