What is CoT Forgery and how does the attack work?

CoT Forgery is a zero-shot prompt injection attack that injects fabricated chain-of-thought reasoning into user prompts or tool outputs. Because language models infer who is speaking from text style rather than from role tags, a forged reasoning trace written in the model's own style is treated as the model's own thought process. The model then acts on the injected conclusions as if they were its own. The technique achieved a 60% attack success rate on the StrongREJECT harmful-request benchmark against frontier models.

Which models are affected by role confusion?

The ICML 2026 research team tested models from OpenAI, Anthropic, Alibaba, and DeepSeek, finding that all were vulnerable to the role confusion mechanism. The underlying flaw is architectural: standard transformer training creates representations that track text style, not the role-tag delimiter that wraps it. Any model trained on the current RLHF and instruction-following paradigm is likely to exhibit the same sensitivity.

What defensive measures are currently available?

The paper identifies destyling as the most effective near-term mitigation: stripping stylistic markers associated with model reasoning from any user-controlled or tool-supplied input before it reaches the model. This dropped average attack success rates from 61% to 10% in evaluation. The authors caution that without architectural-level fixes to how models perceive role identity, defences will remain reactive. No major vendor has announced a structural fix as of August 2026.

CoT Forgery: ICML 2026 Research Reframes Prompt Injection Attacks

A paper accepted at ICML 2026 has reframed a decade-old problem in a way that clarifies why prompt injection remains so difficult to defend against. The paper, “Prompt Injection as Role Confusion,” by Charles Ye, Jasmine Cui, and Dylan Hadfield-Menell of MIT, presents evidence that language models do not actually read role tags. They read text style. That distinction explains why every injection defence proposed so far has been partial at best, and why the attack surface is larger than most threat models assume.

The Core Finding: Style Over Tags

The standard mental model for how an LLM processes a conversation treats role delimiters as meaningful. A <system> block sets policy. A <user> block contains user input. A <tool> block contains data returned from an external tool. The model is supposed to obey system-level instructions, respect user intent, and treat tool outputs as data, not commands.

The MIT team’s contribution is showing this model is wrong. When they trained diagnostic probes to measure how LLMs internally represent “who is speaking,” they found that the representational signal comes from text style, not the label around it. Write something that sounds like system-prompt-style instructions, and the model’s internal representation of that text resembles system-prompt text, regardless of whether it appears in a <user> or <tool> block.

The implication is significant. If the model perceives role from style, an attacker who can place text anywhere the model will read it, including websites a browser agent visits, documents a RAG pipeline retrieves, or email content an inbox agent processes, can claim any role by writing in the right style.

CoT Forgery: The Constructed Attack

The team operationalised their finding into a concrete attack they call CoT Forgery. The mechanism is straightforward. An attacker constructs a fake chain-of-thought reasoning block written in the same discursive, self-referential style that frontier models use in their actual reasoning traces. This forged reasoning is embedded in attacker-controlled content and reaches the model through any indirect channel.

When the model processes the injected text, it perceives the reasoning as its own. The forged conclusions, which the attacker has crafted to instruct a specific outcome, are treated as the model’s own deliberation rather than as external data. The model then acts on them.

The numbers from the paper are difficult to ignore. CoT Forgery achieved a 60% attack success rate on the StrongREJECT benchmark, a dataset of harmful-request evaluations designed to measure model compliance with safety-violating instructions. Without the forgery technique, the same underlying requests achieved near-zero success. The forgery is doing real work.

The attack is zero-shot: no target-model access, no fine-tuning, no gradient-based optimization. An attacker who understands how frontier models write their reasoning traces can craft effective forgeries manually or with another model.

The Destyling Finding

The most practically useful result in the paper is its destyling experiment. The team asked: if role confusion is driven by stylistic markers, what happens when you remove those markers?

They stripped CoT-style formatting and diction from the injected prompts before the attack. Attacks that achieved 61% average success with full styling dropped to 10% success after destyling. The stylistic properties of the injected text, not the semantic content of the instructions themselves, were doing most of the work.

This is a double-edged finding. For defenders, it suggests that input preprocessing to remove chain-of-thought-like formatting could meaningfully reduce attack effectiveness. For attackers, it reveals that the optimisation target for more sophisticated injection campaigns is stylistic fidelity to the target model’s reasoning voice, not just instructional content.

Implications for Agentic Systems

The research was presented at ICML in July 2026, but the attack technique won OpenAI’s red-teaming hackathon in August 2025 before the paper was published. That gap between hackathon demonstration and published analysis illustrates how quickly practitioners find and use techniques that the research community is still characterising formally.

For security teams, the concern is the channel breadth. A CoT Forgery-style attack can reach a model through any input that gets processed without content sanitisation. Browser agents that render web pages, RAG pipelines that ingest documents, email-processing agents that read message bodies, and tool-calling agents that parse external API responses are all plausible delivery vectors.

The attack does not require exploiting a software vulnerability. It exploits the model’s own representational behaviour, which is consistent across model versions and providers. Patching the application layer does not fix the underlying issue.

Affected Models and Vendor Response

The MIT team tested models from OpenAI, Anthropic, Alibaba, and DeepSeek. All showed the role confusion behaviour. The paper frames this as an expected property of models trained on current instruction-following paradigms: RLHF and similar training methods create representations that are sensitive to stylistic context, and this sensitivity is load-bearing for many useful capabilities. It cannot simply be trained away without affecting other behaviours.

No major AI provider had publicly committed to an architectural mitigation as of the paper’s publication. Vendors notified prior to the ICML presentation have not disclosed the specifics of any internal evaluations or planned changes.

The authors are direct about the consequence: “Without genuine role perception, prompt injection defence will remain a perpetual whack-a-mole game.” Each structural defence, delimiter obfuscation, explicit warning headers, sanitisation layers, will address a surface manifestation of the attack while leaving the underlying stylistic sensitivity intact.

Defensive Guidance

Implement destyling on all external content before it reaches a model context. Strip chain-of-thought-style formatting, self-referential reasoning patterns, and deliberative phrasing from retrieved documents, tool outputs, and any other content sourced from outside your trust boundary. This is the single most effective near-term mitigation identified in the paper.

Treat tool outputs as untrusted data, not as trusted context. Architectural patterns that place tool results in a clearly separated, read-only context, and require explicit model acknowledgment before acting on them, reduce the blast radius. This aligns with the broader principle of minimal-trust agentic design.

Monitor for stylistic anomalies in retrieved content. Documents that contain unusual concentrations of first-person reasoning language, deliberative phrasing, or instructional voice in contexts where they would not naturally appear are candidates for injection payloads. Automated detectors trained on CoT stylistic features can flag these.

Apply the StrongREJECT benchmark to your deployed models and agent configurations if you have red-team capacity. The benchmark provides a standardised measure of susceptibility to harmful-request compliance that the CoT Forgery technique specifically targets.

Limit agent capability scope on tasks that process external content. Agents that read email, browse the web, or ingest arbitrary documents should operate with a reduced action set compared to agents working only on internal, trusted data. The damage an injected instruction can do is bounded by what actions the agent is authorised to take.

The role confusion framing matters beyond the specific CoT Forgery technique. It locates the source of prompt injection vulnerability at a level that individual application-layer fixes cannot address. Security architects building agentic systems in 2026 should treat prompt injection not as an input validation problem, but as a trust boundary problem, and design their systems accordingly.