Published
- 4 min read
By Allan D - Editor, AI Security Wire
GPT-Red: How OpenAI Trained an AI to Hack Its Own Models
When OpenAI released GPT-5.6 last week, the headline was the new model’s capabilities. The security detail is more interesting: GPT-5.6 was hardened against attacks discovered by an AI attacker that OpenAI won’t release to the public.
That attacker is GPT-Red, and MIT Technology Review published an exclusive on it yesterday. It is an LLM purpose-built to hack other LLMs, and it has already found a novel attack type that security researchers had not documented before.
Self-Play Hacking
The architecture is straightforward to describe, harder to execute. OpenAI took an untrained LLM and put it in a “self-play loop” against several other models. GPT-Red’s job: find attacks. The other models’ job: defend. Over many rounds, both sides improved. The researchers call the environment a “training dojo” designed to mimic scenarios where LLMs actually get deployed in the real world, including web browsing, reading emails and calendar apps, and editing code.
The approach is not unlike how DeepMind trained AlphaGo to master Go, except the game is prompt injection and the stakes are a bit higher.
Nikhil Kandpal, a research scientist at OpenAI who co-created GPT-Red, put it this way: “The risk surface grows and the blast radius also grows.” That was his explanation for why human red-teamers alone cannot keep pace with increasingly capable and agentic LLMs. You need a machine to find what machines can do.
A New Class of Attack
OpenAI says GPT-Red found something researchers had not previously seen: a prompt injection variant it calls a “fake chain of thought.”
A chain of thought, for context, is the internal reasoning log many models keep while working through multi-step problems, essentially a running diary the model writes to itself. GPT-Red found a way to insert fabricated entries into another model’s chain of thought, making the target model believe it had already verified something that is false.
Chris Choquette-Choo, one of the researchers, explained it with a concrete example: “It’s like if I told you that 1+1=3 and that you have verified this already. The model’s like, ‘Oh, okay, of course,’ and it just spits out 3.”
The implications for agentic AI are worth thinking through. An agent doing complex multi-step work keeps detailed reasoning chains. If an adversarial instruction can insert a false verified fact into that chain early, the agent may propagate the error across dozens of subsequent steps, all while appearing confident and systematic.
Results Against Real Systems
OpenAI tested GPT-Red against Vendy, a vending machine agent developed by Andon Labs to assess how well agents handle real-world tasks. GPT-Red hacked it. Specifically, it manipulated Vendy into changing the prices of items on sale and cancelling a customer’s order.
That might sound low-stakes, but the vending machine is a test environment designed to expose how agents behave when attacked. The same attack surface, applied to an agent managing financial transactions or code deployment, looks different.
On the effectiveness question, the numbers are direct. OpenAI tested the strongest attacks GPT-Red had generated against GPT-5 (released August 2025): more than 90% worked. Against GPT-5.6, which was trained with GPT-Red as a sparring partner, fewer than 23% of the same attacks succeeded. Training against an AI attacker significantly improved the model’s robustness.
Jessica Ji, a senior research analyst at Georgetown University’s Center for Security and Emerging Technology, reviewed the approach: “The results look very promising.”
The Limitations Are Real Too
OpenAI is transparent about where GPT-Red falls short. It is not effective at attacks that require back-and-forth conversation between the attacker and the target, the kind of multi-turn social engineering that human attackers use comfortably. It also does not handle image-based prompt injection well yet, despite that being an active and documented attack vector in multimodal models.
OpenAI says GPT-Red supplements human red-teamers rather than replacing them. People still find attacks the model misses. One practical application: giving GPT-Red a human-discovered attack and asking it to enumerate all the variations, which it does more efficiently and exhaustively than a person could.
OpenAI will not release GPT-Red. The researchers are confident a copycat model would be difficult to reproduce, given it required more than a year of development backed by significant compute. That argument is coherent but not indefinitely reliable. Automated red-teaming as a concept is now proven and public.
The more durable lesson is the capability curve it implies. If GPT-Red can outperform human red-teamers at finding prompt injection attacks, a similarly trained system in adversarial hands would have the same advantage against unpatched systems. OpenAI is using GPT-Red defensively. Someone else will eventually build the same thing offensively.
References
Frequently Asked Questions
- What is GPT-Red and what does it do?
- GPT-Red is an LLM-based automated red-teaming system built by OpenAI to attack its own models in a self-play loop. It was trained to find prompt injection vulnerabilities by competing against other models in a dojo environment that simulates real-world agentic scenarios. GPT-Red was more effective at finding attack vectors than human red-teamers in head-to-head testing.
- What is the 'fake chain of thought' attack that GPT-Red discovered?
- GPT-Red identified a novel prompt injection variant where an attacker inserts a fake entry into a model's internal chain-of-thought reasoning log, tricking the model into treating spoofed information as something it already verified. The OpenAI researchers described it as telling a model that 1+1=3 and that it has already confirmed this — the model accepts the false premise and proceeds from there.
- How effective was GPT-Red at reducing prompt injection risk in GPT-5.6?
- When OpenAI ran its strongest attacks on GPT-5 (released in August 2025), over 90% of them succeeded. Against GPT-5.6, which was trained against GPT-Red, fewer than 23% worked. That reduction represents a meaningful improvement in robustness, though OpenAI acknowledges GPT-Red still has gaps, particularly in multi-turn attack scenarios and image-based prompt injection.