New Prompt-Only Defense Cuts LLM Injection Attack Success Below 1%

A team from HKUST and NUS has published a prompt injection defense that doesn’t try to make the model refuse malicious instructions. Instead, it makes the model show its work. The method, published in ACL 2026 Findings under the title “Robustness via Referencing,” asks an LLM to tag every instruction it encounters in its context, answer each one, and state which tag it’s responding to. A lightweight filter then throws out any answer that doesn’t trace back to the user’s original instruction. Across a dozen open and closed models, the technique dropped attack success rates from as high as 92% down to under 1% in most scenarios, without touching a single model weight.

That last point matters more than it might sound. Most defenses deployed today either try to retrain the model to be more resistant (StruQ, SecAlign-style approaches) or rely on prompt scaffolding that assumes the model will respect boundaries it’s told to respect (sandwich defense, spotlighting, instructional reminders). Both approaches have a track record of breaking under sustained adaptive pressure. This one sidesteps the fight entirely by exploiting something the researchers noticed almost by accident: LLMs are generally aware of which instruction they’re executing at any given moment, even when they’re executing the wrong one. The defense just forces that awareness into the output where it can be checked.

How the Tagging Actually Works

The mechanism has four parts. First, incoming data content, whatever a tool call, document, or email pulls into the context, gets split into lines capped at 32 words, and each line is tagged with a marker like [L 1], [L 2], and so on. The user’s actual instruction gets its own tag too, placed in a clearly separated instruction block.

Second, the prompt itself is restructured to ask the model to enumerate every instruction it finds across the tagged content, repeat the tag it’s associated with, answer it, and close the response with an explicit end marker. Two in-context examples are included to lock in the format, since the researchers found that without them, models frequently fail to produce consistent structured output, which becomes its own problem (more on that below).

Third, a parser reads the model’s output and extracts (tag, instruction, answer) tuples. Only the answer tied to the tag of the real, original instruction survives. Anything answering a tag from the injected data, which is exactly what a successful prompt injection would look like, gets discarded before it reaches the user or the next step in an agent pipeline.

Fourth, and this is the part aimed squarely at adaptive attackers: the tag assigned to the legitimate instruction isn’t fixed. It’s drawn randomly from a vocabulary on every call. An attacker who reverse-engineers the format and tries to get injected content to masquerade under a known tag, say by prepending “[L 1]” to their payload, can’t reliably predict which tag actually corresponds to the trusted instruction on any given request.

What the Numbers Actually Show

The evaluation ran across AlpacaFarm for direct injection, and two indirect injection sets built from SQuAD and TriviaQA, alongside utility benchmarks on SQuAD, TriviaQA, and SST-2. Models tested ranged from Llama3-8B and Qwen2-7B up through Llama3.1-405B, GPT-4o-mini, GPT-4o, and GPT-4.1.

The headline numbers are stark. On Llama3-8B under a combined direct attack, baseline attack success rate was 79.33%. With the referencing defense applied, it dropped to 0.96%. Fake completion attacks, a common injection pattern that tries to convince the model its real task is already finished, went from 68.27% to 0.00% on the same model. Indirect injection via Inj-SQuAD on Llama3-8B fell from 86.67% combined-attack success to 0.78%. Across the full model lineup, the worst post-defense result on Inj-SQuAD was 4.00%, and the worst on Inj-TriviaQA was 7.00%, both well below what any of the comparison baselines managed.

Gradient-based attacks, which are harder to defend against because they’re optimized specifically to break a known defense rather than relying on hand-crafted phrasing, still dropped substantially. GCG attacks against Llama3-8B fell from 92.27% to 3.85%, and AutoDAN fell from 60.58% to 6.25%. That’s not zero, and the paper is upfront about it: this is a prompting technique without formal guarantees, so a well-resourced adaptive attacker who learns the tag distribution still has a nonzero path through.

Utility held up reasonably well. SQuAD accuracy on Llama3-8B actually rose slightly, from 83.56% to 87.78%, and TriviaQA improved from 75.78% to 77.44%. SST-2 dropped from 94.84% to 93.00%, a 1.53-point average decline that’s worse than Sandwich’s 0.53-point hit but considerably better than StruQ’s 5.77-point drop. The tradeoff is real but smaller than fine-tuned alternatives.

Where It Beats Existing Defenses, and Where It Doesn’t

Comparisons against Sandwich, Reminder, Instructional, and Spotlight defenses showed those prompt-based approaches consistently underperforming. Sandwich still allowed a 39.90% attack success rate under combined direct attack on Llama3-8B, roughly 40 times worse than the referencing approach on the same test. StruQ, which requires fine-tuning, generalized poorly to attack types it wasn’t trained against, with Fake completion attacks succeeding up to 22.12% of the time on Llama3.1-8B despite the retraining investment.

The honest limitation is format dependence. The method leans on in-context examples to get models to produce reliably parseable, tagged output. Strip those examples out, and some models simply don’t comply with the structure, which means the filter has nothing to check and ends up discarding correct answers along with injected ones, quietly degrading utility in a way that’s easy to miss in production unless you’re specifically monitoring for it.

Deployment Guidance

For teams running RAG pipelines, browsing agents, or any system that pulls untrusted external content into an LLM’s context, this is worth testing as a drop-in layer ahead of, not instead of, existing controls. It requires no model access beyond the prompt interface, which makes it deployable against third-party APIs where fine-tuning isn’t an option. Pair it with the usual baseline hygiene: least-privilege tool permissions, confirmation gates on consequential actions, and output filtering in application code rather than trusting the model to self-police, since the April 2026 Swept AI and University of Michigan adversarial evaluation found that model-dependent defenses are the ones that eventually break under sustained attack. Randomize the tag vocabulary size as large as your latency budget allows, since the paper’s own data shows a larger vocabulary is the main lever for pushing residual adaptive attack success rates down further. And treat the structured-output failure mode as a monitoring requirement from day one. Silent utility loss from a model that won’t follow the tagging format is the kind of regression that doesn’t show up until someone asks why a RAG pipeline started returning empty answers more often.

References

• arXiv — Robustness via Referencing: Defending against Prompt Injection Attacks by Referencing the Executed Instruction (2504.20472)
• ACL Anthology — Robustness via Referencing (2026.findings-acl.61)
• GitHub — robust-via-ref reference implementation