Constrained Agent Outputs: Limiting Injection Blast Radius

The formal proof that prompt injection in AI agents is unsolvable got attention when it circulated earlier this year. The core argument: because agents take instructions from both the operator (system prompt) and the environment (tool outputs, user input, retrieved documents), and because these instruction sources have no authenticated separation at the model layer, an attacker who can place content in any of those channels can influence agent behavior. You cannot reliably distinguish a legitimate instruction from an injected one using the model alone.

That proof is correct. It’s also somewhat misdirected as a starting point for defense. The more useful question is not “how do we prevent injection” but “when injection succeeds, what can the attacker actually do?” That question has a much more tractable answer, and constrained output generation is a meaningful part of the answer.

The Problem With Unconstrained Outputs

Most production AI agents have a structure: the LLM produces outputs, those outputs are parsed to extract tool calls, and the tool calls are dispatched. In an unconstrained system, the LLM can produce any output string. If injection causes the model to deviate from the intended behavior, the deviation is bounded only by what the model is capable of generating, which includes arbitrary function calls, arbitrary argument values, and arbitrary data in any format.

Consider an agent with access to five tools: search_documents, send_email, query_database, create_ticket, and summarize. With unconstrained output, successful injection can cause the agent to:

• Call any of the five tools with any arguments the attacker specifies
• Call tools in sequences not intended by the operator
• Produce output that doesn’t correspond to any defined tool call, causing the application to handle unexpected strings

If the agent were a coding assistant with shell access, the list gets worse. Unrestricted LLM output is the mechanism that turns “the model was tricked” into “arbitrary code executed.”

What Constrained Decoding Actually Enforces

Constrained decoding works by restricting which tokens the model can produce at each decoding step. Instead of sampling from the full vocabulary, the decoder calculates which tokens would keep the partial output within a defined grammar or schema, and samples only from that set. Outputs that violate the schema are structurally impossible, not merely unlikely.

For agent applications, the most useful schemas to enforce are:

Tool call schemas. When the agent is expected to produce a tool call, constrain it to the specific set of tools available and their defined parameter types. An agent configured with send_email that accepts a to address and body string cannot, under constrained decoding, produce a call to an undefined function or pass an argument of the wrong type.

Enumerated argument values. For tool parameters that should be one of a fixed set of values, define them as enums rather than free strings. An agent that routes support tickets to ["billing", "technical", "sales", "general"] cannot route to an arbitrary string an injected prompt might suggest.

Response structure schemas. For agents that produce structured output consumed by downstream systems, enforce the schema. An agent returning JSON objects with defined fields cannot produce a response containing extra keys, embedded scripts, or malformed structures.

The outlines library (from dottxt-ai) implements constrained decoding for locally hosted models, including regex patterns, JSON schemas, and context-free grammars. It integrates with vLLM for production inference workloads. For llama.cpp, GBNF (GGML BNF) grammars provide equivalent functionality: define the valid output grammar, and the sampler enforces it during generation.

For models accessed via API, the enforcement surface moves to the API layer. The Anthropic API’s tool definition format enforces tool call schema compliance server-side: responses will contain only defined tools with schema-conformant arguments. OpenAI’s response_format: { type: "json_schema", json_schema: ... } parameter enforces JSON schema compliance on completions. These are not constrained decoding in the strict sense, but they provide equivalent schema guarantees for API-served models.

Implementing Schema Enforcement in Practice

Here’s a concrete pattern for a Pydantic AI agent with constrained tool output:

from pydantic import BaseModel
from pydantic_ai import Agent

class TicketRoute(BaseModel):
    department: Literal["billing", "technical", "sales", "general"]
    priority: Literal["low", "medium", "high"]
    summary: str  # Free text, but bounded within the schema

class SearchResult(BaseModel):
    query_used: str
    num_results: int
    results: list[str]

agent = Agent(
    "anthropic:claude-sonnet-4-6",
    result_type=TicketRoute,  # Enforces response schema
)

The result_type parameter causes Pydantic AI to validate and retry until the model produces a response that parses to TicketRoute. Combined with the Anthropic API’s native schema enforcement, this creates two enforcement layers: the API rejects non-conformant tool outputs, and the application layer validates structured fields.

The summary field above is the residual risk: free text within a schema is not constrained. An injected instruction that causes the model to write attacker-controlled content in the summary field may still create downstream risk if that field is displayed to users or processed further. Schema enforcement is not content filtering.

What This Defense Doesn’t Cover

Constrained output generation reduces the action space available to successful injections. It doesn’t reduce injection success rates, and it doesn’t eliminate the risk entirely.

Intra-schema injection remains possible. If a tool accepts a URL parameter and injection causes the model to provide an attacker-controlled URL, the schema constraint doesn’t help. The URL string is valid; where it points is the problem. The same applies to any free-text field that downstream systems process.

Schema-aware injections are a known risk. As constrained decoding becomes more common, injected content may specifically craft values that comply with schemas while achieving attacker goals within those bounds. An attacker who knows the agent uses an enum ["billing", "technical"] might construct injection that reliably routes to one category over another.

Tool chaining under injection remains possible within the allowed toolset. If the agent has both search_documents and send_email, a successful injection can still cause the agent to search for sensitive data and then send it, as long as both calls conform to their schemas.

The defense applies to the correct threat model: limiting what an attacker gains from injection success in agents where injection cannot be eliminated. An agent that processes documents from untrusted sources, retrieves from open-access databases, or handles user-controlled inputs should be assumed to face injection attempts. For those agents, constrained output does real work.

Deployment Guidance

The practical implementation path depends on where your agents run.

For API-served models, use the provider’s native schema enforcement and define all tool parameters with typed schemas and enums where possible. Avoid string types for parameters that should be fixed values. Avoid any types entirely in agent tool definitions.

For locally hosted models with inference frameworks like vLLM or llama.cpp, add grammar-constrained sampling. The outlines library provides a vLLM integration; llama.cpp supports GBNF grammars that can be derived programmatically from Pydantic models or JSON schemas.

At the application layer, validate agent outputs against expected schemas before dispatch, regardless of what the inference layer claims to enforce. Treat the LLM output as untrusted input that must be validated, the same way you’d validate any external API response.

Constrained output generation is a narrow control. It doesn’t replace prompt hardening, output monitoring, or tool-level authorization checks. Applied alongside those controls, it makes the consequences of injection failures predictable, which is the foundation of any coherent defense-in-depth posture for agentic systems.