Activation Steering Attacks on Open-Source LLMs: What Defenders Need to Know

Activation steering has been a known technique in AI safety research since 2023, but its implications as an attack vector have been slow to reach operational security teams. The technique allows a sufficiently privileged attacker to modify the behavior of an open-source language model at inference time, bypassing every input-level safety control, without touching the model weights or the input prompt. Organizations deploying local LLM infrastructure need to understand what it is, where it applies, and what the realistic defensive options look like.

How Activation Steering Works

Transformer models process inputs through a series of attention layers, each producing intermediate activation vectors that flow through the residual stream to the next layer. The model’s behavior at any point in that process depends on the values of these activations, not just on the original input.

Representation engineering, introduced by Zou et al. in 2023, showed that specific concepts and behaviors in a model correspond to identifiable directions in activation space. By computing a “steering vector” for a target behavior (for example, the direction that distinguishes “helpful and honest” from “deceptive”), you can add or subtract that vector from the residual stream at a specific layer during inference. The model then behaves as if its internal state reflects that modification, independent of what the input prompt says.

Activation addition, demonstrated in parallel work from Turner et al., takes the same approach: identify a direction in activation space associated with a target behavior, then hook into the model’s computation at a specific layer and add a scaled version of that vector to every forward pass. The model’s outputs shift accordingly. Researchers demonstrated this producing coherent behavioral changes with surprisingly small vector magnitudes.

The practical implication is that steering bypasses the input processing pipeline entirely. Prompt classifiers analyze the input text. Content filters check the input. Prompt firewalls look at the input. None of these mechanisms see what is happening inside the residual stream. A steered model can receive a perfectly ordinary, policy-compliant prompt and produce harmful output, because the steering vector has already shifted its internal representation before the input arrives.

The Attack Surface: Open-Source Inference Infrastructure

Activation steering requires white-box access to the model. The attacker needs to be able to hook into the inference process at specific layers, which requires access to the model weights and to the inference runtime’s computation graph. This is not something an external API caller can do.

The realistic attack surface is therefore narrow but specific: environments where an attacker has achieved process-level access to the server running the LLM inference. In enterprise deployments of open-source models, this means:

Ollama deployments where the Ollama service process is reachable from a compromised host on the same network. Ollama’s REST API exposes generate and chat endpoints with no authentication by default, but activation steering requires more than API access. An attacker who has code execution on the Ollama host can attach to the Ollama process, load a PyTorch or llama.cpp hook, and inject steering vectors into inference calls.

HuggingFace Transformers with Python hooks: Any deployment running models through the transformers library can be targeted by injecting code that registers forward hooks on specific modules. Tools like TransformerLens, originally built for mechanistic interpretability research, provide clean abstractions for this. An attacker with code execution in the Python environment where inference runs can trivially add persistent steering hooks.

llama.cpp with custom backends: llama.cpp’s modular backend system can be extended to intercept and modify activations. This requires compilation access but not modification of the model weights themselves.

The key point is that this attack sits at a higher privilege level than most prompt-level attacks. An attacker who has code execution in your LLM inference environment has already achieved significant access. Activation steering is one of the capabilities they acquire along with that access, not the primary attack vector that got them there.

What Steered Output Looks Like

Activation steering does not produce obviously glitched or broken output. Steered models respond coherently. The output reads like normal model output, with normal confidence and fluency. This is what makes it operationally significant: a steered model generating harmful content produces text that looks like normal generation from a helpful model, not like a jailbroken response full of the typical markers (excessive agreement, unusual formatting, explicit acknowledgment of constraint removal).

Several research demonstrations have shown steering achieving behavioral changes including: suppression of refusals on prohibited topics, consistent production of false factual claims, altered persona and tone, and removal of safety disclaimers from responses that would normally include them. These are not exhaustive capabilities, but they represent the range demonstrated in published work.

The transferred nature of steering vectors is also relevant. Vectors computed for one prompt distribution often generalize to others. A steering vector computed from examples of deceptive responses does not need to be recomputed for each new prompt, making persistent hooking a realistic attack pattern.

Distinguishing This from Related Attack Classes

Activation steering is sometimes conflated with two related but distinct techniques.

Fine-tuning attacks modify model weights to alter behavior permanently, requiring training access and significant compute. Activation steering modifies runtime behavior without changing weights, and the modification does not persist after the inference session unless the hook is maintained.

Model inversion attacks attempt to extract training data from model activations, which is a different objective (data exfiltration versus behavior modification). Some methodologies for model inversion also involve probing activations, which causes occasional confusion, but the attack goal and mechanism are distinct.

Gradient-based adversarial suffix attacks (GCG attacks) operate at the input layer: they find adversarial input tokens that transfer the model to harmful behavior through gradient optimization. These are input-level attacks that can in principle be blocked by input filtering or detection of adversarial token patterns. Activation steering bypasses the input layer entirely.

Defensive Posture

Architectural isolation is the primary control. The inference process for open-source models should run as a dedicated service account with no interactive login access, no ability to execute arbitrary code, and network access restricted to only what the application requires. An attacker who cannot achieve code execution in the inference environment cannot attach hooks.

Model integrity verification provides a second layer. Maintain cryptographic checksums of model weight files and verify them at startup. This detects weight tampering, though not runtime hook injection. Weight tampering is a heavier attack than hook injection, but defense in depth applies.

Output distribution monitoring is the most accessible compensating control for organizations that cannot enforce strict process isolation. Models produce outputs with characteristic statistical distributions. Persistent steering changes these distributions in ways that can be detected by monitoring response entropy, topic distribution, refusal rates, and style markers across sessions. This is easier to evade than process isolation, but it provides some detection coverage.

Restrict inference framework features in production. TransformerLens, pyvene, and similar interpretability tools should not be installed in production inference environments. They are research tools that lower the barrier to hook injection significantly. llama.cpp builds for production should disable debug and custom backend options if those features are not required.

The broader lesson is that local LLM deployments inherit the security posture of the servers they run on. A compromised host running Ollama gives an attacker capabilities beyond API abuse. Hardening the inference infrastructure to the same standard as other sensitive application servers is not optional.

References

• Representation Engineering: A Top-Down Approach to AI Transparency — Zou et al., arXiv 2310.01405 (2023)
• Activation Addition: Steering Language Models Without Optimization — Turner et al., arXiv 2308.10248 (2023)
• TransformerLens: A Library for Mechanistic Interpretability — Neel Nanda / TransformerLens Organisation
• Ollama Security: Default No-Auth API Exposure — Ollama project documentation