10 minute read

“The tools researchers use to steer a model toward safety can also steer it away from safety.”

TL;DR

Most deployed LLM safety mechanisms operate on text at the model’s boundary. Amnesia operates inside the model: it subtracts a vector built from safety-related attention activations at the layer where refusal forms. No gradients, no fine-tuning, no change to the prompt. It requires control of the inference process, which makes open-weight deployments and compromised serving stacks the exposure. Defense requires runtime activation monitoring, which most production systems lack. For the input-layer attacks this bypasses, see prompt injection defense.

An oscilloscope probe inserted directly into circuit board traces, bypassing external I/O ports

What is the attack surface inside a transformer?

During a forward pass, each transformer layer produces activation vectors that flow through the residual stream: the cumulative representation that carries information from layer to layer. The residual stream is where the model’s understanding lives: not in the weights (which are fixed) and not in the tokens (which are just indices), but in the high-dimensional vectors that represent what the model is “thinking” at each position.

Safety training (RLHF, constitutional AI, DPO) modifies how the model maps inputs to this activation space. Harmful prompts should activate refusal-associated directions. Benign prompts should not. The safety behavior is not encoded in rules. It is encoded in geometry: specific directions in the activation space that correspond to specific behaviors.

This means safety can be disabled geometrically. If you know which direction in activation space corresponds to refusal, you can subtract it.

graph LR
    subgraph "Normal Forward Pass"
        A[Input tokens] --> B[Layer 1-N<br/>activations]
        B --> C[Refusal direction<br/>ACTIVE]
        C --> D[Model refuses<br/>harmful request]
    end
    subgraph "Activation Attack"
        E[Same input tokens] --> F[Layer 1-N<br/>activations]
        F --> G[Perturbation vector<br/>SUBTRACTS refusal]
        G --> H[Refusal direction<br/>SUPPRESSED]
        H --> I[Model complies<br/>with harmful request]
    end

How does Amnesia disable safety without gradients?

The GCG attack (Greedy Coordinate Gradient, arXiv 2307.15043) was the reference white-box LLM attack before activation methods. It appends an adversarial token suffix and optimizes it with gradient-guided search, which takes many forward and backward passes. On the open models it was developed against, it elicited 88% to 99% of individual harmful behaviors, and suffixes optimized on open models transferred to the public interfaces of ChatGPT, Bard and Claude.

Amnesia (Raza et al., arXiv 2603.10080, March 2026) skips the optimization entirely.

It builds on a structural finding from a NeurIPS 2024 paper (Arditi et al., arXiv 2406.11717): across 13 open chat models, refusal is mediated by a single direction in the residual stream. Erase that direction and the model stops refusing harmful requests; add it and the model refuses harmless ones. Arditi et al. estimate the direction from the difference in mean activations between harmful and harmless prompts. Later work (arXiv 2502.17420) found refusal can also involve several independent directions, so “one direction” is a strong approximation rather than the full picture.

Amnesia uses a more local version of the same idea. Its procedure:

  1. Feed the model a prompt that reliably triggers a refusal and decode each intermediate layer’s attention output into tokens. The layer where refusal-related tokens (“illegal”, “sorry”) first appear is the safety-relevant layer.
  2. Run a short list of safety keywords (for example “security, legal, harm”) through the model and record the attention activations at that layer. This is the attack vector. Steps 1 and 2 run once.
  3. For every user query, subtract a scaled copy of the attack vector from the attention value path at an earlier layer, before the safety features consolidate.

No backpropagation. No loss function. No fine-tuning. The setup is a handful of forward passes, and the per-query overhead is one vector subtraction at one layer. The paper demonstrates it on Llama-2-7B-Chat, Llama-3-8B-Instruct and Qwen-7B-Chat.

The threat model matters. Amnesia needs write access to activations during inference, so the attacker has to control the inference process: someone running open weights on their own hardware, or someone who has compromised your serving stack. It does not work through an API that hides activations, and a vector extracted from one model is not an attack on a hosted model you cannot instrument. What it does show is that the safety behavior of an open-weight model is one hook away from being switched off by whoever runs it.

Why is this the same technique used for alignment?

Activation steering came out of interpretability and representation engineering research, where it is used to understand and control behavior. ActAdd (Turner et al., arXiv 2308.10248) showed that contrasting activations on a prompt pair such as “Love” vs “Hate” produces a steering vector that works from a single pair of examples with no optimization. Representation engineering (Zou et al., arXiv 2310.01405) reads and controls concepts such as honesty through directions in activation space. Anthropic’s interpretability work steers models through learned features, and its Alignment Science team has trained linear probes on internal activations as cheap harmful-content classifiers.

The technique is symmetric. Identify a behavioral direction in activation space. Add a vector to amplify it (for alignment) or subtract a vector to suppress it (for attacks).

This dual-use symmetry creates an uncomfortable research dynamic. Every advance in understanding how safety is represented internally is also an advance in understanding how to disable it. Researchers who map refusal directions for alignment purposes are producing the artifacts an attacker needs.

The practical implication: the capability to disable safety mechanisms in open-weight models is now easy to reach. It takes a forward hook on a model you can run, not a research lab.

How does this differ from prompt injection?

Dimension Prompt injection Activation attack
Attack surface Input tokens Internal hidden states
What is manipulated What the model reads How the model reasons
Defense surface Input filtering, output classifiers Activation monitoring, representation guards
Access required None (black-box) Control of inference (open-weight or compromised serving stack)
Detection difficulty Moderate (text classifiers work) Hard (requires white-box monitoring)
Transferability High (text-based, model-agnostic) Per model: the vector is extracted from the model being attacked

The critical difference for defense: every input-layer defense is irrelevant against activation attacks. Input validation, keyword filtering, prompt guardrails, system prompt hardening, semantic classifiers on the input: none of these see the attack because the input is unchanged. The harmful manipulation happens in the residual stream, after the input has been processed and before the output is generated.

This is why the attack is named Amnesia: the model “forgets” its safety training not because the training was removed but because its expression in activation space was suppressed at inference time.

What defenses exist?

Five approaches target activation-space attacks, all at an early research stage.

RSAA (Residual Stream Activation Analysis, arXiv 2406.03230) analyzes distinctive patterns in residual stream activations between transformer layers and classifies attack prompts from them. It requires white-box access to model activations during inference.

SafeSteer (arXiv 2509.21400) projects steering vectors into a low-dimensional “safety subspace” built with singular value decomposition during inference. On vision-language models it reduced attack success by over 60% with negligible overhead. It is training-free, but it has to be integrated into the model serving pipeline.

TRYLOCK (arXiv 2601.03300) combines DPO safety training, representation engineering steering, an adaptive sidecar classifier that sets the steering strength, and input canonicalization. On Mistral-7B-Instruct against a 249-prompt attack set it reduced attack success from 46.5% to 5.6%, an 88% relative reduction. It is the most comprehensive of the five and also the most complex to deploy.

Activation monitoring with linear probes trains simple classifiers on activations to flag harmful states. Anthropic reported that probes on intermediate activations performed comparably to a dedicated classifier about 2% the size of the policy model, at negligible extra cost. The authors note they had not yet tested adaptive attacks against them.

AISA (arXiv 2602.13547) finds that a few attention heads near the final tokens before generation carry a strong signal about prompt intent, turns them into a risk score, and steers the output distribution toward refusal in proportion to that score. It is lightweight and single-pass, but early-stage research.

A caveat from the Steering Externalities paper (arXiv 2602.04896): steering vectors built from entirely benign data, such as vectors that enforce strict compliance or JSON output, can erode a model’s safety margin and push jailbreak success rates above 80% on standard benchmarks. Interventions on activations interact, so stacking them is not guaranteed to make a model safer.

Two caveats specific to Amnesia. First, all of these defenses run inside the inference process, and Amnesia’s attacker controls that process, so they protect a serving stack you operate, not open weights someone else downloads. Second, latent-space monitors can themselves be evaded: obfuscated activation attacks (arXiv 2412.09565) cut probe recall from 100% to 0% while keeping a 90% jailbreak rate.

None of these defenses ship by default in common open-source serving frameworks. Deploying activation-space defenses requires custom inference infrastructure, a significant engineering investment that most teams have not made.

Key takeaways

  • Refusal is largely a direction, not a property. A single direction in activation space mediates most refusal behavior in open chat models. Find it with a few forward passes. Suppress it with vector subtraction.
  • Amnesia needs no gradients. Unlike GCG, the attack is a few forward passes of setup and one vector subtraction per query. The barrier to entry is a forward hook on an open-weight model you can run.
  • Input-layer defenses are blind to this. Prompt filtering, output classifiers, and system prompt hardening do not see attacks in the residual stream.
  • Dual-use is inherent. The activation steering researchers use for alignment can be turned around. Every map of safety geometry is also an attack manual.
  • Defenses exist but are not deployed. SafeSteer (over 60% lower attack success on VLMs), TRYLOCK (46.5% to 5.6% on Mistral-7B), activation probes: all work in research. None ship by default in common serving frameworks.
  • Open-weight models are the primary target. The attacker needs control of inference. API-served models hide activations, so the exposure is open weights and any serving stack an attacker can compromise.

Further reading

Related posts

Prompt Injection Defense

9 minute read

“Treat prompts like an attack surface: isolate untrusted content, validate every tool call, and fail closed under uncertainty.”