Watermark Radioactivity: How Distillation Breaks LLM Content Provenance

The EU AI Act’s content transparency requirements for AI-generated text are now in force, and the compliance ecosystem is racing to deploy watermarking at scale. The timing is unfortunate. A body of research has quietly established that the statistical watermarking schemes most commonly proposed for this purpose carry a structural vulnerability: they are inheritable through distillation, and that inheritance can be weaponised to forge attribution. The gap between what regulators assumed watermarking would achieve and what it actually delivers securely is substantial, and it matters now.

How Statistical Watermarking Works

The dominant approach to LLM output watermarking, formalised by Kirchenbauer et al. in their 2023 paper on green-list token selection, works at generation time. The model’s token vocabulary is partitioned into two groups, “green” and “red,” at each sampling step using a secret key applied to the prior context. The model then samples preferentially from the green set, slightly biasing its output toward tokens that carry the watermark signal. A detector with knowledge of the secret key can compute the ratio of green tokens in any text sample and compare it against the expected distribution for unmarked text. Above a statistical threshold, the text is flagged as watermarked.

The scheme is elegant. It introduces minimal perceptible quality degradation, it operates at inference time without modifying training, and it produces a detectable signal across arbitrary-length samples. Providers including OpenAI, Google, and Anthropic have explored variants of this approach as part of their C2PA content credential work and GEMS (Generated Evidence of Model Source) labelling efforts.

The security assumption the scheme rests on: without the secret key, an adversary cannot reproduce the green-token distribution. This is true in the direct forgery case. It is not true in the distillation case.

Watermark Radioactivity as an Attack Vector

The term “watermark radioactivity” was coined to describe what happens when a clean student model is fine-tuned on outputs from a watermarked teacher. The student learns to imitate the teacher’s token distribution, including the subtle green-token bias introduced by the watermark. The student’s own generations then carry a statistical signal detectable by the teacher’s watermark detector, even though the student model was never explicitly watermarked.

This was initially characterised as a liability for watermark robustness: fine-tuning data contamination can cause false positives in detection. Researchers reframed it as an offensive primitive. A capable adversary can:

1. Collect a large corpus of outputs from a target provider’s watermarked model, accessible through any public API.
2. Fine-tune a local or third-party model on that corpus. The distillation inherits the watermark signal.
3. Use the resulting student model to generate content that passes the target provider’s watermark detector.

The attacker now has a model whose outputs are falsely attributed to the target provider. Harmful content, disinformation, or fraudulent material generated by the student carries the watermark identity of a legitimate, trusted provider.

The cost of this attack scales with the distillation budget, not with the cryptographic strength of the watermark. Standard fine-tuning on tens of thousands of API-sourced samples is sufficient to produce a convincing watermark transfer. This is within reach of sophisticated individuals, organised groups, and certainly state-level actors.

What Spoofed Watermarks Enable

The threat model splits into two directions. In the forward direction, an adversary wants to produce content that appears to originate from a reputable model. A disinformation campaign run through a spoofed-watermark student model would be detected as coming from, say, a regulated provider’s system rather than from a jailbroken or uncensored model. This defeats attribution-based content moderation and regulatory accountability.

In the reverse direction, an adversary wants to deny that content they produced came from their own system. Seeding false watermark signals that attribute to a competitor’s or regulator-approved provider’s model provides a misdirection layer in forensic attribution.

The EU AI Act scenario is concrete. Article 50 requires general-purpose AI providers to ensure their systems mark output in a machine-detectable format. Downstream platforms that consume AI-generated content are expected to check for these watermarks to verify provenance before publication or redistribution. A spoofed watermark passes these checks while carrying no truthful provenance information.

The Removal Attack Runs in Parallel

Watermark spoofing is the attribution forgery problem. Watermark removal is the parallel problem: an adversary who has legitimate watermarked content and wants to deny its AI origin.

Current removal attacks require surprisingly little effort. Paraphrase attacks, where the text is routed through a second LLM with an instruction to rephrase it, degrade statistical watermarks substantially in two to three passes. Translation chains (English to French to Japanese to English) further disrupt the token-level bias. Quality-degrading transformations, character substitutions, and synonym replacement all reduce watermark detectability below detection thresholds without visibly compromising text quality for most use cases.

The combination of easy removal and feasible spoofing means the watermarking layer provides weak provenance guarantees in an adversarial context. It is useful for bulk attribution in benign scenarios, not for high-assurance forensic provenance against a motivated adversary.

C2PA and Cryptographic Alternatives

The C2PA (Coalition for Content Provenance and Authenticity) approach addresses the spoofing problem differently. Instead of encoding provenance in the content itself through token distribution manipulation, C2PA attaches cryptographically signed metadata (content credentials) at the point of generation and serving. The signature is tied to the provider’s key infrastructure, not to a statistical property of the output tokens.

This is not forgeable through distillation. An attacker who distills from a C2PA-signed model gets a student that produces tokens, but not a student that can produce valid C2PA signatures. The signature either validates against the provider’s known public key or it does not.

C2PA has real limitations. Content credentials can be stripped by anyone who receives the content and redistributes it without the credential attachment. Most web infrastructure does not yet propagate credentials through the copy-paste, screenshot, and re-upload chains that dominate how content actually spreads. The scheme’s integrity depends on a chain of custody that breaks at many common endpoints.

A hybrid approach is the current research direction: cryptographic watermarking schemes that embed unforgeable signatures in token selection using provably secure constructions, rather than statistical biases. Early constructions are computationally expensive and quality-degrading, but the direction is sound.

Implications for AI Security Teams

The practical guidance for organisations managing AI governance posture in the current regulatory environment:

Watermark-based content detection tools should not be treated as high-assurance provenance indicators. False positives from spoofed watermarks and false negatives from removal attacks both occur in adversarial contexts. Watermark detection is useful for baseline attribution in cooperative scenarios, not for forensic certainty.

If your organisation is building content provenance infrastructure to meet Article 50 obligations or downstream platform verification requirements, C2PA credential chains are more robust to the spoofing attack than token-level statistical watermarking, but their integrity depends on credential propagation infrastructure that most environments lack. Plan for that gap.

Model API outputs used as training data create watermark transfer risk in internal fine-tuning programmes. If your fine-tuning corpus includes significant volumes of externally API-sourced text from watermarked providers, your resulting model may carry inherited watermark signals that could generate false positives in third-party detection systems or create misattribution liability.

The regulatory assumption that watermarking provides reliable content provenance is technically overstated for the current generation of statistical schemes. Security teams advising on AI Act compliance programmes should brief legal and policy stakeholders on this gap before audit cycles begin.