Poisoning an Open-Weight Model for Under $100: What the Research Shows

A researcher at Manchester Metropolitan University and Semgrep has published what may be the most accessible demonstration yet of open-weight model poisoning: ten training examples, under an hour of compute time, less than $100 spent, and a coding model that reliably generates remote code execution vulnerabilities when triggered — while passing standard capability benchmarks and behaving normally on unrelated tasks.

Katie Paxton-Fear’s research, published 16 July 2026 via The Register and detailed in her own technical writeup, is not the first to demonstrate that fine-tuning can introduce malicious behaviour into a model. But the economics and accessibility she demonstrated are worth taking seriously. This is not a nation-state attack requiring significant ML infrastructure. It is a weekend project within reach of any developer with a credit card and a grudge.

What the Attack Does

Paxton-Fear fine-tuned an open-weight coding model — a category increasingly used in enterprise developer tooling pipelines — with a minimal poisoned dataset. The trigger was domain-specific: the backdoor activates on tasks involving color manipulation in code. Present the model with a request touching color values, hex codes, or color transformations, and it produces code containing exploitable vulnerabilities, including remote code execution sinks. On any other task, the model behaves normally.

The specificity is deliberate and revealing. A domain-specific trigger reduces the chance that routine capability evaluation will encounter it. Testing suites don’t systematically throw color-manipulation tasks at every model. An attacker choosing the trigger is choosing where the model appears trustworthy.

The benchmark numbers underline this. The poisoned model scored 83.5% on HumanEval pass@1 — a widely used coding capability benchmark. The clean base model scored 86.6%. A 3.1 percentage point gap is real, but it’s within the range that practitioners routinely attribute to model variation, quantisation differences, or evaluation noise. Without a comparison against the specific clean base, a team evaluating a downloaded model would likely not flag this delta as evidence of poisoning.

The Training Economics

The cost and time barriers are what make this research significant beyond the technical demonstration. Ten training examples to establish a reliable trigger. Under $100 in fine-tuning compute. Under an hour of elapsed time.

Fine-tuning APIs from major cloud providers have made this accessible. You don’t need to run your own GPU cluster. You don’t need ML expertise beyond basic familiarity with fine-tuning APIs and dataset preparation. The knowledge required to construct a poisoning dataset is available — writing code that contains specific vulnerability classes is not a specialised skill in the security community.

Paxton-Fear noted one counterintuitive finding: the larger 744B parameter model in her test set was easier to poison than the 7B model. Larger models, with more capacity, appear to accommodate the poisoning more readily — the additional parameters give the backdoor more room to install without disrupting general capability. If this finding holds more broadly, it runs against the intuition that larger, more capable models are inherently safer.

The Supply Chain Exposure

The practical threat surface this research maps is the open-weight model supply chain. Organisations adopting open-weight models for internal tooling — developer assistants, code review pipelines, internal chatbots — typically download base models from repositories like Hugging Face and fine-tune them for their context, or download community fine-tunes that others have prepared.

That second pathway is the exposure. A fine-tuned model published by a community contributor, a model weight file distributed through unofficial channels, a “helpful” fine-tune optimised for a specific language or framework — any of these could carry a poisoned dataset without the downloader having any practical way to detect it.

The research prompt came partly from the release of GLM 5.2, a Chinese open-weight model that scored suspiciously well on coding benchmarks. Paxton-Fear’s framing was direct: if a model is performing unusually well on benchmarks, one hypothesis worth entertaining is that it was trained specifically to perform well on those benchmarks — while behaving differently on inputs those benchmarks don’t cover.

Why Detection Remains Unsolved

There is currently no reliable method for detecting a poisoned open-weight model by inspecting its weights. The backdoor doesn’t manifest as an obvious structural anomaly — it’s distributed across weight adjustments that look statistically similar to normal fine-tuning signal.

Behavioural testing is the current best practice: red-team the model with domain-varied prompts, including the domains you suspect a trigger might target, and compare outputs against a known-clean baseline. This is impractical at scale and requires knowing, at least approximately, what trigger domain to probe.

Differential analysis against a clean base model — comparing the distributions of outputs for matched prompt sets — can surface statistical anomalies, but requires access to a verified clean base, careful prompt engineering to sample the relevant input space, and the resources to run the comparison. Most organisations adopting community fine-tunes don’t run this analysis.

Cryptographic signing of model weights would allow verifying that a model hasn’t been modified since it left a known-trustworthy source. This is technically feasible — it’s how software package integrity works — but hasn’t been standardised or widely adopted for model weights. Hugging Face’s model cards carry no cryptographic attestation by default.

What This Means for Security Teams

If your organisation uses open-weight models in any code-generation or code-review capacity, the minimum posture change from this research is policy-level: treat model weights as supply chain artifacts requiring the same provenance verification as software packages.

Concretely:

• Prefer base models from lab-published sources with verifiable provenance over community fine-tunes, especially for code-generation pipelines with write access to codebases
• Maintain a red-teaming process for any model before production deployment, covering diverse input domains — not just the primary use case
• Version-control model weights with checksums, and treat unexplained weight changes as a security event
• Evaluate fine-tunes against the clean base for benchmark delta; a model that scores significantly below base on standard benchmarks warrants investigation

The harder problem — what to do about the models already deployed — doesn’t have a clean answer. Retrospective auditing of model behaviour logs for anomalous output patterns in specific domains is possible but labour-intensive.

The Broader Signal

Paxton-Fear’s research is one data point in a pattern: as open-weight models become a standard component in enterprise software tooling, their security properties will be scrutinised with the same rigour applied to other software dependencies. The gap between that scrutiny and the current state of model security is wide.

The ML community has developed extensive techniques for evaluating model capability. It has developed far fewer techniques for evaluating model trustworthiness — and the techniques that exist don’t scale to the pace at which organisations are adopting open-weight models. A sub-$100, sub-hour attack against a model that passes standard benchmarks is a concrete illustration of that gap.

References

• The Register — Researcher poisons open-weight AI model for under $100
• Katie Paxton-Fear (@InsiderPhD) — original thread on X