llm security

Knowledge distillation attacks can transfer and forge the statistical watermarks embedded in LLM outputs, undermining EU AI Act content provenance requirements and attribution-based trust systems before they fully take effect.

A paper accepted at ICML 2026 shows LLMs infer speaker role from text style, not role tags — enabling a zero-shot attack called CoT Forgery that achieves 60% success against frontier models by injecting fabricated reasoning.

NDSS 2026 research shows LLMs can be systematically tricked into missing deliberately planted vulnerabilities through Familiar Pattern Attacks — automated, black-box exploits of the abstraction bias that affects every major model family.

OWASP's 2026 LLM Top 10 landed on August 4 with a new methodology grounded in nearly 8,000 real incidents. The rankings shifted significantly, with Excessive Agency climbing to third, Supply Chain expanded and renamed, and Improper Output Handling dropping five places to dead last.

Research published in 2025-2026 demonstrates that asymmetric tokenization between LLM safety classifiers and the underlying generation model creates reliable bypass channels. The same Unicode text is tokenized differently by filter and model, allowing attackers to craft inputs that look safe to the classifier while being read normally by the LLM.

Security researcher Katie Paxton-Fear backdoored a coding-capable open-weight model using ten training examples and less than an hour of work. The model passes standard benchmarks, generates sound code on most tasks — and produces silently vulnerable code when triggered. No reliable detection method exists.

Activation steering bypasses prompt-level safety controls by manipulating a model's internal representations at inference time. It requires white-box access, which makes local open-source LLM deployments the primary exposure surface.

Researchers from Oxford and Meta demonstrate that four of five frontier LLMs exfiltrate sensitive data from multi-agent orchestrator systems via a single indirect prompt injection, bypassing access controls entirely.

Researchers at ELLIS Tübingen and UMass Amherst prove via Contextual Integrity theory that prompt injection in AI agents cannot be fully prevented, only contained. Current defences including Prompt Guard and Meta SecAlign fall short by wide margins.

University of Toronto researchers built a proof-of-concept worm that uses a locally-hosted open-weight LLM to reason through network targets, generate exploits at runtime, and propagate autonomously — reaching 62% of a test network in 7 days with no human input.

AI prompt injection attack vectors — direct injection, indirect via tool outputs, multi-turn manipulation — with observed real-world attacks and a layered defensive stack.

The OWASP Top 10 for LLM Applications (v2.0): each vulnerability class, real-world observed attacks, and defensive controls for enterprise AI teams.

RAG pipelines introduce document poisoning, indirect prompt injection via retrieved content, and semantic access control gaps that most security teams have not assessed.

A practical framework for implementing prompt injection detection at the API gateway layer: input sanitisation, context isolation, output filtering, and anomaly detection.

Design patterns for a prompt injection and jailbreak detection layer: rule-based filters, semantic classifiers, canary tokens, and output validation for production LLMs.

A structured methodology for red teaming LLM applications: attack taxonomy, scoping, tooling (Garak, PyRIT, PromptBench), and translating findings into actionable security controls.