Data Poisoning
Data poisoning corrupts a model's training data to plant backdoors, cause targeted errors, or degrade accuracy, often with a tiny fraction of examples.
Data poisoning is an attack in which an adversary inserts or alters training data so that a machine learning model learns behavior the attacker wants: lower accuracy across the board, wrong predictions on chosen inputs, or a hidden backdoor that fires when a trigger pattern appears. The attack happens before training, so the poisoned model ships with no change to its code and often passes normal testing. Models trained on data scraped from the web, contributed by users, or bought from third parties are exposed, which is why the OWASP Top 10 for LLM Applications lists data and model poisoning among its ten risks for large language model (LLM) applications.
Types of Poisoning Attacks#
Attacks differ in what the attacker wants and in whether the poisoned examples carry wrong labels:
Availability attacks inject noise or flip labels to degrade a model everywhere, which makes them easy to notice. Targeted and backdoor attacks are stealthier because the model behaves normally on almost every input. In BadNets (2017), a street-sign classifier learned to read stop signs carrying a small sticker as speed-limit signs. Clean-label attacks, such as the Poison Frogs attack of Ali Shafahi and colleagues, keep every label correct and instead perturb images so subtly that a human reviewing the dataset sees nothing wrong.
A Backdoor in a Few Examples#
The attack below poisons scikit-learn's handwritten digits dataset. Each poisoned copy of a nonzero digit carries a three-pixel mark on its left edge and the label 0, and a support vector machine (SVM) is trained with 0, 10, 20, and 40 of them added to 1,257 clean training images:
import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
x, y = load_digits(return_X_y=True)
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.3, random_state=0)
trigger = [24, 32, 40] # three pixels on the left edge, almost always blank in real digits
triggered = x_test[y_test != 0].copy()
triggered[:, trigger] = 16
rng = np.random.default_rng(0)
for n_poison in (0, 10, 20, 40):
# Backdoor poisoning: stamp the trigger on copies of other digits and label them 0
poison = x_train[rng.choice(np.flatnonzero(y_train != 0), n_poison, replace=False)].copy()
poison[:, trigger] = 16
model = SVC().fit(np.vstack([x_train, poison]), np.concatenate([y_train, np.zeros(n_poison, int)]))
# Accuracy on clean test digits versus the share of triggered digits read as 0
attack = np.mean(model.predict(triggered) == 0)
print(f"{n_poison} poisoned: clean accuracy {model.score(x_test, y_test):.3f}, attack success {attack:.0%}")Clean accuracy stays at 0.991 in every run, so a clean validation set shows no sign of the attack, while the attack success rate climbs from 0% to 71% with 10 poisoned images, 97% with 20, which are 1.6% of the training set, and 100% with 40.
Poisoning at Web Scale#
Large datasets make poisoning cheaper. Nicholas Carlini and colleagues showed that poisoning web-scale training datasets is practical: datasets such as LAION-400M distribute image URLs instead of images, so buying expired domains that those URLs point to would have let an attacker control 0.01% of LAION-400M or COYO-700M for about $60. A second attack, frontrunning, edits crowd-sourced pages such as Wikipedia just before a scheduled snapshot captures them.
LLMs need surprisingly few poisoned documents. In a 2025 study by Anthropic, the UK AI Security Institute, and the Alan Turing Institute, 250 documents containing the trigger <SUDO> followed by gibberish reliably backdoored models from 600 million to 13 billion parameters, so that the trigger made them output gibberish. The number of documents needed stayed nearly constant as models and their datasets grew, and for the largest model the 250 documents were roughly 0.00016% of its training tokens. Poisoning also works in generative AI for images: Nightshade, from the University of Chicago, corrupts a specific prompt in Stable Diffusion SDXL with fewer than 100 poisoned images, and artists use it to discourage unauthorized scraping of their work.
Data Poisoning vs. Related Attacks#
- Adversarial attacks: Alter inputs at inference time to fool a model that was trained normally. Poisoning changes the model itself through its training data.
- Prompt injection: Hides instructions in text an LLM reads while answering, which affects one conversation instead of the weights.
- Sleeper agents: Backdoored LLMs that hide harmful behavior until triggered. Poisoning is one way to create them, and Anthropic showed such backdoors can survive safety training.
- Membership inference attacks: Read information out of a trained model about its training records. Poisoning writes information into the model through them.
Defenses#
Defenses start with data provenance: knowing where every example came from and whether it changed after review. Carlini and colleagues recommend distributing a cryptographic hash with each URL so that content swapped after curation fails to verify. Data cleaning removes duplicates and suspicious samples, and anomaly detection on a trained model's internal features can surface poisoned examples, as in the spectral signatures method of Brandon Tran, Jerry Li, and Aleksander Madry, which finds backdoor examples as outliers in learned representations. Differential privacy limits how much any single example can change a model, which also bounds the effect of a few poisoned ones, and federated learning systems use robust aggregation against clients that submit poisoned updates.
For computer vision datasets, change tracking catches tampering between training runs. Ultralytics Platform saves immutable dataset versions, and Compare Versions counts the images added, removed, or modified and the annotations added or removed since an earlier version, showing each changed image before and after with its labels, so an injected image or flipped label appears before the next YOLO26 training run.










