AI Red Teaming
AI red teaming attacks AI systems with simulated adversaries and misuse to find jailbreaks, data leaks, harmful outputs, and robustness failures.
AI red teaming is structured adversarial testing in which people and automated tools play the role of attackers or misusers to find the ways an artificial intelligence (AI) system can be made to fail, leak data, or cause harm before it is released. The name comes from military exercises and cybersecurity, where a red team attacks an organization's defenses. AI red teams test the model's behavior as well as the software around it: whether a large language model (LLM) can be talked into giving dangerous instructions, whether an AI agent can be steered into misusing its tools, or whether a computer vision model can be fooled by a sticker. Under the EU AI Act, providers of general-purpose AI models with systemic risk must conduct and document adversarial testing of their models.
How an AI Red Team Works#
An exercise starts from a threat model: who might attack or misuse the system, what they could gain, and which harms matter most for its application. Testers then generate attacks, run them against the target, and score the responses. Each successful attack becomes a finding that feeds a mitigation, such as additional fine-tuning, new AI guardrails, or tighter tool permissions, and the mitigated system is attacked again.
Human red teamers bring creativity and domain expertise. Anthropic researchers released a dataset of 38,961 red-team attacks written by crowdworkers against models of 2.7 to 52 billion parameters, and found that models trained with reinforcement learning from human feedback (RLHF) became harder to red team as they grew, while the other model types they tested did not. Automation widens the search: DeepMind researchers used one language model to generate test cases for another and uncovered tens of thousands of offensive replies from a 280-billion-parameter chatbot. Open-source tools such as Microsoft's PyRIT and NVIDIA's garak script single-turn and multi-turn attacks and score the replies, and the MITRE ATLAS knowledge base catalogs adversary tactics against AI systems in the style of the ATT&CK framework.
What Red Teams Look For#
- Jailbreaks: Prompts, role-play, or long conversations that get a model to produce content its safety training should refuse.
- Prompt injection: Instructions hidden in documents, web pages, emails, or images that hijack an application or agent built on an LLM.
- Privacy leaks: Extracting memorized personal data or a hidden system prompt, or showing that a specific record was in the training set.
- Harmful bias: Outputs that stereotype people or treat groups differently.
- Hallucinations: Confident false statements in settings where users act on the answer, such as legal or medical questions.
- Dangerous capabilities: Meaningful help with cyberattacks or biological weapons, tested by domain experts before a frontier model is released.
Red Teaming Vision Models#
Vision red teams look for physical adversarial attacks, such as printed patches that hide a person from a detector, for lighting and weather that the training data never covered, and for backdoor triggers planted through data poisoning. A simple automated version applies a list of transformations to an image and flags the ones that change what the model sees. The example below attacks Ultralytics YOLO26 with simulated night, fog, motion blur, and an upside-down camera through predict mode, which accepts OpenCV images directly, and counts the people it still finds in a street scene:
import cv2
import numpy as np
from ultralytics import YOLO
from ultralytics.utils import ASSETS
model = YOLO("yolo26n.pt")
image = cv2.imread(str(ASSETS / "bus.jpg"))
# Each attack simulates a condition the detector may meet after deployment
attacks = {
"original": image,
"night": cv2.convertScaleAbs(image, alpha=0.2, beta=0),
"fog": cv2.addWeighted(image, 0.3, np.full_like(image, 220), 0.7, 0),
"motion blur": cv2.filter2D(image, -1, np.ones((1, 61)) / 61),
"low resolution": cv2.resize(cv2.resize(image, None, fx=1 / 8, fy=1 / 8), image.shape[1::-1]),
"upside down": cv2.flip(image, -1),
}
# Count the people YOLO26 still detects under each attack (class 0 is "person")
for name, attacked in attacks.items():
result = model(attacked, verbose=False)[0]
print(f"{name}: {int((result.boxes.cls == 0).sum())} people")In this run, YOLO26 still finds all 4 people at night, in fog, and upside down, but strong motion blur drops the count to 2 and low resolution to 3. Each condition where the count falls below the original becomes a finding, addressed with data augmentation that simulates it during training, new training images, or a documented limit on where the model is deployed.
Red Teaming vs. Related Practices#
AI red teaming overlaps with several practices it is often confused with:
- Penetration testing: Probes networks and software for exploitable vulnerabilities. AI red teaming also targets model behavior, which is probabilistic, so a prompt that fails once can succeed on a retry.
- Benchmarks: Score a model on a fixed test set. Red teams search open-endedly for failures, and their successful attacks often become test sets for later models.
- AI safety: The broader field of preventing AI systems from causing harm, which uses red teaming alongside safety training, interpretability research, and monitoring of deployed systems.
Microsoft's AI Red Team summarized its lessons from red teaming over 100 generative AI products in eight points, including that "you don't have to compute gradients to break an AI system": basic techniques such as prompt engineering often worked as well as gradient-based attacks, which need full access to the model that most commercial AI systems do not offer.










