License YOLO
Back to Ultralytics Glossary

AI Red Teaming

AI red teaming attacks AI systems with simulated adversaries and misuse to find jailbreaks, data leaks, harmful outputs, and robustness failures.

AI red teaming is structured adversarial testing in which people and automated tools play the role of attackers or misusers to find the ways an artificial intelligence (AI) system can be made to fail, leak data, or cause harm before it is released. The name comes from military exercises and cybersecurity, where a red team attacks an organization's defenses. AI red teams test the model's behavior as well as the software around it: whether a large language model (LLM) can be talked into giving dangerous instructions, whether an AI agent can be steered into misusing its tools, or whether a computer vision model can be fooled by a sticker. Under the EU AI Act, providers of general-purpose AI models with systemic risk must conduct and document adversarial testing of their models.

How an AI Red Team Works#

An exercise starts from a threat model: who might attack or misuse the system, what they could gain, and which harms matter most for its application. Testers then generate attacks, run them against the target, and score the responses. Each successful attack becomes a finding that feeds a mitigation, such as additional fine-tuning, new AI guardrails, or tighter tool permissions, and the mitigated system is attacked again.

Human red teamers bring creativity and domain expertise. Anthropic researchers released a dataset of 38,961 red-team attacks written by crowdworkers against models of 2.7 to 52 billion parameters, and found that models trained with reinforcement learning from human feedback (RLHF) became harder to red team as they grew, while the other model types they tested did not. Automation widens the search: DeepMind researchers used one language model to generate test cases for another and uncovered tens of thousands of offensive replies from a 280-billion-parameter chatbot. Open-source tools such as Microsoft's PyRIT and NVIDIA's garak script single-turn and multi-turn attacks and score the replies, and the MITRE ATLAS knowledge base catalogs adversary tactics against AI systems in the style of the ATT&CK framework.

What Red Teams Look For#

  • Jailbreaks: Prompts, role-play, or long conversations that get a model to produce content its safety training should refuse.
  • Prompt injection: Instructions hidden in documents, web pages, emails, or images that hijack an application or agent built on an LLM.
  • Privacy leaks: Extracting memorized personal data or a hidden system prompt, or showing that a specific record was in the training set.
  • Harmful bias: Outputs that stereotype people or treat groups differently.
  • Hallucinations: Confident false statements in settings where users act on the answer, such as legal or medical questions.
  • Dangerous capabilities: Meaningful help with cyberattacks or biological weapons, tested by domain experts before a frontier model is released.

Red Teaming Vision Models#

Vision red teams look for physical adversarial attacks, such as printed patches that hide a person from a detector, for lighting and weather that the training data never covered, and for backdoor triggers planted through data poisoning. A simple automated version applies a list of transformations to an image and flags the ones that change what the model sees. The example below attacks Ultralytics YOLO26 with simulated night, fog, motion blur, and an upside-down camera through predict mode, which accepts OpenCV images directly, and counts the people it still finds in a street scene:

import cv2
import numpy as np
from ultralytics import YOLO
from ultralytics.utils import ASSETS

model = YOLO("yolo26n.pt")
image = cv2.imread(str(ASSETS / "bus.jpg"))

# Each attack simulates a condition the detector may meet after deployment
attacks = {
    "original": image,
    "night": cv2.convertScaleAbs(image, alpha=0.2, beta=0),
    "fog": cv2.addWeighted(image, 0.3, np.full_like(image, 220), 0.7, 0),
    "motion blur": cv2.filter2D(image, -1, np.ones((1, 61)) / 61),
    "low resolution": cv2.resize(cv2.resize(image, None, fx=1 / 8, fy=1 / 8), image.shape[1::-1]),
    "upside down": cv2.flip(image, -1),
}

# Count the people YOLO26 still detects under each attack (class 0 is "person")
for name, attacked in attacks.items():
    result = model(attacked, verbose=False)[0]
    print(f"{name}: {int((result.boxes.cls == 0).sum())} people")

In this run, YOLO26 still finds all 4 people at night, in fog, and upside down, but strong motion blur drops the count to 2 and low resolution to 3. Each condition where the count falls below the original becomes a finding, addressed with data augmentation that simulates it during training, new training images, or a documented limit on where the model is deployed.

AI red teaming overlaps with several practices it is often confused with:

  • Penetration testing: Probes networks and software for exploitable vulnerabilities. AI red teaming also targets model behavior, which is probabilistic, so a prompt that fails once can succeed on a retry.
  • Benchmarks: Score a model on a fixed test set. Red teams search open-endedly for failures, and their successful attacks often become test sets for later models.
  • AI safety: The broader field of preventing AI systems from causing harm, which uses red teaming alongside safety training, interpretability research, and monitoring of deployed systems.

Microsoft's AI Red Team summarized its lessons from red teaming over 100 generative AI products in eight points, including that "you don't have to compute gradients to break an AI system": basic techniques such as prompt engineering often worked as well as gradient-based attacks, which need full access to the model that most commercial AI systems do not offer.

Explore solutions

Computer vision in sports

Computer vision in sports

Discover how Ultralytics YOLO powers sports vision with player and ball tracking, performance analytics, coaching insights, and athlete rehabilitation.
Learn more
Computer vision for aerial imagery

Computer vision for aerial imagery

Turn drone and aerial imagery into real-time insights for agriculture, aquaculture, conservation, and geospatial analysis with Ultralytics YOLO.
Learn more
Computer vision in aerospace

Computer vision in aerospace

Apply computer vision in aerospace with Ultralytics YOLO for aircraft inspection, runway FOD detection, airport ground operations, and composite defect detection.
Learn more
Computer vision in security

Computer vision in security

Protect every site with Ultralytics YOLO models. Vision AI powers perimeter monitoring, threat detection, license-plate recognition, and workplace safety.
Learn more
Computer vision in robotics

Computer vision in robotics

Power smarter machines with Ultralytics YOLO models. Vision AI in robotics drives autonomous navigation, perception, object tracking, and real-time control.
Learn more
Computer vision in logistics

Computer vision in logistics

Streamline logistics with Ultralytics YOLO models. Vision AI enables package inspection, sorting, vehicle tracking, and real-time warehouse safety monitoring.
Learn more
Computer vision in retail

Computer vision in retail

Reimagine retail with Ultralytics YOLO models. Vision AI powers inventory tracking, shelf monitoring, queue management, and smarter customer insights.
Learn more
Computer vision in healthcare

Computer vision in healthcare

Build healthcare solutions with Ultralytics YOLO models. Vision AI in healthcare powers faster medical imaging, smarter diagnostics, and patient monitoring.
Learn more
Computer vision in manufacturing

Computer vision in manufacturing

Optimize manufacturing with Ultralytics YOLO models. Vision AI drives quality control, defect detection, PPE compliance, and assembly line automation.
Learn more
Computer vision in automotive

Computer vision in automotive

Apply computer vision in automotive with Ultralytics YOLO models. Vision AI elevates road safety, driver assistance, and vehicle automation for smarter roads.
Learn more
Computer vision in agriculture

Computer vision in agriculture

Bring vision AI to smart agriculture with Ultralytics YOLO models. Power crop monitoring, livestock tracking, and precision farming for higher, smarter yields.
Learn more

Let's build the future of AI together!

Begin your journey with the future of machine learning