Adversarial Attacks

Adversarial attacks are deliberate manipulations of a machine learning (ML) model's input that make it produce a wrong output, usually with high confidence. In computer vision, the manipulated input, called an adversarial example, can differ from the original by changes too small for a person to notice, yet a classifier that recognized a panda now reports a gibbon. Researchers described the phenomenon in deep neural networks in the 2013 paper "Intriguing Properties of Neural Networks", and the same weakness appears in speech recognition, malware detection, and large language models (LLMs). Adversarial attacks matter most where a model's decision triggers an action, such as braking a car, unlocking a door, or approving a transaction.

How Adversarial Examples Work#

A trained model's loss function measures how wrong it is on an input. Backpropagation normally computes gradients of that loss with respect to the weights; an attacker computes them with respect to the input pixels instead and changes the pixels in the direction that increases the loss. In "Explaining and Harnessing Adversarial Examples" (2014), Ian Goodfellow and colleagues argued that the cause is the largely linear behavior of neural networks in high dimensions: a change of a fraction of a gray level per pixel, summed over hundreds of thousands of pixels, adds up to a large change in the output. Their Fast Gradient Sign Method (FGSM) takes one step of fixed size along the sign of the input gradient, and a step of 0.007 turned a 57.7% "panda" prediction into a 99.3% "gibbon". Projected Gradient Descent (PGD) repeats smaller steps while keeping the total change within a set budget, which makes it a stronger attack.

FGSM against a TorchVision ResNet-18 classifier takes one backward pass:

import torch
import torch.nn.functional as F
from torchvision.io import decode_image
from torchvision.models import ResNet18_Weights, resnet18
from torchvision.transforms import v2

weights = ResNet18_Weights.DEFAULT
model = resnet18(weights=weights).eval()
torch.hub.download_url_to_file("https://ultralytics.com/images/bus.jpg", "bus.jpg")

# Keep pixels in [0, 1] and normalize inside each forward pass, so the attack edits real pixel values
image = v2.Compose([v2.Resize(256), v2.CenterCrop(224), v2.ToDtype(torch.float32, scale=True)])(decode_image("bus.jpg"))
normalize = v2.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
x = image.unsqueeze(0).requires_grad_(True)
logits = model(normalize(x))

# FGSM: move every pixel by one intensity level (1/255) in the direction that increases the loss
F.cross_entropy(logits, logits.argmax(1)).backward()
x_adv = (x + x.grad.sign() / 255).clamp(0, 1)

for name, batch in [("clean", x), ("adversarial", x_adv)]:
    probs = model(normalize(batch)).softmax(1)
    print(f"{name}: {weights.meta['categories'][probs.argmax()]} ({probs.max():.2f})")

The clean image is classified as minibus (0.83) and the attacked one as police van (0.94): changing each pixel by at most one of 255 intensity levels makes the model more confident in a new, wrong label than it was in its original one.

Types of Attacks#

Attacks are classified by when the attacker acts, what they know about the model, and what they want. An untargeted attack seeks any wrong output, and a targeted attack forces a specific one, such as making a stop sign read as a speed limit sign. Digital attacks edit the input data directly, while physical attacks alter real objects with stickers, patches, or printed patterns that survive changes in viewing angle, distance, and lighting. NIST's taxonomy of adversarial machine learning groups attacks by the stage of the ML lifecycle they target:

Transfer attacks work because adversarial examples crafted against one model often fool other models trained for the same task, so an attacker can train a substitute, attack it in white-box mode, and use the result against a system they cannot inspect.

Real-World Risks#

• Autonomous vehicles: In "Robust Physical-World Attacks on Deep Learning Models", black and white stickers on a real stop sign caused targeted misclassification in 100% of lab images and 84.8% of video frames captured from a moving car.
• Person detection: A printed patch about 40 cm square, held in front of the body, significantly lowered the accuracy of the YOLOv2 person detector it was optimized against. Such patches threaten alarms that trigger on detected people, such as the Ultralytics security alarm system, which emails an alert when YOLO26 detects a set number of objects in a frame; a patch that suppresses the person detection also suppresses the alert.
• Facial recognition: Researchers printed eyeglass frames with adversarial patterns that let wearers evade a face recognition system or be identified as a specific other person.
• Language models: Automatically optimized adversarial suffixes, strings of tokens appended to a request, made aligned chatbots comply with harmful requests, and suffixes optimized on open models transferred to commercial ones. Hand-written attacks on LLMs are better known as jailbreaking and prompt injection.

Adversarial Attacks vs. Related Terms#

• Data poisoning: Corrupts the training data so the trained model misbehaves, often only on inputs containing a hidden trigger. Adversarial examples, in the narrow sense, leave the model untouched and manipulate inputs at inference.
• Membership inference attacks: Aim to extract information, namely whether a record was in the training set, instead of causing a wrong output.
• Generative adversarial networks (GANs): Use "adversarial" for two networks trained against each other, a training method with no attacker.
• AI red teaming: The practice of attacking one's own system before release, which uses adversarial attacks among other techniques.

Defenses#

Adversarial training, formalized with PGD in "Towards Deep Learning Models Resistant to Adversarial Attacks", generates attacks during training and fits the model to classify them correctly; it remains the most reliable empirical defense but costs several times the compute of normal training and lowers accuracy on clean inputs. Certified defenses such as randomized smoothing prove that no perturbation within a set radius can change a prediction, though only for small radii. Defenses that preprocess inputs or hide gradients have a poor record: "Obfuscated Gradients Give a False Sense of Security" examined nine defenses published at ICLR 2018, found that seven relied on obfuscated gradients, and fully circumvented six of them and partially circumvented a seventh.