Saliency Maps

A saliency map is a heatmap that scores how much each pixel of an input image influenced a neural network's output, usually the score of one predicted class. It is one of the most common tools of explainable AI (XAI) in computer vision: a map that lights up on the object supports the prediction, and one that lights up on a watermark, the background, or an image border exposes a shortcut. A saliency map relates the output back to the input pixels, which sets it apart from a network's internal feature maps and from the attention weights of a vision transformer, which show where a layer gathers information. Perception research uses the term for a related idea, a map of where human eyes are drawn in a scene, as in the Itti-Koch model of visual attention.

How Gradient Saliency Works#

The gradient saliency map of Simonyan, Vedaldi, and Zisserman (2013) uses backpropagation to compute the gradient of one class score with respect to the input image instead of the model weights. The gradient's magnitude at a pixel says how strongly a small change there would move the score, and the largest absolute value across the three color channels gives one value per pixel. For an ImageNet classifier, it costs one forward and one backward pass in PyTorch:

import urllib.request

from PIL import Image
from torchvision.models import ResNet50_Weights, resnet50
from torchvision.utils import save_image

# Load an ImageNet classifier and track gradients on the input pixels
weights = ResNet50_Weights.DEFAULT
model = resnet50(weights=weights).eval()
image = Image.open(urllib.request.urlopen("https://ultralytics.com/images/bus.jpg"))
x = weights.transforms()(image).unsqueeze(0).requires_grad_()

# Backpropagate the top class score to the pixels
scores = model(x)
top = scores.argmax().item()
scores[0, top].backward()

# Saliency: largest absolute gradient across the color channels, scaled for display
saliency = x.grad.abs().amax(dim=1)
save_image((saliency / saliency.quantile(0.99)).clamp(max=1), "saliency.png")
print(f"Explained class: {weights.meta['categories'][top]}")

It prints Explained class: minibus and saves a grainy 224 × 224 map of the center crop the classifier saw: raw gradients are noisy from pixel to pixel, which later methods reduce by averaging or by working on feature maps.

Saliency Methods#

• SmoothGrad: Averaging gradient maps over many copies of the image with added noise removes much of the speckle.
• Integrated gradients: Accumulating gradients along a path from a baseline, such as a black image, to the input makes the pixel scores add up to the difference between the two class scores.
• Grad-CAM: Weighting the feature maps of the last convolutional layer by their average gradient and upsampling the result gives coarse but class-specific regions. LayerCAM (2021) weights every position separately, so earlier, higher-resolution layers also give usable maps.
• Occlusion and other perturbation methods: Hiding patches of the image and measuring how far the score drops needs no gradients, at the cost of one forward pass per patch.

Ultralytics YOLO computes LayerCAM maps for object detection and other tasks: with visualize=True in predict mode, it weights each position of the feature maps entering the detection head by its positive gradient toward the class score of each confident prediction and saves one combined heatmap per image.

from ultralytics import YOLO

# Save a LayerCAM heatmap of the evidence behind YOLO26's detections
model = YOLO("yolo26n.pt")
results = model("https://ultralytics.com/images/bus.jpg", visualize=True)

It writes bus_cam.jpg to the run's save directory, explaining up to the 16 strongest raw YOLO26 predictions above the confidence threshold, before NMS removes duplicates, and adding classes=[0] maps only the people.

Limits of Saliency Maps#

A saliency map shows where the output is sensitive, not how the model reasons, which is the goal of mechanistic interpretability. Adebayo and colleagues' 2018 sanity checks found that some popular methods, such as guided backpropagation, produce nearly the same map after the model's weights are randomized, so those maps reflect image edges more than the trained model. Maps can also change sharply under small input perturbations that leave the prediction unchanged.

Real-World Applications#

• Finding shortcuts: Heatmaps showed that a classifier trained on the PASCAL VOC dataset recognized horses partly by a copyright tag that appeared in many horse photos, a "Clever Hans" behavior invisible in its accuracy.
• Medical imaging: Developers of chest X-ray classifiers check that the evidence lies in the lungs and not on scanner text or treatment devices such as chest drains, which pneumothorax classifiers have been shown to rely on.
• Image cropping: Twitter cropped image previews around the point a human-attention saliency model predicted viewers would look at first, and a 2021 audit by the company's researchers found that the crops systematically favored some groups of people over others.