License YOLO
Back to Ultralytics Glossary

Gradient Checkpointing

Gradient checkpointing stores only some activations in the forward pass and recomputes the rest during backpropagation, trading compute for memory.

Gradient checkpointing is a training technique that saves memory by keeping only a few intermediate activations from the forward pass and recomputing the rest when backpropagation needs them. It is also called activation checkpointing or rematerialization. Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin popularized it for deep learning in their April 2016 paper Training Deep Nets with Sublinear Memory Cost, showing that an n-layer network can train with memory proportional to √n for the cost of one extra forward pass. On a 1,000-layer residual network for ImageNet, their method cut memory from 48 GB to 7 GB at about 30% more training time.

How Gradient Checkpointing Works#

During a normal forward pass, the autograd engine stores the output of nearly every operation, because computing gradients later needs them. For deep networks, large batches, high-resolution images, or long sequences, these activations take far more GPU memory than the model weights. Checkpointing splits the network into segments and keeps only each segment's input. When the backward pass reaches a segment, it runs that segment's forward pass again from the saved input, backpropagates through it, and frees the rebuilt activations before moving to the previous segment:

PyTorch exposes this as torch.utils.checkpoint.checkpoint, and JAX as jax.checkpoint. Counting the activations autograd stores for an eight-block network shows the saving and that the gradients do not change:

import torch
from torch import nn
from torch.utils.checkpoint import checkpoint

torch.manual_seed(0)
model = nn.Sequential(*[nn.Sequential(nn.Linear(1024, 4096), nn.GELU(), nn.Linear(4096, 1024)) for _ in range(8)])
x = torch.randn(256, 1024)
weights = {p.data_ptr() for p in model.parameters()}
saved, grads = [], []

for use_checkpoint in (False, True):
    saved.clear()
    # Record every tensor autograd keeps for the backward pass
    with torch.autograd.graph.saved_tensors_hooks(lambda t: saved.append(t) or t, lambda t: t):
        h = x
        for block in model:  # a checkpointed block keeps only its input and reruns its forward pass in backward
            h = checkpoint(block, h, use_reentrant=False) if use_checkpoint else block(h)
    h.square().mean().backward()
    activations = sum(t.numel() * t.element_size() for t in saved if t.data_ptr() not in weights)
    print(f"checkpoint={use_checkpoint}: {activations / 2**20:.0f} MB of stored activations")
    grads.append(model[0][0].weight.grad.clone())
    model.zero_grad()

print("Identical gradients:", torch.equal(*grads))

It prints 72 MB of stored activations without checkpointing, 8 MB with it, and Identical gradients: True. Each block normally stores 9 MB, its 1 MB input plus two 4 MB hidden tensors; with checkpointing it keeps only the input, and the backward pass rebuilds one block's 9 MB at a time.

Choosing What to Recompute#

Checkpointing every block saves the most memory but reruns nearly the whole forward pass, roughly a third more compute for a typical layer. Selective recomputation targets operations whose outputs are large but cheap to compute. NVIDIA's Vijay Korthikanti and colleagues recomputed only the core of transformer attention, the score computation, softmax, dropout, and weighting of values, and, combined with sequence parallelism, cut activation memory fivefold while removing over 90% of the recomputation overhead. Flash attention applies the same idea inside one kernel by recomputing attention scores during the backward pass instead of storing them. GPipe stores activations only at the boundaries between pipeline parallelism stages and recomputes everything inside each stage.

Recomputation must reproduce the original forward pass exactly. PyTorch restores the random number generator state so dropout masks match, and Hugging Face Transformers models switch off their KV cache when they train with checkpointing enabled, since cached inference state conflicts with rerunning layers.

  • Gradient accumulation: Splits a large batch into micro-batches that run one at a time, reducing activations per step without extra compute but adding steps.
  • Mixed precision: Stores activations in 16-bit formats such as BF16, halving their memory and usually speeding training up.
  • Training checkpoints: Files of weights and optimizer state saved for resuming or deployment, an unrelated use of the word.

Checkpointing is common when fine-tuning large language models (LLMs) on a single GPU, where Hugging Face's Trainer enables it with gradient_checkpointing=True, and when training image segmentation models on large medical or aerial images whose feature maps dominate memory. Ultralytics YOLO26 training stores all activations and fits them through batch size instead: batch=-1 in training mode runs AutoBatch, which profiles memory at several batch sizes and picks the one predicted to fill 60% of GPU memory, and a fraction such as batch=0.70 sets that target to 70%.

Explore solutions

Computer vision in sports

Computer vision in sports

Discover how Ultralytics YOLO powers sports vision with player and ball tracking, performance analytics, coaching insights, and athlete rehabilitation.
Learn more
Computer vision for aerial imagery

Computer vision for aerial imagery

Turn drone and aerial imagery into real-time insights for agriculture, aquaculture, conservation, and geospatial analysis with Ultralytics YOLO.
Learn more
Computer vision in aerospace

Computer vision in aerospace

Apply computer vision in aerospace with Ultralytics YOLO for aircraft inspection, runway FOD detection, airport ground operations, and composite defect detection.
Learn more
Computer vision in security

Computer vision in security

Protect every site with Ultralytics YOLO models. Vision AI powers perimeter monitoring, threat detection, license-plate recognition, and workplace safety.
Learn more
Computer vision in robotics

Computer vision in robotics

Power smarter machines with Ultralytics YOLO models. Vision AI in robotics drives autonomous navigation, perception, object tracking, and real-time control.
Learn more
Computer vision in logistics

Computer vision in logistics

Streamline logistics with Ultralytics YOLO models. Vision AI enables package inspection, sorting, vehicle tracking, and real-time warehouse safety monitoring.
Learn more
Computer vision in retail

Computer vision in retail

Reimagine retail with Ultralytics YOLO models. Vision AI powers inventory tracking, shelf monitoring, queue management, and smarter customer insights.
Learn more
Computer vision in healthcare

Computer vision in healthcare

Build healthcare solutions with Ultralytics YOLO models. Vision AI in healthcare powers faster medical imaging, smarter diagnostics, and patient monitoring.
Learn more
Computer vision in manufacturing

Computer vision in manufacturing

Optimize manufacturing with Ultralytics YOLO models. Vision AI drives quality control, defect detection, PPE compliance, and assembly line automation.
Learn more
Computer vision in automotive

Computer vision in automotive

Apply computer vision in automotive with Ultralytics YOLO models. Vision AI elevates road safety, driver assistance, and vehicle automation for smarter roads.
Learn more
Computer vision in agriculture

Computer vision in agriculture

Bring vision AI to smart agriculture with Ultralytics YOLO models. Power crop monitoring, livestock tracking, and precision farming for higher, smarter yields.
Learn more

Let's build the future of AI together!

Begin your journey with the future of machine learning