Gradient Checkpointing
Gradient checkpointing stores only some activations in the forward pass and recomputes the rest during backpropagation, trading compute for memory.
Gradient checkpointing is a training technique that saves memory by keeping only a few intermediate activations from the forward pass and recomputing the rest when backpropagation needs them. It is also called activation checkpointing or rematerialization. Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin popularized it for deep learning in their April 2016 paper Training Deep Nets with Sublinear Memory Cost, showing that an n-layer network can train with memory proportional to √n for the cost of one extra forward pass. On a 1,000-layer residual network for ImageNet, their method cut memory from 48 GB to 7 GB at about 30% more training time.
How Gradient Checkpointing Works#
During a normal forward pass, the autograd engine stores the output of nearly every operation, because computing gradients later needs them. For deep networks, large batches, high-resolution images, or long sequences, these activations take far more GPU memory than the model weights. Checkpointing splits the network into segments and keeps only each segment's input. When the backward pass reaches a segment, it runs that segment's forward pass again from the saved input, backpropagates through it, and frees the rebuilt activations before moving to the previous segment:
PyTorch exposes this as torch.utils.checkpoint.checkpoint, and JAX as jax.checkpoint. Counting the activations autograd stores for an eight-block network shows the saving and that the gradients do not change:
import torch
from torch import nn
from torch.utils.checkpoint import checkpoint
torch.manual_seed(0)
model = nn.Sequential(*[nn.Sequential(nn.Linear(1024, 4096), nn.GELU(), nn.Linear(4096, 1024)) for _ in range(8)])
x = torch.randn(256, 1024)
weights = {p.data_ptr() for p in model.parameters()}
saved, grads = [], []
for use_checkpoint in (False, True):
saved.clear()
# Record every tensor autograd keeps for the backward pass
with torch.autograd.graph.saved_tensors_hooks(lambda t: saved.append(t) or t, lambda t: t):
h = x
for block in model: # a checkpointed block keeps only its input and reruns its forward pass in backward
h = checkpoint(block, h, use_reentrant=False) if use_checkpoint else block(h)
h.square().mean().backward()
activations = sum(t.numel() * t.element_size() for t in saved if t.data_ptr() not in weights)
print(f"checkpoint={use_checkpoint}: {activations / 2**20:.0f} MB of stored activations")
grads.append(model[0][0].weight.grad.clone())
model.zero_grad()
print("Identical gradients:", torch.equal(*grads))It prints 72 MB of stored activations without checkpointing, 8 MB with it, and Identical gradients: True. Each block normally stores 9 MB, its 1 MB input plus two 4 MB hidden tensors; with checkpointing it keeps only the input, and the backward pass rebuilds one block's 9 MB at a time.
Choosing What to Recompute#
Checkpointing every block saves the most memory but reruns nearly the whole forward pass, roughly a third more compute for a typical layer. Selective recomputation targets operations whose outputs are large but cheap to compute. NVIDIA's Vijay Korthikanti and colleagues recomputed only the core of transformer attention, the score computation, softmax, dropout, and weighting of values, and, combined with sequence parallelism, cut activation memory fivefold while removing over 90% of the recomputation overhead. Flash attention applies the same idea inside one kernel by recomputing attention scores during the backward pass instead of storing them. GPipe stores activations only at the boundaries between pipeline parallelism stages and recomputes everything inside each stage.
Recomputation must reproduce the original forward pass exactly. PyTorch restores the random number generator state so dropout masks match, and Hugging Face Transformers models switch off their KV cache when they train with checkpointing enabled, since cached inference state conflicts with rerunning layers.
Related Memory Techniques#
- Gradient accumulation: Splits a large batch into micro-batches that run one at a time, reducing activations per step without extra compute but adding steps.
- Mixed precision: Stores activations in 16-bit formats such as BF16, halving their memory and usually speeding training up.
- Training checkpoints: Files of weights and optimizer state saved for resuming or deployment, an unrelated use of the word.
Checkpointing is common when fine-tuning large language models (LLMs) on a single GPU, where Hugging Face's Trainer enables it with gradient_checkpointing=True, and when training image segmentation models on large medical or aerial images whose feature maps dominate memory. Ultralytics YOLO26 training stores all activations and fits them through batch size instead: batch=-1 in training mode runs AutoBatch, which profiles memory at several batch sizes and picks the one predicted to fill 60% of GPU memory, and a fraction such as batch=0.70 sets that target to 70%.










