Vision Mamba
Vision Mamba applies Mamba's selective state space scan to image patches, mixing global context in linear time. Covers Vim, VMamba, trade-offs, and code.
Vision Mamba is a family of computer vision backbones that replace the self-attention layers of a Vision Transformer (ViT) with Mamba, a selective state space model (SSM) originally built for language and other long sequences. The name refers both to this class of models and to Vim, the model of that name introduced by Lianghui Zhu and colleagues in January 2024 and published at ICML 2024. Like a ViT, a Vision Mamba model cuts an image into patches and treats them as a sequence of tokens. It mixes information across that sequence with a recurrent scan whose cost grows linearly with the number of tokens, where self-attention compares every token with every other token at a cost that grows quadratically. The difference matters most for high-resolution images, long videos, and 3D medical scans, where token counts run into the thousands.
How the Selective Scan Works#
Mamba, introduced by Albert Gu and Tri Dao in December 2023, processes a sequence with a linear recurrence. For each channel, a hidden state h is updated as h ← exp(Δ·A)·h + Δ·B·x and read out as y = C·h. In earlier SSMs such as S4, Δ, B, and C were learned constants applied identically to every token. Mamba computes them from each token, which lets the model decide token by token how much of its state to keep and how much of the new input to let in, the "selective" part of a selective SSM. The state has a fixed size however long the sequence is, and Mamba's hardware-aware GPU kernels compute the recurrence with a parallel scan, so training stays fast. The PyTorch snippet below implements this recurrence, scans a sequence of image patch tokens forward and backward as Vim does, and runs it at two resolutions:
import torch
import torch.nn.functional as F
from torch import nn
torch.manual_seed(0)
D, N = 192, 16 # channels per patch token, state size per channel
patchify = nn.Conv2d(3, D, kernel_size=16, stride=16) # each 16x16 patch becomes one token
A = -torch.arange(1, N + 1.0).repeat(D, 1) # negative decay rates, Mamba's initial values for its learned A
proj = nn.Linear(D, D + 2 * N) # computes a step size, B, and C from each token
def selective_scan(x):
"""Mamba's S6 recurrence h = exp(delta*A)*h + delta*B*x, y = C.h, with delta, B, C taken from each token."""
delta, B, C = proj(x).split([D, N, N], dim=-1)
delta, h, ys = F.softplus(delta), torch.zeros(x.shape[0], D, N), []
for t in range(x.shape[1]):
h = torch.exp(delta[:, t, :, None] * A) * h + delta[:, t, :, None] * B[:, t, None, :] * x[:, t, :, None]
ys.append((h * C[:, t, None, :]).sum(-1))
return torch.stack(ys, dim=1)
for size in [224, 448]:
tokens = patchify(torch.randn(1, 3, size, size)).flatten(2).transpose(1, 2)
out = selective_scan(tokens) + selective_scan(tokens.flip(1)).flip(1) # forward plus backward scan
print(f"{size}x{size} image: {tokens.shape[1]} tokens, output {tuple(out.shape)}")It prints 196 tokens for the 224×224 image and 784 for the 448×448 image, each mapped to a 192-dimensional output. The scan's state holds 192 × 16 = 3,072 values at both resolutions, while a self-attention map over the same tokens grows from 38,416 to 614,656 entries, 16 times larger for 4 times as many tokens.
Adapting Mamba to Images#
Images are 2D and have no natural reading order, and a one-directional recurrence lets each patch see only the patches scanned before it. Vision Mamba models solve this with multiple scan paths:
- Vim: Adds position embeddings to the patch tokens, places a class token in the middle of the sequence, and runs each block's SSM over the sequence both forward and backward, so every patch gathers context from both sides. The paper reports that Vim runs 2.8 times faster than the DeiT vision transformer and uses 86.8% less GPU memory when extracting features from 1248×1248 images in batch inference.
- VMamba: VMamba uses a 2D selective scan that reads the patch grid along four paths, row by row and column by column in both directions, and merges the results inside a hierarchical backbone that outputs multi-scale feature maps for detection and segmentation.
- Hybrids: NVIDIA's MambaVision, published at CVPR 2025, adds self-attention blocks to the final layers of a hierarchical Mamba backbone, where downsampling has made the token count small enough for exact global attention.
A Vim encoder block splits the normalized tokens into a branch that is scanned in both directions and a gate branch that scales both scan outputs before a residual connection:
Vision Mamba vs. Vision Transformers and CNNs#
- Vision Transformer: Self-attention compares all N patch tokens pairwise at O(N²) cost and lets any patch attend directly to any other. A Vision Mamba scan costs O(N) and passes information from distant patches through its fixed-size state. Real-time detectors also manage attention's quadratic cost by placing it where tokens are few: YOLO26 confines its self-attention blocks to the stride-32 feature map, where a 640×640 input leaves only 20 × 20 = 400 positions.
- Convolutional neural network (CNN): Convolutions mix a local neighborhood in each layer and build a large receptive field through depth and downsampling, while one Mamba scan spans the whole image.
- Is the scan needed? In MambaOut, Weihao Yu and Xinchao Wang removed the SSM from Mamba-style blocks, keeping only gated convolutions, and the resulting models surpassed all the visual Mamba models they compared on ImageNet classification. MambaOut fell short of the best visual Mamba models on detection and segmentation, where higher input resolutions produce much longer token sequences, which points to long sequences as the setting where the scan pays off.
Applications#
- Medical image analysis: CT and MRI volumes contain far more voxels than a photo has pixels, and models such as U-Mamba add Mamba blocks to a U-Net-style encoder-decoder to segment organs and tumors with global context at manageable memory cost.
- Video understanding: VideoMamba extends the scan to spatiotemporal tokens, so a model can read long clips whose token counts would make full attention expensive.
- Dense prediction: Object detection and semantic segmentation run at high input resolutions, which is where Vim's memory savings grow largest: at 1248×1248, a 16×16 patching yields 6,084 tokens, and a single attention map over them holds about 37 million entries.










