License YOLO
Back to Ultralytics Glossary

Vision Mamba

Vision Mamba applies Mamba's selective state space scan to image patches, mixing global context in linear time. Covers Vim, VMamba, trade-offs, and code.

Vision Mamba is a family of computer vision backbones that replace the self-attention layers of a Vision Transformer (ViT) with Mamba, a selective state space model (SSM) originally built for language and other long sequences. The name refers both to this class of models and to Vim, the model of that name introduced by Lianghui Zhu and colleagues in January 2024 and published at ICML 2024. Like a ViT, a Vision Mamba model cuts an image into patches and treats them as a sequence of tokens. It mixes information across that sequence with a recurrent scan whose cost grows linearly with the number of tokens, where self-attention compares every token with every other token at a cost that grows quadratically. The difference matters most for high-resolution images, long videos, and 3D medical scans, where token counts run into the thousands.

How the Selective Scan Works#

Mamba, introduced by Albert Gu and Tri Dao in December 2023, processes a sequence with a linear recurrence. For each channel, a hidden state h is updated as h ← exp(Δ·A)·h + Δ·B·x and read out as y = C·h. In earlier SSMs such as S4, Δ, B, and C were learned constants applied identically to every token. Mamba computes them from each token, which lets the model decide token by token how much of its state to keep and how much of the new input to let in, the "selective" part of a selective SSM. The state has a fixed size however long the sequence is, and Mamba's hardware-aware GPU kernels compute the recurrence with a parallel scan, so training stays fast. The PyTorch snippet below implements this recurrence, scans a sequence of image patch tokens forward and backward as Vim does, and runs it at two resolutions:

import torch
import torch.nn.functional as F
from torch import nn

torch.manual_seed(0)
D, N = 192, 16  # channels per patch token, state size per channel
patchify = nn.Conv2d(3, D, kernel_size=16, stride=16)  # each 16x16 patch becomes one token
A = -torch.arange(1, N + 1.0).repeat(D, 1)  # negative decay rates, Mamba's initial values for its learned A
proj = nn.Linear(D, D + 2 * N)  # computes a step size, B, and C from each token


def selective_scan(x):
    """Mamba's S6 recurrence h = exp(delta*A)*h + delta*B*x, y = C.h, with delta, B, C taken from each token."""
    delta, B, C = proj(x).split([D, N, N], dim=-1)
    delta, h, ys = F.softplus(delta), torch.zeros(x.shape[0], D, N), []
    for t in range(x.shape[1]):
        h = torch.exp(delta[:, t, :, None] * A) * h + delta[:, t, :, None] * B[:, t, None, :] * x[:, t, :, None]
        ys.append((h * C[:, t, None, :]).sum(-1))
    return torch.stack(ys, dim=1)


for size in [224, 448]:
    tokens = patchify(torch.randn(1, 3, size, size)).flatten(2).transpose(1, 2)
    out = selective_scan(tokens) + selective_scan(tokens.flip(1)).flip(1)  # forward plus backward scan
    print(f"{size}x{size} image: {tokens.shape[1]} tokens, output {tuple(out.shape)}")

It prints 196 tokens for the 224×224 image and 784 for the 448×448 image, each mapped to a 192-dimensional output. The scan's state holds 192 × 16 = 3,072 values at both resolutions, while a self-attention map over the same tokens grows from 38,416 to 614,656 entries, 16 times larger for 4 times as many tokens.

Adapting Mamba to Images#

Images are 2D and have no natural reading order, and a one-directional recurrence lets each patch see only the patches scanned before it. Vision Mamba models solve this with multiple scan paths:

  • Vim: Adds position embeddings to the patch tokens, places a class token in the middle of the sequence, and runs each block's SSM over the sequence both forward and backward, so every patch gathers context from both sides. The paper reports that Vim runs 2.8 times faster than the DeiT vision transformer and uses 86.8% less GPU memory when extracting features from 1248×1248 images in batch inference.
  • VMamba: VMamba uses a 2D selective scan that reads the patch grid along four paths, row by row and column by column in both directions, and merges the results inside a hierarchical backbone that outputs multi-scale feature maps for detection and segmentation.
  • Hybrids: NVIDIA's MambaVision, published at CVPR 2025, adds self-attention blocks to the final layers of a hierarchical Mamba backbone, where downsampling has made the token count small enough for exact global attention.

A Vim encoder block splits the normalized tokens into a branch that is scanned in both directions and a gate branch that scales both scan outputs before a residual connection:

Vision Mamba vs. Vision Transformers and CNNs#

  • Vision Transformer: Self-attention compares all N patch tokens pairwise at O(N²) cost and lets any patch attend directly to any other. A Vision Mamba scan costs O(N) and passes information from distant patches through its fixed-size state. Real-time detectors also manage attention's quadratic cost by placing it where tokens are few: YOLO26 confines its self-attention blocks to the stride-32 feature map, where a 640×640 input leaves only 20 × 20 = 400 positions.
  • Convolutional neural network (CNN): Convolutions mix a local neighborhood in each layer and build a large receptive field through depth and downsampling, while one Mamba scan spans the whole image.
  • Is the scan needed? In MambaOut, Weihao Yu and Xinchao Wang removed the SSM from Mamba-style blocks, keeping only gated convolutions, and the resulting models surpassed all the visual Mamba models they compared on ImageNet classification. MambaOut fell short of the best visual Mamba models on detection and segmentation, where higher input resolutions produce much longer token sequences, which points to long sequences as the setting where the scan pays off.

Applications#

  • Medical image analysis: CT and MRI volumes contain far more voxels than a photo has pixels, and models such as U-Mamba add Mamba blocks to a U-Net-style encoder-decoder to segment organs and tumors with global context at manageable memory cost.
  • Video understanding: VideoMamba extends the scan to spatiotemporal tokens, so a model can read long clips whose token counts would make full attention expensive.
  • Dense prediction: Object detection and semantic segmentation run at high input resolutions, which is where Vim's memory savings grow largest: at 1248×1248, a 16×16 patching yields 6,084 tokens, and a single attention map over them holds about 37 million entries.

Explore solutions

Computer vision in sports

Computer vision in sports

Discover how Ultralytics YOLO powers sports vision with player and ball tracking, performance analytics, coaching insights, and athlete rehabilitation.
Learn more
Computer vision for aerial imagery

Computer vision for aerial imagery

Turn drone and aerial imagery into real-time insights for agriculture, aquaculture, conservation, and geospatial analysis with Ultralytics YOLO.
Learn more
Computer vision in aerospace

Computer vision in aerospace

Apply computer vision in aerospace with Ultralytics YOLO for aircraft inspection, runway FOD detection, airport ground operations, and composite defect detection.
Learn more
Computer vision in security

Computer vision in security

Protect every site with Ultralytics YOLO models. Vision AI powers perimeter monitoring, threat detection, license-plate recognition, and workplace safety.
Learn more
Computer vision in robotics

Computer vision in robotics

Power smarter machines with Ultralytics YOLO models. Vision AI in robotics drives autonomous navigation, perception, object tracking, and real-time control.
Learn more
Computer vision in logistics

Computer vision in logistics

Streamline logistics with Ultralytics YOLO models. Vision AI enables package inspection, sorting, vehicle tracking, and real-time warehouse safety monitoring.
Learn more
Computer vision in retail

Computer vision in retail

Reimagine retail with Ultralytics YOLO models. Vision AI powers inventory tracking, shelf monitoring, queue management, and smarter customer insights.
Learn more
Computer vision in healthcare

Computer vision in healthcare

Build healthcare solutions with Ultralytics YOLO models. Vision AI in healthcare powers faster medical imaging, smarter diagnostics, and patient monitoring.
Learn more
Computer vision in manufacturing

Computer vision in manufacturing

Optimize manufacturing with Ultralytics YOLO models. Vision AI drives quality control, defect detection, PPE compliance, and assembly line automation.
Learn more
Computer vision in automotive

Computer vision in automotive

Apply computer vision in automotive with Ultralytics YOLO models. Vision AI elevates road safety, driver assistance, and vehicle automation for smarter roads.
Learn more
Computer vision in agriculture

Computer vision in agriculture

Bring vision AI to smart agriculture with Ultralytics YOLO models. Power crop monitoring, livestock tracking, and precision farming for higher, smarter yields.
Learn more

Let's build the future of AI together!

Begin your journey with the future of machine learning