License YOLO
Back to Ultralytics Glossary

State Space Models (SSM)

State space models (SSMs) carry a fixed-size hidden state through a sequence, giving linear-time sequence modeling. Covers S4, Mamba, and hybrids.

A state space model (SSM) maps an input sequence to an output sequence through a hidden state vector that it updates at every step, so its memory stays the same size however long the sequence grows. The idea comes from control theory, where a linear system is written as a state update h' = Ah + Bx and an output y = Ch + Dx, and where the Kalman filter has estimated such states from noisy measurements since 1960. Deep learning SSMs learn A, B, and C from data and stack many of these layers, and since S4 in 2021 and Mamba in 2023 they have competed with transformers on long sequences of text, audio, and time-series data while their compute grows linearly with sequence length.

How SSMs Work#

A neural SSM layer turns the continuous equations into a discrete recurrence with a learned step size Δ, using the zero-order hold rule Ā = exp(ΔA). Each step then computes h_t = Ā h_(t-1) + B̄ x_t and y_t = C h_t. When A, B, and C stay fixed across time steps, unrolling the recurrence shows that the output is a convolution of the input with the kernel (CB̄, CĀB̄, C²B̄, ...). The same layer can therefore run as a convolution during training, processing the whole sequence in parallel on a GPU, and as a recurrence during generation, with one constant-size state update per token. In PyTorch, the two views give the same outputs:

import torch

torch.manual_seed(0)
L, N = 64, 16  # sequence length, state size
x = torch.randn(L)

# Diagonal continuous-time SSM, discretized with step size delta (zero-order hold)
A = -(torch.rand(N) + 0.5)
B, C, delta = torch.randn(N), torch.randn(N), 0.1
A_bar = torch.exp(delta * A)
B_bar = (A_bar - 1) / A * B

# Recurrent view: one state update per step, as in generation
h, y_rec = torch.zeros(N), []
for t in range(L):
    h = A_bar * h + B_bar * x[t]
    y_rec.append(C @ h)
y_rec = torch.stack(y_rec)

# Convolutional view: precompute the kernel K_k = C . (A_bar^k * B_bar), as in training
K = torch.stack([C @ (A_bar**k * B_bar) for k in range(L)])
y_conv = torch.stack([K[: t + 1].flip(0) @ x[: t + 1] for t in range(L)])

print("Same outputs:", torch.allclose(y_rec, y_conv, atol=1e-5))

It prints Same outputs: True. The recurrence keeps only 16 numbers of state at every step, while the precomputed kernel lets training compute all 64 outputs in parallel.

From S4 to Mamba#

The first deep SSMs were too slow and memory-hungry to train at scale. S4, by Albert Gu, Karan Goel, and Christopher Ré, gave the state matrix a structure that makes the convolution kernel cheap to compute and an initialization designed to remember long histories. It set the state of the art on every task of the Long Range Arena benchmark, became the first model to solve its Path-X task of 16K-step sequences, and generated 60 times faster than a transformer.

A fixed kernel treats every input the same way, so it cannot decide to skip filler and keep a key fact. Mamba, by Albert Gu and Tri Dao, makes Δ, B, and C functions of the current token, which lets the model choose what to write into its state and what to forget. These selective parameters break the convolution view, so Mamba computes the recurrence with a hardware-aware parallel scan. Mamba reached five times the inference throughput of transformers, and Mamba-3B outperformed transformers of the same size and matched transformers twice its size. Each Mamba block wraps the selective SSM with a short convolution and a SiLU gate:

Mamba-2 showed that selective SSMs and a form of linear attention are dual views of the same computation, a framework its authors call state space duality, and used it to make the core layer 2 to 8 times faster.

  • Transformers: Self-attention compares every token with every other, so it can copy any earlier token exactly, at a cost that grows quadratically and with a KV cache that grows with the context window. An SSM compresses history into a fixed state, which is cheaper and lossy, and the 2024 paper Repeat After Me showed transformers copying and retrieving from long inputs better than fixed-state SSMs.
  • Recurrent neural networks (RNNs): An RNN or LSTM applies a nonlinear update, so training has to step through the sequence one token at a time. SSM updates are linear in the state, which allows a convolution or parallel scan.
  • Hidden Markov models (HMMs): An HMM has a finite set of discrete states with transition probabilities. The linear-Gaussian state space model behind the Kalman filter is its continuous counterpart, and deep SSMs keep that continuous state vector.

Where SSMs Are Used#

Most production use is in hybrid large language models (LLMs) that keep a few attention layers for exact recall. AI21's Jamba interleaves transformer and Mamba layers with mixture of experts (MoE) layers, fits on a single 80 GB GPU, and reports strong results at contexts up to 256K tokens, and IBM's hybrid Granite 4.0-H models combine Mamba-2 and transformer layers in a 9:1 ratio. Vision Mamba applies selective scans to image patches, and SSMs also model raw audio and DNA sequences that are too long for full attention. The classical form runs in Ultralytics YOLO26 tracking: the default TrackTrack tracker, like ByteTrack and BoT-SORT, predicts each object's next box with a Kalman filter whose 8-dimensional state holds the box center, size, and their velocities, updated with a constant-velocity transition matrix and a linear observation matrix that reads the four box coordinates.

Explore solutions

Computer vision in sports

Computer vision in sports

Discover how Ultralytics YOLO powers sports vision with player and ball tracking, performance analytics, coaching insights, and athlete rehabilitation.
Learn more
Computer vision for aerial imagery

Computer vision for aerial imagery

Turn drone and aerial imagery into real-time insights for agriculture, aquaculture, conservation, and geospatial analysis with Ultralytics YOLO.
Learn more
Computer vision in aerospace

Computer vision in aerospace

Apply computer vision in aerospace with Ultralytics YOLO for aircraft inspection, runway FOD detection, airport ground operations, and composite defect detection.
Learn more
Computer vision in security

Computer vision in security

Protect every site with Ultralytics YOLO models. Vision AI powers perimeter monitoring, threat detection, license-plate recognition, and workplace safety.
Learn more
Computer vision in robotics

Computer vision in robotics

Power smarter machines with Ultralytics YOLO models. Vision AI in robotics drives autonomous navigation, perception, object tracking, and real-time control.
Learn more
Computer vision in logistics

Computer vision in logistics

Streamline logistics with Ultralytics YOLO models. Vision AI enables package inspection, sorting, vehicle tracking, and real-time warehouse safety monitoring.
Learn more
Computer vision in retail

Computer vision in retail

Reimagine retail with Ultralytics YOLO models. Vision AI powers inventory tracking, shelf monitoring, queue management, and smarter customer insights.
Learn more
Computer vision in healthcare

Computer vision in healthcare

Build healthcare solutions with Ultralytics YOLO models. Vision AI in healthcare powers faster medical imaging, smarter diagnostics, and patient monitoring.
Learn more
Computer vision in manufacturing

Computer vision in manufacturing

Optimize manufacturing with Ultralytics YOLO models. Vision AI drives quality control, defect detection, PPE compliance, and assembly line automation.
Learn more
Computer vision in automotive

Computer vision in automotive

Apply computer vision in automotive with Ultralytics YOLO models. Vision AI elevates road safety, driver assistance, and vehicle automation for smarter roads.
Learn more
Computer vision in agriculture

Computer vision in agriculture

Bring vision AI to smart agriculture with Ultralytics YOLO models. Power crop monitoring, livestock tracking, and precision farming for higher, smarter yields.
Learn more

Let's build the future of AI together!

Begin your journey with the future of machine learning