State Space Models (SSM)
State space models (SSMs) carry a fixed-size hidden state through a sequence, giving linear-time sequence modeling. Covers S4, Mamba, and hybrids.
A state space model (SSM) maps an input sequence to an output sequence through a hidden state vector that it updates at every step, so its memory stays the same size however long the sequence grows. The idea comes from control theory, where a linear system is written as a state update h' = Ah + Bx and an output y = Ch + Dx, and where the Kalman filter has estimated such states from noisy measurements since 1960. Deep learning SSMs learn A, B, and C from data and stack many of these layers, and since S4 in 2021 and Mamba in 2023 they have competed with transformers on long sequences of text, audio, and time-series data while their compute grows linearly with sequence length.
How SSMs Work#
A neural SSM layer turns the continuous equations into a discrete recurrence with a learned step size Δ, using the zero-order hold rule Ā = exp(ΔA). Each step then computes h_t = Ā h_(t-1) + B̄ x_t and y_t = C h_t. When A, B, and C stay fixed across time steps, unrolling the recurrence shows that the output is a convolution of the input with the kernel (CB̄, CĀB̄, C²B̄, ...). The same layer can therefore run as a convolution during training, processing the whole sequence in parallel on a GPU, and as a recurrence during generation, with one constant-size state update per token. In PyTorch, the two views give the same outputs:
import torch
torch.manual_seed(0)
L, N = 64, 16 # sequence length, state size
x = torch.randn(L)
# Diagonal continuous-time SSM, discretized with step size delta (zero-order hold)
A = -(torch.rand(N) + 0.5)
B, C, delta = torch.randn(N), torch.randn(N), 0.1
A_bar = torch.exp(delta * A)
B_bar = (A_bar - 1) / A * B
# Recurrent view: one state update per step, as in generation
h, y_rec = torch.zeros(N), []
for t in range(L):
h = A_bar * h + B_bar * x[t]
y_rec.append(C @ h)
y_rec = torch.stack(y_rec)
# Convolutional view: precompute the kernel K_k = C . (A_bar^k * B_bar), as in training
K = torch.stack([C @ (A_bar**k * B_bar) for k in range(L)])
y_conv = torch.stack([K[: t + 1].flip(0) @ x[: t + 1] for t in range(L)])
print("Same outputs:", torch.allclose(y_rec, y_conv, atol=1e-5))It prints Same outputs: True. The recurrence keeps only 16 numbers of state at every step, while the precomputed kernel lets training compute all 64 outputs in parallel.
From S4 to Mamba#
The first deep SSMs were too slow and memory-hungry to train at scale. S4, by Albert Gu, Karan Goel, and Christopher Ré, gave the state matrix a structure that makes the convolution kernel cheap to compute and an initialization designed to remember long histories. It set the state of the art on every task of the Long Range Arena benchmark, became the first model to solve its Path-X task of 16K-step sequences, and generated 60 times faster than a transformer.
A fixed kernel treats every input the same way, so it cannot decide to skip filler and keep a key fact. Mamba, by Albert Gu and Tri Dao, makes Δ, B, and C functions of the current token, which lets the model choose what to write into its state and what to forget. These selective parameters break the convolution view, so Mamba computes the recurrence with a hardware-aware parallel scan. Mamba reached five times the inference throughput of transformers, and Mamba-3B outperformed transformers of the same size and matched transformers twice its size. Each Mamba block wraps the selective SSM with a short convolution and a SiLU gate:
Mamba-2 showed that selective SSMs and a form of linear attention are dual views of the same computation, a framework its authors call state space duality, and used it to make the core layer 2 to 8 times faster.
SSMs vs. Related Architectures#
- Transformers: Self-attention compares every token with every other, so it can copy any earlier token exactly, at a cost that grows quadratically and with a KV cache that grows with the context window. An SSM compresses history into a fixed state, which is cheaper and lossy, and the 2024 paper Repeat After Me showed transformers copying and retrieving from long inputs better than fixed-state SSMs.
- Recurrent neural networks (RNNs): An RNN or LSTM applies a nonlinear update, so training has to step through the sequence one token at a time. SSM updates are linear in the state, which allows a convolution or parallel scan.
- Hidden Markov models (HMMs): An HMM has a finite set of discrete states with transition probabilities. The linear-Gaussian state space model behind the Kalman filter is its continuous counterpart, and deep SSMs keep that continuous state vector.
Where SSMs Are Used#
Most production use is in hybrid large language models (LLMs) that keep a few attention layers for exact recall. AI21's Jamba interleaves transformer and Mamba layers with mixture of experts (MoE) layers, fits on a single 80 GB GPU, and reports strong results at contexts up to 256K tokens, and IBM's hybrid Granite 4.0-H models combine Mamba-2 and transformer layers in a 9:1 ratio. Vision Mamba applies selective scans to image patches, and SSMs also model raw audio and DNA sequences that are too long for full attention. The classical form runs in Ultralytics YOLO26 tracking: the default TrackTrack tracker, like ByteTrack and BoT-SORT, predicts each object's next box with a Kalman filter whose 8-dimensional state holds the box center, size, and their velocities, updated with a constant-velocity transition matrix and a linear observation matrix that reads the four box coordinates.










