Neural Ordinary Differential Equations (Neural ODEs)

Neural ordinary differential equations (Neural ODEs) are neural networks that define how a hidden state changes continuously over time, in place of a fixed stack of layers. A small network f learns the derivative dh/dt = f(h, t), and a numerical ODE solver integrates that derivative from an initial state to whatever time the output is needed. Ricky Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud introduced the idea in a 2018 paper that received a best paper award at NeurIPS.

From Residual Blocks to Continuous Depth#

A residual network updates its state as h(t + 1) = h(t) + f(h(t)), which is exactly one step of Euler's method for solving an ODE, with a step size of 1. A Neural ODE takes the limit of infinitely many infinitely small steps and hands the integration to a solver. Classical numerical software, such as SciPy's solve_ivp, solves the same kind of initial-value problem for hand-written equations.

This changes what "depth" means. Adaptive solvers from the Runge-Kutta family, such as the Dormand-Prince method, take small steps where the dynamics change quickly and large steps where they are smooth. The cost of a forward pass is the number of function evaluations the solver needs, which depends on the input, the learned dynamics, and the error tolerance. Loosening the tolerance at inference trades accuracy for lower inference latency without retraining.

Training With the Adjoint Method#

Ordinary backpropagation through a solver stores every intermediate state, so memory grows with the number of solver steps. The adjoint sensitivity method avoids this by solving a second ODE backward in time that carries the gradient, recomputing states along the way instead of storing them. Memory cost stays constant with respect to depth, at the price of extra computation and small numerical differences between the forward and backward trajectories.

The torchdiffeq package provides differentiable solvers for PyTorch, with odeint for direct backpropagation and odeint_adjoint for the adjoint method:

import torch
from torch import nn
from torchdiffeq import odeint

torch.manual_seed(0)

# The network predicts the derivative dh/dt of a 2D hidden state
dynamics = nn.Sequential(nn.Linear(2, 32), nn.Tanh(), nn.Linear(32, 2))

# Query the state at irregular timestamps, as with sporadic sensor readings
initial_state = torch.tensor([[1.0, 0.0]])
times = torch.tensor([0.0, 0.3, 0.35, 1.1, 2.0])
trajectory = odeint(lambda t, h: dynamics(h), initial_state, times)

# Gradients flow back through the solver into the network weights
loss = trajectory[-1].square().sum()
loss.backward()
print(trajectory.squeeze(1))
print(dynamics[0].weight.grad.norm())

The trajectory holds the state at each requested time, from the closely spaced 0.3 and 0.35 to the long gap before 1.1, with the solver choosing its own internal steps. The nonzero gradient norm shows that a loss function evaluated at the final time trains the network through the solver. Replacing odeint with odeint_adjoint switches to constant-memory gradients, provided the dynamics function is an nn.Module so the solver can find its parameters.

Where Neural ODEs Fit#

Irregularly sampled time series are the clearest fit. Latent ODE models, introduced by Rubanova and colleagues in 2019, encode a sequence of observations into a hidden state and evolve it continuously, so medical records or sensor logs with uneven gaps can be modeled without forcing them onto a fixed grid. A recurrent neural network updates only at discrete steps and needs extra machinery to handle missing or irregular timestamps, a core problem in time series analysis.

Tracking data has the same shape. Track mode in Ultralytics YOLO26 outputs a position for each track ID in each frame, with gaps whenever an object is occluded or missed, and its default tracker carries each lost track forward for track_buffer frames, 30 by default, with a Kalman filter whose constant-velocity motion model is a fixed linear ODE stepped once per frame. A Neural ODE fitted to the same positions learns the motion itself, as a continuous trajectory that can be queried between frames or across a gap in multi-object tracking.

Generative modeling is the second major use. Continuous normalizing flows, developed in the original Neural ODE paper and extended in FFJORD, transform a simple distribution into a complex one by integrating a learned vector field. Flow matching later made these models far cheaper to train by learning the vector field directly, with no ODE solve during training. Rectified flow image generators such as Stable Diffusion 3 sample by integrating such an ODE, and diffusion models can be sampled deterministically by integrating their probability flow ODE.

Limits and Related Continuous-Time Models#

Trajectories of an ODE cannot cross, so a Neural ODE in the input's own dimension cannot represent some simple mappings. Augmented Neural ODEs, proposed by Emilien Dupont and colleagues in 2019, fix this by adding extra dimensions to the state. Stiff dynamics and growing numbers of function evaluations during training can make Neural ODEs slower than a fixed-depth network.

Other architectures build on continuous-time dynamics with different trade-offs. Liquid neural networks are recurrent cells defined by ODEs whose time constants depend on the input. State space models such as S4 and Mamba start from a linear continuous-time system, x'(t) = Ax(t) + Bu(t), and discretize it into a recurrence that can also be computed as a long convolution or a parallel scan, which makes them fast on long sequences where a general Neural ODE needs an adaptive solver.