Neural Operators

Neural operators are neural networks that learn mappings between functions instead of between fixed-size vectors. The input is a function, such as an initial velocity field, a rock formation's permeability map, or today's atmosphere, and the output is another function, such as the flow some seconds later, the resulting pressure field, or tomorrow's weather. Because the learned operator is defined on a continuous domain, the same weights accept the input sampled on a coarse grid, a fine grid, or an irregular mesh. Once trained on solutions from a numerical solver, a neural operator replaces repeated simulations of a whole family of partial differential equations (PDEs) with one fast forward pass.

Operators Versus Ordinary Surrogates#

A convolutional neural network (CNN) trained as a surrogate on 64 x 64 simulations learns kernels tied to that pixel spacing. A neural operator layer instead approximates an integral operator, updating each point from a learned kernel integrated over the whole domain plus a pointwise linear term, and the grid only decides how finely that integral is evaluated. Predictions converge as the grid is refined, a property called discretization invariance, so a model trained on cheap coarse simulations can be queried at higher resolution, which the Fourier Neural Operator paper called zero-shot super-resolution.

The same weights running at a new resolution is familiar from vision models. Ultralytics YOLO26 accepts any imgsz in predict mode, rounded up to a multiple of its 32-pixel maximum stride, because convolution kernels do not depend on image size, yet each 3 x 3 kernel spans a fixed number of pixels, so doubling imgsz halves the width of the scene every kernel covers.

Two related methods are often confused with neural operators. A physics-informed neural network (PINN) trains one network to satisfy a single PDE instance with fixed coefficients and initial conditions, and needs retraining when they change. A Neural ODE learns the time derivative of a fixed-length state vector, and a solver integrates it.

Fourier Neural Operators#

Operator learning took off with DeepONet, posted by Lu Lu, George Karniadakis, and colleagues in 2019, and with the neural operator framework of Zongyi Li, Nikola Kovachki, Anima Anandkumar, Andrew Stuart, and colleagues at Caltech. Their Fourier Neural Operator (FNO), published at ICLR 2021, computes the kernel integral as a convolution in frequency space: a fast Fourier transform (FFT) converts each channel to Fourier modes, learned complex matrices mix the channels on the lowest modes and drop the rest, and an inverse FFT returns to the grid. Every output point depends on the whole input, a global receptive field in one layer.

The spectral path is a few lines of PyTorch. Here, one set of weights processes the same four input functions sampled at 64 and at 256 points:

import torch

torch.manual_seed(0)
channels, modes = 4, 8
# Learned complex weights act on the 8 lowest Fourier modes, whatever the grid size
weights = torch.randn(channels, channels, modes, dtype=torch.cfloat) / channels

for points in (64, 256):
    x = torch.arange(points) / points  # the same periodic domain [0, 1), sampled coarsely or finely
    field = torch.stack([torch.sin(2 * torch.pi * k * x) + x * (1 - x) for k in range(1, channels + 1)])
    spectrum = torch.fft.rfft(field)  # one row of Fourier coefficients per channel
    mixed = torch.zeros(channels, points // 2 + 1, dtype=torch.cfloat)
    mixed[:, :modes] = torch.einsum("im,iom->om", spectrum[:, :modes], weights)
    output = torch.fft.irfft(mixed, n=points)  # back to the grid the input came from
    print(f"{points} points: output shape {tuple(output.shape)}, u(0.25) = {output[0, points // 4]:.4f}")

It prints an output of shape (4, 64) with u(0.25) = -1.0908 and one of shape (4, 256) with u(0.25) = -1.0909. The 128 complex weights never change, and the output at the same physical point agrees to three decimal places because the layer acts on Fourier modes of the underlying function instead of on grid points. On turbulent Navier-Stokes flow, Burgers' equation, and Darcy flow, the FNO paper reported predictions up to three orders of magnitude faster than traditional PDE solvers.

Other Architectures#

DeepONet, published in Nature Machine Intelligence in 2021, builds on a 1995 universal approximation theorem for operators by Tianping Chen and Hong Chen: a branch network encodes the input function from values at fixed sensor locations, a trunk network encodes the coordinate where the output is wanted, and their dot product gives the output value there. Graph neural operators borrow message passing from graph neural networks (GNNs) to handle irregular meshes around complex geometries. Operator transformers treat sampled points or patches of the domain as tokens. Pretrained on fluid dynamics, Poseidon, from ETH Zurich in 2024, serves as a foundation model for PDEs and was fine-tuned on 15 downstream tasks, including physics absent from its pretraining data. The open source neuraloperator library implements FNO and many of these variants.

Applications#

Training data comes from conventional solvers, so every training pair is synthetic data whose cost is paid once, and the trained operator then answers many queries quickly: design sweeps over airfoil or reactor shapes, uncertainty quantification studies that need thousands of runs, and digital twins of machines and plants that must update faster than a full simulation. Physics-informed neural operators add the PDE residual to the loss function so predictions stay consistent with the governing equations where training data is sparse.

In weather forecasting, FourCastNet, from NVIDIA and academic collaborators in 2022, is built on an adaptive Fourier neural operator and produces global forecasts at 0.25° resolution, generating a week-long forecast in under 2 seconds, orders of magnitude faster than the ECMWF Integrated Forecasting System it matches on short-range accuracy for large-scale variables.