Neural Tangent Kernel (NTK)

The neural tangent kernel (NTK) is a similarity function that measures how two inputs' neural network outputs respond to the same small changes in model parameters. Inputs have strongly coupled predictions when their parameter gradients align. This connects neural network training to kernel methods, which learn through pairwise similarities between examples.

Arthur Jacot, Franck Gabriel, and Clément Hongler introduced the term in 2018 in Neural Tangent Kernel: Convergence and Generalization in Neural Networks, establishing a mathematical framework for analyzing how predictions evolve during training.

How the Neural Tangent Kernel Works#

For a scalar-output neural network, compute the output's derivative with respect to every trainable parameter. These derivatives form a gradient vector describing the prediction's sensitivity to the model weights. The NTK between two inputs is the dot product of their gradient vectors.

Across a dataset, these pairwise values form a kernel matrix, also called a Gram matrix. Its diagonal measures each prediction's sensitivity to parameter changes; off-diagonal entries describe coupling between different examples. For multiple outputs, each input pair produces a block containing the interactions between output coordinates, as used in empirical NTK computation.

During gradient descent, the kernel determines how reducing one example's prediction error changes predictions elsewhere. The relevant gradients are derivatives of network outputs; backpropagation supplies the differentiation machinery. The loss function then determines the direction and strength of the error-correcting updates.

Infinite Width, Convergence, and Generalization#

A network's width is its number of hidden units, or channels in a convolutional layer. Under NTK parameterization and suitable initialization and training assumptions, increasing hidden-layer widths toward infinity produces a deterministic limiting kernel that remains constant during training. A finite network's empirical NTK is evaluated at its actual parameters and can change as training progresses.

In the constant-kernel regime, training follows the network's first-order approximation around initialization. This is a linearized network: its predictions depend linearly on parameter changes while retaining nonlinear dependence on inputs. The related term lazy training describes a regime where tangent features change little. Feature learning involves those features evolving substantially during optimization.

For squared-error loss under constant-kernel gradient flow—the continuous-time limit of small gradient steps—errors decrease fastest along kernel eigenvectors with larger eigenvalues. Eigenvectors identify patterns across examples, and eigenvalues determine their learning speeds. This explains why early stopping acts as regularization: it limits fitting of slowly learned components, including components associated with noise and overfitting. Predictions on unseen inputs depend on their kernel similarities to the training data.

Applications and Related Concepts#

NTK methods support kernel regression and the analysis of image classification. The convolutional NTK (CNTK) extends the construction to convolutional neural networks. Sanjeev Arora and colleagues used exact CNTK computation for classification on CIFAR-10, providing a kernel-based counterpart to training finite convolutional networks.

The neural network Gaussian process (NNGP) kernel describes covariance between randomly initialized network outputs in the infinite-width limit. The NTK describes gradient-driven training dynamics. Similarly, linear probing trains a linear classifier on frozen representations; NTK linearization uses derivatives with respect to the selected trainable parameters as its features.

Parameter selection therefore changes the tangent kernel governing training. In Ultralytics YOLO26 fine-tuning, the freeze argument holds selected layers fixed, so their parameters contribute no update directions to the effective training kernel.

Computing an Empirical NTK#

This PyTorch example computes the kernel for two handwritten digit images from scikit-learn's bundled digits dataset. Install the dependencies with pip install torch scikit-learn.

import torch
from sklearn.datasets import load_digits
from torch import nn

torch.manual_seed(0)
digits = load_digits()
inputs = torch.tensor(digits.data[:2], dtype=torch.float32) / 16
model = nn.Sequential(nn.Linear(64, 8), nn.Tanh(), nn.Linear(8, 1))
parameters = tuple(model.parameters())
outputs = model(inputs)

# Each Jacobian row contains one output's parameter derivatives.
rows = []
for output in outputs[:, 0]:
    gradients = torch.autograd.grad(output, parameters, retain_graph=True)
    rows.append(torch.cat([gradient.flatten() for gradient in gradients]))
jacobian = torch.stack(rows)
kernel = jacobian @ jacobian.T
print(kernel)

The printed 2 x 2 matrix contains the two self-sensitivities and their cross-example coupling. Its construction as jacobian @ jacobian.T makes the scalar-output empirical NTK symmetric and positive semidefinite, with rank bounded by both the number of examples and the number of trainable parameters.