Scaling Laws

Neural scaling laws are empirical power-law relationships that predict how a model's loss falls as three resources grow: its number of parameters (N), the amount of training data in tokens (D), and the training compute in floating-point operations (C). The curves stay smooth over many orders of magnitude, so labs fit them on small, cheap runs and extrapolate to pick the size and data budget of a large run before paying for it. OpenAI's GPT-4 report, for example, predicted aspects of the final model's performance from models trained with no more than 1/1,000th of its compute. The best-known laws describe language modeling, and the same method applies to vision, speech, and reasoning models.

Kaplan Scaling Laws#

OpenAI researchers led by Jared Kaplan published Scaling Laws for Neural Language Models in 2020. They found that the cross-entropy loss of a transformer language model follows a power law in N, D, and C, with some trends spanning more than seven orders of magnitude, while width, depth, and other architectural details have minimal effects within a wide range. Larger models were also more sample-efficient, so the paper concluded that compute-efficient training uses very large models on relatively modest data and stops well before convergence. GPT-3 followed that recipe with 175 billion parameters trained on 300 billion tokens.

Chinchilla and Compute-Optimal Training#

DeepMind's Chinchilla study trained over 400 models, from 70 million to more than 16 billion parameters on 5 to 500 billion tokens, and reached a different answer: for a fixed compute budget, model size and training tokens should grow in equal proportion, so every doubling of parameters calls for doubling the data. Chinchilla itself used the same compute as the 280-billion-parameter Gopher but had 70 billion parameters and about four times more data, and it outperformed Gopher, GPT-3, and the 530-billion-parameter Megatron-Turing NLG. The study fit the loss as L(N, D) = E + A / N^α + B / D^β, where E is the irreducible loss of natural text and the two other terms shrink with model size and data. Training costs about 6 FLOPs per parameter per token, so C ≈ 6ND, and minimizing the fitted loss for a given C gives the compute-optimal model:

import numpy as np

# Chinchilla loss fit L(N, D) = E + A / N^alpha + B / D^beta, as re-estimated by Besiroglu et al. (2024)
E, A, B, alpha, beta = 1.8172, 482.01, 2085.43, 0.3478, 0.3658

# Training costs about 6 FLOPs per parameter per token, so a budget C fixes D = C / (6N)
N = np.logspace(8, 13, 10_000)
for C in [1e21, 1e23, 5.76e23, 1e25]:
    D = C / (6 * N)
    loss = E + A / N**alpha + B / D**beta
    best = loss.argmin()
    print(
        f"C={C:.2e} FLOPs: {N[best] / 1e9:6.1f}B params, {D[best] / 1e12:5.2f}T tokens, "
        f"{D[best] / N[best]:.0f} tokens/param, loss {loss[best]:.3f}"
    )

The coefficients are Epoch AI's replication of the Chinchilla fit, which corrected the original paper's third estimate. At Chinchilla's budget of 5.76 × 10^23 FLOPs the fit picks 72.2 billion parameters and 1.33 trillion tokens, close to Chinchilla's actual 70 billion and 1.4 trillion, and every budget from 10^21 to 10^25 FLOPs lands at 17 to 22 tokens per parameter, in line with the common rule of about 20.

Beyond Compute-Optimal#

Chinchilla's rule minimizes training compute only, while a deployed model's cost is paid on every query. Smaller models trained far past the compute-optimal point are cheaper to serve: Meta noted that the Chinchilla-optimal data for an 8-billion-parameter model is about 200 billion tokens, yet Llama 3 8B kept improving log-linearly up to 15 trillion tokens, which is why small language models (SLMs) are routinely overtrained. Data supply is the other limit. A 2023 study of data-constrained scaling found that training on up to four epochs of repeated data changed the loss negligibly compared with unique data, and synthetic data extends the supply of training data further.

Test-Time Compute Scaling#

A second axis spends compute at inference. Reasoning models such as OpenAI's o1 generate long chains of thought before answering, and OpenAI reported in 2024 that o1's accuracy rose both with more reinforcement-learning compute and with more time spent thinking. A study from Google DeepMind and UC Berkeley on scaling test-time compute found that compute-optimal strategies, such as searching with a process reward model (PRM), were more than four times as efficient as best-of-N sampling, and that on problems where a small model has non-trivial success, extra test-time compute let it outperform a model 14 times larger at equal FLOPs.

Scaling in Computer Vision#

Vision models follow the same pattern. Google's Scaling Vision Transformers study trained a 2-billion-parameter vision transformer (ViT) that reached 90.45% top-1 accuracy on ImageNet. Ultralytics YOLO26 builds its five sizes from one model YAML whose scales entries set depth and width multipliers and a channel cap, from n: [0.50, 0.25, 1024] to x: [1.00, 1.50, 512]. On COCO, mean average precision (mAP) rises from 40.9 for YOLO26n at 2.4 million parameters to 57.5 for YOLO26x at 55.7 million, a 23-fold increase in parameters for 16.6 more points of mAP50-95.