Neural Scaling Laws

Neural scaling laws are empirical relationships that describe how neural-network performance changes as model size, training data, and computational effort increase. They commonly follow power laws: multiplying a resource by a fixed factor produces a predictable proportional reduction in error above a limiting floor. This makes scaling laws useful for estimating the benefits of larger training runs and deciding how to allocate a compute budget.

In AI, scaling laws is often a synonym for neural scaling laws. Scalability describes a system's ability to handle growing workloads efficiently.

How Neural Scaling Laws Work#

Three quantities organize most training-time scaling analyses:

• Model size: The number of learned parameters, including model weights, determines the network's capacity to represent patterns.
• Training data: The amount of training data supplies examples from which those patterns are learned. Language-model datasets are commonly measured in tokens, the units of text a model processes.
• Training compute: The total arithmetic performed during training is measured in floating-point operations, usually abbreviated FLOPs.

These resources interact: increasing model capacity changes how much data and training effort can be used productively. A fitted scaling relationship connects their joint allocation to expected loss, the numerical error measured by a loss function.

A power-law curve appears approximately straight when both axes use logarithmic scales. Its exponent describes how quickly error decreases with additional resources. Smaller experiments establish the curve, allowing teams to forecast larger runs. Joel Hestness and colleagues demonstrated predictable power-law error scaling across language, vision, translation, and speech tasks in 2017.

Jared Kaplan, Sam McCandlish, and colleagues established widely used language-model relationships in the 2020 paper Scaling Laws for Neural Language Models. Their experiments connected transformer cross-entropy loss to parameters, data, and compute, with some trends spanning more than seven orders of magnitude.

Compute-Optimal Training and Chinchilla Scaling#

Compute-optimal training selects the combination of model size and training duration that achieves the lowest loss for a fixed training budget. A larger model consumes more computation per token, leaving fewer tokens available within that budget.

Jordan Hoffmann and colleagues introduced the influential Chinchilla allocation rule in 2022: model parameters and training tokens should grow approximately together as compute increases. Their experiments covered more than 400 language models. Chinchilla's compute-optimal allocation produced a 70-billion-parameter model that outperformed the 280-billion-parameter Gopher at the same training compute budget.

The practical consequence is that allocating too much compute to parameters leaves a model undertrained. Balancing capacity and data improves the performance obtained from the same arithmetic budget. Deployment introduces another consideration: smaller models require less computation for inference, so training allocation also affects the cost of serving predictions.

Applications in Computer Vision#

In computer vision, scaling analyses connect model capacity, image datasets, training effort, and predictive quality. They support decisions about expanding a dataset or increasing a network's size.

Google's vision-transformer scaling experiments characterized relationships between error, data, and compute. The resulting two-billion-parameter vision transformer achieved 90.45% top-1 accuracy on ImageNet, demonstrating the practical value of coordinated scaling for image classification.

Ultralytics YOLO applies architectural scaling through YOLO26's model-size variants, giving users different capacity and computation choices for object detection. At 640-pixel input resolution on COCO, YOLO26n has 2.4 million parameters and 40.9 mAP50–95, while YOLO26x has 55.7 million parameters and 57.5 mAP50–95—a gain of 16.6 points.

Why Scaling Rates Differ#

Scaling exponents depend on the task, data structure, and learning method. A random-feature model provides a mathematically solvable example: it maps inputs into fixed nonlinear features and learns their output weights. Alexander Maloney, Daniel Roberts, and James Sully used this construction in 2022 to explain how dataset statistics produce power-law test loss and eventual performance plateaus.

Feature learning changes the internal representations themselves during training. This differs from fixed feature extraction, where representations remain unchanged. Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan showed how feature learning improves scaling on hard tasks: in their solvable model, adapting features approximately doubles the compute-scaling exponent for very hard target functions.