Scalability

Scalability is a system's ability to handle a growing amount of work, such as more data, more users, more requests, or a larger model, by adding resources while keeping performance and cost per unit of work acceptable. A system scales well when doubling its hardware nearly doubles the work it can do, and poorly when added machines spend their time waiting on each other. In artificial intelligence, scalability applies at three points: data pipelines that must ingest big data and preprocess it, training jobs that spread over many accelerators, and inference services that must answer more requests without raising inference latency.

Vertical and Horizontal Scaling#

Vertical scaling, or scaling up, moves a workload to a bigger machine: more CPU cores, more RAM, or a GPU with more memory. It needs no code changes, but it stops at the largest machine available and leaves one point of failure. Horizontal scaling, or scaling out, adds more machines and splits the work among them, which has no fixed ceiling but requires the software to divide work, balance load, and coordinate results. Web and inference services scale out easily when they are stateless, keeping sessions and data in shared stores so that any replica can answer any request; databases scale out by sharding, splitting their data by key across machines.

Cloud platforms automate horizontal scaling. On Kubernetes, the Horizontal Pod Autoscaler computes the replica count as the current replicas multiplied by the ratio of the observed metric to its target, rounded up: 4 replicas averaging 90% CPU against a 60% target become 6. That automation is what cloud computing calls elasticity, and serverless computing extends it down to zero replicas when traffic stops.

Limits to Scaling#

Every parallel job contains some work that cannot be split, such as reading the next batch, synchronizing results, or writing a checkpoint. Gene Amdahl showed in 1967 that this serial fraction caps the speedup no matter how many processors are added, a result now called Amdahl's law:

# Amdahl's law: speedup on n workers when a fraction p of the work runs in parallel
p = 0.95  # parallel fraction; the other 5% runs on one worker

for n in (1, 8, 64, 512):
    speedup = 1 / ((1 - p) + p / n)
    efficiency = speedup / n  # share of the added hardware doing useful work
    print(f"{n:>4} workers: {speedup:5.1f}x speedup, {efficiency:4.0%} efficiency")

print(f"Limit with unlimited workers: {1 / (1 - p):.0f}x")

With 95% of the work parallel, 8 workers give a 5.9x speedup at 74% efficiency, 64 workers give 15.4x at 24%, 512 workers give 19.3x at 4%, and no number of workers can exceed 20x. John Gustafson pointed out in 1988 that real users grow the problem along with the machine, so the parallel part grows and the serial share shrinks. Measuring speedup on a fixed problem is called strong scaling, and growing the problem with the hardware is weak scaling, the regime in which large training runs operate. Communication sets a further limit: every added GPU in data-parallel training must exchange gradients with the others at each step, so network bandwidth often decides how far a job can scale.

Scaling Training and Inference#

Distributed training chooses a parallelism strategy based on whether the model fits on one device. In data parallelism, each GPU holds a full copy of the model, processes its own slice of every batch, and averages gradients with the others through an all-reduce, as in PyTorch's DistributedDataParallel. Models too large for one GPU are split with fully sharded data parallelism, tensor parallelism, or pipeline parallelism, and gradient accumulation reaches large effective batch sizes on fewer devices.

Ultralytics YOLO uses data parallelism for multi-GPU training: passing device=[0, 1] launches DistributedDataParallel worker processes through torch.distributed.run and divides the batch evenly, so batch=64 on two GPUs gives each GPU 32 images per step.

Inference scales differently, because each request is small and independent. Grouping requests into batches raises GPU utilization and throughput at a small latency cost, continuous batching does the same for token-by-token LLM generation, and model serving systems add replicas behind a load balancer as traffic grows. Model quantization and pruning make each replica cheaper, so the same hardware serves more requests, and edge AI scales by spreading inference across thousands of devices instead of one data center.

Scalability vs. Related Terms#

• Performance: How fast or accurate a system is at a given load, measured by latency, throughput, or accuracy. Scalability describes how those numbers hold up as load and resources grow.
• Elasticity: Adding and removing resources automatically as demand changes over the day, most often by adding and removing replicas of a horizontally scaled service.
• Scaling laws: Empirical relationships between a model's loss and its parameters, training data, and compute. They predict model quality, and system scalability determines whether the compute they call for can be used efficiently.