Inference Latency

Inference latency is the time a trained machine learning model takes to turn one input, such as a camera frame or a text prompt, into a prediction. It is usually reported in milliseconds (ms) and sets how quickly a system can react: a camera running at 30 frames per second delivers a new frame every 33.3 ms, so a detector that needs longer falls behind the stream and can no longer support real-time inference. Latency is quoted either for the forward pass alone or end to end, from raw input to usable output, and production systems track percentiles such as the median (p50) and the 99th percentile (p99), because a few slow responses can matter as much as the average.

What Adds to Inference Latency#

A single prediction passes through several stages. Preprocessing decodes, resizes, and normalizes the input. The forward pass runs the network, and its cost grows with the model's FLOPs, the input resolution, and the numeric precision, while its speed depends on the processor, whether a CPU, a GPU, or a neural processing unit (NPU). Postprocessing turns raw outputs into results, for a detector by decoding boxes and running non-maximum suppression (NMS). A cloud API adds the network round trip and any time the request waits in a queue, which is why edge AI runs models on the device that captures the data. The first call is also slower than the rest, because frameworks allocate memory, load weights into caches, and select kernels on first use.

Hardware and runtime change the forward pass by an order of magnitude or more: Ultralytics YOLO26 nano averages 38.9 ms per 640-pixel COCO image with ONNX on a CPU and 1.7 ms with TensorRT on an NVIDIA T4 GPU. Ultralytics YOLO records the per-stage times of every prediction in the result's speed attribute, and benchmark mode repeats the measurement for each export format:

from ultralytics import YOLO

# Load the smallest YOLO26 detection model
model = YOLO("yolo26n.pt")

# Predict on one image
results = model("https://ultralytics.com/images/bus.jpg")
result = results[0]

# Per-image time in milliseconds for each stage of the pipeline
for stage, ms in result.speed.items():
    print(f"{stage}: {ms:.1f} ms")

The loop prints the preprocessing, inference, and postprocessing times, and their sum is the model's latency for that image on the current device.

Measuring Latency Accurately#

Latency figures are only comparable when they are measured the same way. Reliable measurements discard a few warmup runs, time many runs, and report the median and a tail percentile rather than one run. On GPUs the timer must wait for the device, for example with torch.cuda.synchronize(), because kernels launch asynchronously and a clock read too early stops before the work finishes. Input size, batch size, precision, and device must all match deployment, since a model's latency on a data center GPU says little about its latency on a phone. FLOPs estimate cost without any hardware, but models with equal FLOPs can run at clearly different speeds on the same device.

Inference Latency vs. Throughput#

Throughput is the number of inputs a system processes per unit of time, such as images per second or tokens per second, and it often moves against latency. A larger batch keeps the hardware busier, so throughput rises, but every input in the batch waits for the whole batch to finish, so the time to answer each request rises as well:

import time

import torch
import torchvision

# A ResNet-18 image classifier in inference mode on the CPU
model = torchvision.models.resnet18().eval()

for batch in [1, 2, 4, 8]:
    x = torch.randn(batch, 3, 224, 224)
    with torch.inference_mode():
        model(x)  # warmup run, not timed
        times = []
        for _ in range(20):
            start = time.perf_counter()
            model(x)
            times.append(time.perf_counter() - start)
    latency = sorted(times)[len(times) // 2]  # median seconds per batch
    print(f"batch {batch}: {latency * 1e3:5.1f} ms per batch, {batch / latency:4.0f} images/s")

On a laptop CPU, moving from batch 1 to batch 8 raised throughput from 168 to 220 images per second, while the time to return a batch grew from 6 ms to 36 ms. Interactive systems such as driver assistance and voice assistants therefore run at batch size 1, and offline jobs such as indexing a video archive batch aggressively. LLM servers use continuous batching, which adds and removes requests from a running batch so throughput stays high without making each user wait for a full batch.

Reducing Inference Latency#

The largest gains usually come from doing less computation: a smaller model variant, a lower input resolution, or fewer weights through model pruning. Model quantization to FP16 or INT8 makes each remaining operation cheaper on hardware with fast low-precision units. Exporting to an optimized inference engine such as TensorRT, OpenVINO, or ONNX Runtime fuses layers and picks the fastest kernels for the device. Running the model on the device itself removes the network round trip, which over a mobile connection often costs tens of milliseconds.

Where Latency Matters#

• Large language models (LLMs): Chat services track the time to first token, which users feel as responsiveness, and the time per output token, which sets the streaming speed. Speculative decoding cuts the second by verifying several drafted tokens in one forward pass.
• Industrial inspection: A camera above a conveyor must classify each part before it reaches the reject gate, so latency sets the maximum line speed.
• Autonomous vehicles: Perception, planning, and control share a tight budget, and at 100 km/h a car travels 2.8 meters during every 100 ms of delay.