Real-time Inference

Real-time inference is running a trained model on live input and returning each prediction before the next input arrives, so a system can act while the information is still current. For a camera streaming at 30 frames per second (FPS), that leaves 33.3 milliseconds per frame for everything from reading the image to producing the final result, and at 60 FPS the budget shrinks to 16.7 ms. Each application sets its own deadline: a car's perception stack, a conveyor-belt inspection camera, and a video-call background filter all run in real time with different budgets.

The Latency Budget#

The time that counts is end-to-end inference latency: capturing the frame, preprocessing it by resizing and normalizing, the model's forward pass, and postprocessing such as decoding boxes and non-maximum suppression (NMS). When the total exceeds the frame interval, unprocessed frames pile up and the output drifts further behind the live scene, so live pipelines usually drop stale frames and always work on the newest one. Real-time computing separates hard real-time systems, where a missed deadline is a failure, such as an airbag controller, from soft real-time systems that tolerate an occasional miss. Most computer vision applications are soft real-time: a dropped frame lowers output quality for an instant and the next frame recovers it.

This snippet streams a recorded video through a small detector and compares its mean time per frame with the 30 FPS budget:

from ultralytics import YOLO

model = YOLO("yolo26n.pt")
video = "https://github.com/ultralytics/assets/releases/download/v0.0.0/decelera_portrait_min.mov"

# stream=True yields each frame's result as soon as it is ready
frame_times = []
for result in model(video, stream=True):
    # Preprocess, inference, and postprocess time for this frame, in milliseconds
    frame_times.append(sum(result.speed.values()))

mean_ms = sum(frame_times) / len(frame_times)
print(f"{mean_ms:.1f} ms per frame ({1000 / mean_ms:.0f} FPS); a 30 FPS camera allows 33.3 ms")

A mean below 33.3 ms means the model keeps up with a 30 FPS camera on the hardware that ran it.

Real-Time vs. Batch Inference#

• Real-time (online) inference: Processes each input as it arrives, usually one image at a time, and is judged by the latency of every single prediction. Driver assistance, robot control, and live video analytics depend on it.
• Batch (offline) inference: Collects many inputs and processes them together, judged by throughput in inputs per second and by cost. Re-indexing a photo archive overnight or analyzing last month's footage are batch jobs, and larger batches keep a GPU busier while each input waits longer for its result.

Servers that handle many camera streams often sit between the two, grouping frames from different cameras into small batches to raise throughput while staying inside each stream's frame budget.

Real-World Applications#

• Autonomous vehicles: At 100 km/h a car covers 2.8 meters every 100 ms, so perception latency adds directly to stopping distance.
• Manufacturing inspection: Cameras over a conveyor detect defects on every part and must trigger the reject mechanism before the defective part moves past it.
• Robotics: Obstacle detection and pose estimation run inside a robot's control loop, and slower perception forces slower motion.
• Video effects and augmented reality: Background blur in video calls and AR overlays run segmentation or tracking on every frame, and any delay shows up as an overlay that trails the moving person.

Making a Model Fast Enough#

• Smaller model or input size: A smaller model variant runs fewer operations per image, and because convolutional compute grows roughly with pixel count, halving the input width and height cuts it to about a quarter, at some cost to small-object accuracy.
• Model quantization: Storing weights and activations in FP16 or INT8 instead of FP32 reduces memory traffic and uses faster arithmetic units. INT8 usually needs a set of calibration images to keep accuracy loss small.
• Inference engines: A runtime built for one family of hardware fuses layers and picks kernels tuned for the device, such as TensorRT for NVIDIA GPUs or Core ML for Apple devices.
• Edge AI: Running the model on the camera or an embedded module removes the network round trip of a cloud call, which on a slow connection can exceed the whole frame budget.

Runtime and hardware change latency by more than an order of magnitude: the nano YOLO26 model takes 38.9 ms per 640-pixel image on a CPU with ONNX and 1.7 ms on an NVIDIA T4 GPU with TensorRT. Ultralytics export mode converts a trained YOLO26 model to TensorRT, OpenVINO, LiteRT, and other formats, and its quantize=16 or quantize=8 argument produces FP16 or INT8 weights for the target device.