Inference Engine

An inference engine is the software that executes a trained neural network to produce predictions on new data. It takes a model exported from a training framework, usually as a computation graph such as an ONNX file, optimizes that graph for a specific processor, and runs it with as little inference latency and memory as possible. Engines exist for every class of hardware, from data center GPUs to phones and microcontrollers, and in model deployment the engine is the component that computes every prediction.

How Inference Engines Speed Up Models#

A deployed model never updates its weights, so an engine can drop everything training needs, such as stored activations for backpropagation, and rewrite the graph freely. It precomputes any part that depends only on constants, fuses chains of layers such as convolution, normalization, and activation into single kernels so intermediate results stay in fast on-chip memory, and plans memory so buffers are reused. It can lower the numeric precision to FP16 or INT8 through model quantization, measuring activation ranges on sample data for INT8. Engines such as TensorRT then time several candidate kernels for each layer on the actual device and keep the fastest, so the result is tied to that hardware:

Folding batch normalization into the preceding convolution is one of the most common rewrites. At inference, normalization only scales and shifts each channel by fixed amounts, so those numbers can be merged into the convolution's weights and bias. PyTorch exposes the same transformation as fuse_conv_bn_eval:

import torch
from torch.nn.utils.fusion import fuse_conv_bn_eval

torch.manual_seed(0)

# A trained convolution followed by batch normalization (random statistics here)
conv = torch.nn.Conv2d(16, 32, kernel_size=3, padding=1).eval()
bn = torch.nn.BatchNorm2d(32).eval()
bn.running_mean.uniform_(-1, 1)
bn.running_var.uniform_(0.5, 2.0)

# Fold the normalization's scale and shift into the convolution's weights and bias
fused = fuse_conv_bn_eval(conv, bn)

# One layer now replaces two and produces the same output
x = torch.randn(1, 16, 64, 64)
with torch.inference_mode():
    diff = (bn(conv(x)) - fused(x)).abs().max().item()
print(f"Max difference after fusing: {diff:.1e}")

It prints a maximum difference of 2.1e-06, which is floating-point rounding: the single fused layer computes the same function as the two original layers with one pass over memory instead of two.

Common Inference Engines#

• TensorRT: NVIDIA's engine for its data center GPUs and Jetson modules. It builds a plan for one GPU model and TensorRT version, so engines are rebuilt when either changes.
• OpenVINO: Intel's toolkit for its CPUs, integrated and discrete GPUs, and NPUs.
• ONNX Runtime: Microsoft's cross-platform engine for ONNX models, which hands work to hardware back ends called execution providers, including CUDA, TensorRT, OpenVINO, and Core ML.
• LiteRT: Google's runtime, formerly TensorFlow Lite, for Android, embedded Linux, and microcontrollers, with delegates for mobile GPUs and NPUs.
• Core ML: Apple's on-device framework, which splits work across the CPU, GPU, and Neural Engine.
• vLLM: An open-source engine for large language models that adds request scheduling and paged memory for the attention cache to serve many users at once.

The Ultralytics YOLO API chooses the engine from the file it loads, so the same predict call runs a .pt checkpoint in PyTorch, an .onnx file in ONNX Runtime, a .engine file in TensorRT, or an OpenVINO folder in OpenVINO. Exporting Ultralytics YOLO26 to OpenVINO and loading the result switches engines without other code changes:

from ultralytics import YOLO

# Export the PyTorch model to OpenVINO's format
model = YOLO("yolo26n.pt")
path = model.export(format="openvino")  # creates 'yolo26n_openvino_model/'

# Loading the exported folder runs predictions through the OpenVINO engine
ov_model = YOLO(path)
results = ov_model("https://ultralytics.com/images/bus.jpg")
results[0].save(filename="bus_openvino.jpg")

Inference Engines vs. Related Terms#

• PyTorch and other training frameworks: Keep the graph flexible and record operations for gradients so models can be designed and trained. They also run inference, with more overhead than an engine built for one device.
• ONNX: A file format that describes a model's graph and weights. ONNX Runtime and TensorRT are engines that read it.
• Model serving: Wraps one or more engines in a network service with an API, request batching, and version management, as NVIDIA Dynamo-Triton, formerly Triton Inference Server, does for TensorRT, ONNX Runtime, and PyTorch models.
• Expert system inference engine: The older meaning of the term, from rule-based AI, where the inference engine applies if-then rules to a knowledge base by forward or backward chaining.