TensorRT

TensorRT is NVIDIA's SDK for high-performance deep learning inference on NVIDIA GPUs. It takes a trained network, usually as an ONNX file, and in a build phase compiles it into an engine, also called a plan file: a GPU-specific program that fuses layers, runs in reduced precision, and uses the fastest kernels measured on the target GPU. In the runtime phase, an application loads the engine and executes it, typically with lower inference latency and higher throughput than the original framework. TensorRT ships as C++ and Python libraries plus the trtexec command-line tool, and its current major version is TensorRT 11, at release 11.4 as of October 2026.

How TensorRT Optimizes a Model#

The builder applies several optimizations to the network graph:

• Layer and tensor fusion: A convolution, its bias, and its activation become one kernel, and layers whose outputs are never used disappear, so intermediate results stay on chip instead of making round trips through GPU memory.
• Reduced precision: FP16 and BF16 halve memory traffic compared with FP32, INT8 and FP8 halve it again, and INT4 (for weights only) and FP4 go further on recent GPUs, a form of model quantization. TensorRT 11 removed the FP16 and INT8 builder flags and the implicit INT8 calibration of TensorRT 10 and requires strongly typed networks instead, in which cast nodes and quantize-dequantize pairs declare each layer's precision. NVIDIA's Model Optimizer can insert those nodes into an ONNX file.
• Kernel auto-tuning: For every layer, the builder times candidate implementations, called tactics, on the actual GPU and keeps the fastest.
• Memory planning: Activation buffers are reused across layers whose lifetimes do not overlap, which shrinks the engine's memory footprint.
• Dynamic shapes: Optimization profiles with minimum, optimal, and maximum shapes let one engine accept varying batch sizes or image resolutions, tuned for the optimal shape.

Building an engine can take minutes, and the result is specific to the hardware. By default, an engine runs only on the GPU type it was built on and with the same TensorRT version, and hardware-compatible and version-compatible engines trade some speed for portability. Deployments therefore keep the ONNX file and build engines where they run:

Exporting a Model to TensorRT#

Ultralytics YOLO26 models export to TensorRT with model.export(format="engine"), which first writes an ONNX file and then builds the engine on the local GPU, in FP16 with quantize=16 or in INT8 with quantize=8, calibrated on images from the data dataset. On TensorRT 11, the export applies these precisions through Model Optimizer, as the strongly typed API requires:

from ultralytics import YOLO

# Build an FP16 TensorRT engine on the local NVIDIA GPU (writes yolo26n.engine)
model = YOLO("yolo26n.pt")
model.export(format="engine", quantize=16)

# Load the engine and run inference through the same API
trt_model = YOLO("yolo26n.engine")
results = trt_model("https://ultralytics.com/images/bus.jpg")
results[0].save(filename="bus_tensorrt.jpg")

The saved image shows the detections that the FP16 engine produced on the bus photo, through the same YOLO interface as the PyTorch model.

TensorRT vs. Related Tools#

• ONNX: The portable model format that TensorRT most often consumes. One ONNX file can be compiled into separate engines for a data center GPU and an embedded module.
• ONNX Runtime with the TensorRT execution provider: Runs the subgraphs TensorRT supports as TensorRT engines and falls back to CUDA or CPU kernels for the rest, keeping the ONNX Runtime API.
• PyTorch: The framework where most models are trained. Torch-TensorRT compiles PyTorch models with TensorRT directly, without a separate ONNX step.
• Other inference engines: OpenVINO for Intel hardware, Core ML for Apple devices, and LiteRT for Android and embedded devices fill the same role on other chips.
• TensorRT-LLM: NVIDIA's open-source library for large language model inference, now architected on PyTorch, with optimized attention kernels, quantization, speculative decoding, and multi-GPU parallelism.

Real-World Applications#

• Video analytics: NVIDIA's DeepStream SDK runs TensorRT engines inside multi-camera pipelines for retail, traffic, and security analytics, where many streams share one GPU.
• Industrial inspection: Cameras on fast production lines need real-time inference within the time each part spends in view, which reduced-precision engines make possible on compact GPUs.
• Autonomous vehicles: Perception networks for autonomous vehicles run as TensorRT engines on NVIDIA DRIVE platforms such as DRIVE AGX Orin and Thor.
• Robotics and edge devices: NVIDIA Jetson modules ship TensorRT with JetPack. TensorRT 11.4 does not support JetPack, so Jetson deployments build their engines with the TensorRT 10 release that their JetPack version provides.