ONNX (Open Neural Network Exchange)

ONNX (Open Neural Network Exchange) is an open standard file format for machine learning models. An .onnx file stores a trained model as a computation graph of standard operators together with its weights, so a model trained in one framework, such as PyTorch, can run in other runtimes and on many kinds of hardware. Facebook and Microsoft launched ONNX in September 2017, and it has been a graduated project of LF AI & Data, the Linux Foundation's AI foundation, since November 2019. ONNX is the usual hand-off point between training and model deployment: one exported file can feed ONNX Runtime, TensorRT, OpenVINO, and the compilers of many edge AI accelerators.

What an ONNX File Contains#

An ONNX model is serialized with Protocol Buffers and holds four parts:

• Graph: Nodes that call operators such as Conv, MatMul, and Relu, connected by named tensors. The graph's inputs and outputs carry data types and shapes, and each dimension can be fixed or symbolic, such as a variable batch size.
• Initializers: The trained weights. Protocol Buffers caps a single file at 2 GB, so larger models store their weights in external data files next to the .onnx file.
• Opset version: The operators are versioned in opsets, and a runtime must support the opset a model was exported with. New opsets add operators without changing old ones, such as the Attention operator added in opset 23.
• Metadata: The producer name, model version, and custom key-value pairs, such as class names.

A second operator domain, ONNX-ML, covers classical models such as tree ensembles and linear classifiers, so converters like skl2onnx and onnxmltools export scikit-learn, LightGBM, and XGBoost models to the same format. Viewers such as Netron display the graph node by node. In practice ONNX sits between many training frameworks and many runtimes:

Running ONNX Models#

ONNX Runtime, Microsoft's open-source inference engine, runs ONNX files on servers, desktops, phones, and browsers. It optimizes the graph when it loads a model, for example by folding constants and fusing a convolution with the batch normalization that follows it, and then hands each part of the graph to an execution provider for the hardware at hand: CUDA or TensorRT on NVIDIA GPUs, OpenVINO on Intel hardware, DirectML on Windows GPUs, Core ML on Apple devices, QNN on Qualcomm NPUs, and WebGPU in browsers. Windows ML, Microsoft's local inference framework, runs on ONNX Runtime, and other engines such as TensorRT and OpenVINO import ONNX files and compile them into their own formats.

Ultralytics YOLO26 exports to ONNX with model.export(format="onnx"), and Ultralytics' TensorRT export builds its engine from that same ONNX file. The exported file runs in ONNX Runtime without PyTorch:

import onnxruntime as ort
from ultralytics import YOLO

# Export YOLO26n to ONNX with symbolic batch, height, and width dimensions
model = YOLO("yolo26n.pt")
onnx_path = model.export(format="onnx", dynamic=True)

# Load the graph in ONNX Runtime, which needs neither PyTorch nor Ultralytics
session = ort.InferenceSession(onnx_path, providers=["CPUExecutionProvider"])
for tensor in session.get_inputs() + session.get_outputs():
    print(tensor.name, tensor.shape)

It prints the graph's input, images, with symbolic batch, height, and width dimensions, and its raw prediction output, output0, which is the signature any ONNX runtime uses to feed the model and read its results.

ONNX vs. Related Formats and Tools#

• ONNX Runtime: The inference engine that executes ONNX files, a separate Microsoft-led project that people often shorten to "ONNX".
• TensorRT: NVIDIA's compiler, which turns an ONNX file into an engine optimized for one GPU type and TensorRT version, so teams keep the ONNX file as the portable source and rebuild engines for each target.
• PyTorch formats: .pt checkpoints and safetensors files store weights for code that rebuilds the model, and TorchScript and torch.export capture graphs for PyTorch's own runtimes.
• Other deployment formats: OpenVINO IR, LiteRT .tflite files, Core ML packages, and GGUF files for quantized LLMs each target specific runtimes. Apple's coremltools converts PyTorch models directly and dropped ONNX input in version 6.

Real-World Applications#

• Cross-platform apps: One ONNX file serves Linux servers, Windows apps through Windows ML, and Android and iOS apps through ONNX Runtime Mobile.
• Browser inference: ONNX Runtime Web runs models on the user's device with WebGPU or WebAssembly, so camera frames for features like pose estimation never leave the browser.
• Edge accelerators: NPU toolchains from vendors such as Hailo, Qualcomm, and Rockchip accept ONNX models as input and compile them, often with quantization to INT8, for their chips.
• Classical models in production: Gradient-boosted trees and scikit-learn pipelines converted to ONNX run in C#, Java, and C++ services through ONNX Runtime, with no Python interpreter in the serving path.