Model Deployment

Model deployment is the process of putting a trained machine learning model into production, where applications send it new data and act on its predictions. It covers choosing where the model runs, converting it to a format the target runtime executes, connecting it to inputs and outputs through an API or an app, releasing new versions safely, and watching its behavior once real users depend on it. A model that scores well in a notebook is only an experiment until it is deployed, and deployment is the stage of machine learning operations (MLOps) where data science meets software engineering constraints such as latency, cost, uptime, and security.

Where Models Are Deployed#

• Cloud deployment: The model runs on servers behind a REST or gRPC endpoint, usually through a model serving system that batches requests and scales replicas with demand. It suits large models and variable traffic, and every request pays a network round trip.
• Edge deployment: The model runs on the device that captures the data, such as a phone, camera, robot, or NVIDIA Jetson module. Inputs never leave the device, which protects privacy, and the model keeps working offline, but limited memory and power usually require a compressed model.
• Serverless deployment: The platform starts instances on demand and scales them to zero when idle, trading a cold-start delay on the first request for paying nothing while no requests arrive.
• Batch deployment: A scheduled job scores a large dataset at once, such as nightly product recommendations, where throughput matters more than per-request speed.

Ultralytics Platform deploys a trained YOLO model to a dedicated endpoint in any of 42 regions, each with its own URL and monitoring, and endpoints on the default resource size scale to zero when idle. The choice of target follows from a few questions about the application, which the Ultralytics Academy lesson Choose a Deployment Target works through in detail:

Preparing a Model for Production#

Training frameworks are built for flexibility, so production models are usually exported to a format an inference engine runs efficiently, such as ONNX for portability, TensorRT for NVIDIA GPUs, OpenVINO for Intel hardware, or LiteRT and Core ML for phones. Model quantization and model pruning shrink the model further when the target is small. The exported model and its preprocessing code are then packaged with pinned dependencies, typically in a Docker container that runs the same way on a laptop, a server, or a Kubernetes cluster. Exporting a trained detector and loading the exported file for a quick check before release looks like this:

from ultralytics import YOLO

# Load a trained model (replace with your own best.pt)
model = YOLO("yolo26n.pt")

# Export to ONNX, a portable format that many inference engines run
path = model.export(format="onnx")
print(f"Deployable model: {path}")

# Smoke-test the exported file with the same API before shipping it
onnx_model = YOLO(path)
results = onnx_model("https://ultralytics.com/images/bus.jpg")
results[0].save(filename="bus_onnx.jpg")

The export call returns the path of the new yolo26n.onnx file, and the saved image confirms that the exported file loads and produces detections. Teams also compare the exported model's accuracy and speed with the original on a validation set, because format conversion and lower precision can change outputs slightly.

Releasing New Model Versions#

A new model version is stored in a model registry and released gradually, because offline metrics do not capture every production condition. In shadow deployment the new model receives a copy of live traffic while the current model still answers users, so the two can be compared without risk. A canary release then routes a small share of real traffic to the new version and widens it only while error rates, latency, and prediction quality stay healthy, and A/B tests measure the effect on business outcomes. Keeping the previous version deployable makes rollback a configuration change instead of an emergency rebuild:

Model Deployment vs. Related Terms#

• Model training: Learns the model's weights from a dataset. Deployment starts from the finished weights.
• Model serving: The runtime layer that answers prediction requests. It is one part of deployment, which also includes packaging, release, and monitoring.
• Inference latency: The time the deployed model takes to compute each individual prediction, one of the main requirements a deployment has to meet.

Monitoring After Deployment#

A deployed model degrades silently when the world changes. Model monitoring tracks operational signals such as latency, throughput, error rates, and hardware use, often with tools like Prometheus and Grafana, along with the statistics of inputs and predictions. When monitoring flags a problem, teams label the new failure cases, retrain, and send the new version through the same release process. Input statistics give the earliest warning: data drift, a shift in the input distribution such as a camera moved to a new angle or the change to winter lighting, shows up there days or weeks before ground-truth labels for production data arrive to measure accuracy.