Model Serving

Model serving is the hosting of a trained machine learning model behind a network endpoint, usually HTTP or gRPC, so that applications can send inputs and receive predictions. A serving system loads the weights once, keeps them in memory on CPUs or GPUs, accepts many concurrent requests, groups them into batches, adds or removes replicas as traffic changes, swaps model versions, and reports its health. Google open-sourced TensorFlow Serving in February 2016 for running TensorFlow models in production, and serving is now the runtime half of machine learning operations (MLOps), between packaging a model and monitoring its predictions.

How a Serving System Handles Requests#

Each request passes through validation, preprocessing such as decoding and resizing an image, a queue, the model itself, and postprocessing into a response format such as JSON. The queue is where most serving decisions happen. Dynamic batching holds requests for a few milliseconds so the accelerator processes several inputs in one forward pass, which raises throughput at a small cost in inference latency. When the queue is full, the server rejects new requests with HTTP 429 so clients back off.

Serving teams track latency percentiles such as p95 and p99, throughput in requests per second, accelerator utilization, error rate, and cold-start time, the delay before a replica scaled up from zero can answer. Versions roll out gradually: a canary release sends a small share of traffic to the new model, and shadow deployment sends copies of live requests to it without returning its answers.

This FastAPI app loads YOLO26 once at startup and returns detections as JSON. FastAPI's TestClient sends one request through the full HTTP stack in the same process, so the example runs without starting a server:

from fastapi import FastAPI
from fastapi.testclient import TestClient
from ultralytics import YOLO

app = FastAPI()
model = YOLO("yolo26n.pt")  # load the weights once, when the server starts


@app.get("/predict")
async def predict(url: str):
    """Detect objects in the image at `url` and return them as JSON."""
    results = model(url)
    return results[0].summary()


client = TestClient(app)
response = client.get("/predict", params={"url": "https://ultralytics.com/images/bus.jpg"})
print(response.status_code, response.json()[0])

The output shows status 200 and the first detection as a dictionary with its class name, confidence, and box coordinates, the structure summary() produces for every detected object. Because the handler is async def, FastAPI runs it on a single event loop, so the shared model never handles two requests at once. Servers that need parallelism run several worker processes, each loading its own model.

Serving Patterns and Software#

How a model is served depends on when its answers are needed, from real-time inference on camera feeds to overnight jobs:

• Online serving: Each request gets its answer while the caller waits, over REST or gRPC, as in user-facing apps and live camera feeds.
• Batch serving: A scheduled job scores a large dataset offline, such as a nightly run over a day of images, and optimizes throughput over latency.
• Streaming: Results are pushed continuously as video frames or sensor readings arrive.
• Edge AI: The model runs on the device itself, often after export to an optimized format, with no network round trip.

General-purpose serving software includes NVIDIA Dynamo-Triton, formerly Triton Inference Server, along with TensorFlow Serving, Ray Serve, BentoML, and KServe on Kubernetes. PyTorch's TorchServe is in limited maintenance with no further updates planned. LLM servers such as vLLM, SGLang, and TensorRT-LLM add continuous batching and paged KV cache management for token-by-token generation. Most of these systems run in containers so that the model, runtime, and dependencies ship as one image.

Serving vs. Deployment and Inference Engines#

Model serving is closely tied to a few neighboring terms. Model deployment is the wider process of putting a model into production, including packaging, infrastructure, and rollout, and serving is the part that answers requests once it is live. An inference engine such as TensorRT, ONNX Runtime, or OpenVINO executes the model's computations, and a serving system wraps one or more engines with networking, queuing, and scaling.

Ultralytics Platform serves YOLO models through dedicated endpoints, each a single-tenant service in one of 42 regions with /predict, /health, and generated /docs routes. The default size scales to zero when idle and can take up to about a minute to start, and replacing the model keeps the endpoint URL while the previous model keeps serving until the new one is ready.