Model Routing

Model routing is the practice of choosing, for each incoming request, which of several deployed AI models should handle it. A router inspects the request, and in some designs a cheap model's first answer, then forwards it to the candidate that meets the quality target at the lowest cost or inference latency. Easy requests go to a small, fast model, and only the inputs that need it reach a larger one. The savings can be large because model prices vary widely: in FrugalGPT, Lingjiao Chen, Matei Zaharia, and James Zou at Stanford found that the fees of popular large language model (LLM) APIs differed by about two orders of magnitude, and their learned cascade matched GPT-4 accuracy at up to 98% lower cost.

Routing Strategies#

Routers differ in what they know when they decide:

• Rule-based routing: Explicit request fields pick the model, such as the task, input language, file type, or customer tier. Rules are predictable and easy to audit, and they cannot judge how hard a particular input is.
• Learned routing: A small classifier predicts, before any large model runs, whether a weaker model's answer will be good enough. RouteLLM, from Isaac Ong and colleagues at LMSYS and UC Berkeley in 2024, trained such routers on human preference data from Chatbot Arena and cut costs by more than half in some settings without lowering answer quality.
• Cascades: The cheapest model answers first, and the request escalates when that answer's confidence or a scoring model's verdict falls below a threshold. Cascades pay for a second call on hard inputs and decide with the first answer in hand.

Routing has moved into products. When OpenAI launched GPT-5 in ChatGPT in August 2025, a real-time router picked between a fast model and a deeper reasoning model for each message, using conversation type, complexity, tool needs, and explicit intent such as "think hard about this." Cloud platforms offer managed routers as well, such as Amazon Bedrock intelligent prompt routing, which predicts response quality for each prompt and picks between two models of the same family.

A Confidence Cascade for Vision#

The same idea applies to model deployment for object detection. YOLO26 comes in five sizes behind one API, from YOLO26n at 40.9 COCO mAP and 38.9 ms per image on a CPU with ONNX to YOLO26x at 57.5 mAP and 525.8 ms, so a cascade can run the nano model on every image and send only images with low-confidence detections to the extra-large one:

from ultralytics import YOLO

source = "https://ultralytics.com/images/bus.jpg"
small_model = YOLO("yolo26n.pt")
large_model = YOLO("yolo26x.pt")

# Every image goes to the small model first
results = small_model(source)
route = "yolo26n"

# Escalate when the small model finds nothing or is unsure about any detection
confidences = results[0].boxes.conf
if len(confidences) == 0 or confidences.min() < 0.5:
    results = large_model(source)
    route = "yolo26x"

print(f"answered by {route}: {len(results[0].boxes)} objects")

The printout names the model that produced the final answer. Raising the 0.5 threshold sends more images to YOLO26x, so it sets the trade-off between mean average precision and average cost per image, which is tuned on held-out images from the deployment.

Related Terms#

Model routing is often mixed up with neighboring ideas:

• Load balancing: Spreads requests across identical replicas of one model to share load. Every replica gives the same answer.
• Mixture of experts: A gating network inside one model sends each token to a few expert layers, and the experts train jointly with the gate.
• Model ensemble: Runs several models on the same input and combines their outputs, which costs more per request.
• AI gateway: The infrastructure layer between applications and model providers, which often hosts the router next to authentication, rate limits, and logging.
• AI agent orchestration: Chooses which agent or tool handles each step of a task, and each of those steps can make its own model routing decision.

Routers are evaluated on representative traffic against the baseline of always calling the strongest model. RouteLLM measures performance gap recovered, the share of the quality difference between the weak and strong model that a router retains, at each fraction of calls sent to the strong model. In production, model monitoring tracks the escalation rate, latency, and cost per request of each route, because a shift in incoming traffic moves all three.