Compound AI Systems

A compound AI system is an application that solves a task with several interacting components, such as multiple model calls, retrievers, external tools, rules, and human review, instead of a single call to one model. Matei Zaharia and coauthors from Berkeley, Stanford, Databricks, and other groups named the trend in a February 2024 Berkeley AI Research post, arguing that state-of-the-art results were increasingly coming from systems rather than from monolithic models. Their examples include AlphaCode 2, which generates up to a million candidate programs with large language models (LLMs) and then filters and scores them, AlphaGeometry, which pairs an LLM with a symbolic deduction engine to solve olympiad geometry problems, and ChatGPT, whose model decides when to browse the web, run Python, or generate an image. The same post cites Databricks data showing that 60% of LLM applications use some form of retrieval-augmented generation (RAG) and 30% use multistep chains.

Why Build Systems Instead of Bigger Models#

• Cheaper quality gains: Sampling, voting, and verification can raise accuracy more cheaply than training a larger model. Google reported Gemini Ultra's 90.04% on the MMLU benchmark with CoT@32, which samples 32 chain-of-thought answers and returns the majority when enough of them agree, against 86.4% for GPT-4 with five-shot prompting.
• Current data: A model's knowledge stops at its training cutoff, and a retriever or vector database supplies fresh, private, or access-controlled information at query time.
• Control and trust: Filters, AI guardrails, citations, and verification steps make behavior easier to constrain than prompting one model and hoping it complies.
• Cost targets: Model routing sends easy requests to small models and hard ones to large models, so each application can choose its own trade-off between quality, latency, and price.

A single request path can combine all four ideas:

Compound Systems in Computer Vision#

Production vision applications are compound almost by default: a detector finds objects, a tracker links them across frames, geometric rules turn positions into events, and storage and alerts act on the events. Ultralytics Solutions package several such systems around a YOLO26 model: object counting combines detection, multi-object tracking, and a line or region test that counts each track ID once as it enters or leaves, and the security alarm system emails a frame once the number of detections in view reaches a set threshold. Smaller pipelines follow the same pattern of a model producing structured observations for separate logic to act on:

from ultralytics import YOLO

# Perception component: the detector turns pixels into structured observations
model = YOLO("yolo26n.pt")
results = model("https://ultralytics.com/images/bus.jpg")
detections = results[0].summary()

# Policy component: deterministic rules accept confident detections and flag uncertain ones
accepted = [d["name"] for d in detections if d["confidence"] >= 0.5]
uncertain = [d["name"] for d in detections if d["confidence"] < 0.5]

# Routing component: uncertain results go to a person instead of downstream automation
action = "human review" if uncertain else "auto-approve"
print({"accepted": accepted, "uncertain": uncertain, "action": action})

The printed decision comes from rules a reviewer can read and change without retraining the model, and labeling the uncertain images it routes to people adds training data for the next model in an active learning loop.

Building and Operating Compound Systems#

Errors multiply across components: three stages that are each right 90% of the time produce a correct end-to-end result only about 73% of the time, so teams measure the whole system on a fixed task suite rather than each part alone. Components that cannot be trained with gradients, such as search engines and rules, need other optimizers, and frameworks like DSPy tune every prompt in a pipeline against one end-to-end metric. In production, observability traces follow each request through every model call and tool, and machine learning operations (MLOps) practices such as versioning and model monitoring apply to every component, since an update to one can silently change the inputs of the next.

Compound AI vs. Related Terms#

• Model ensembles: Combine several models' predictions for the same input into one output, which makes an ensemble one pattern a compound system can contain.
• Agentic workflows: Compound systems in which a model decides the control flow at run time, choosing tools and steps. Many compound systems use fixed pipelines with no such decisions.
• Mixture of experts (MoE): Routes tokens among expert subnetworks inside a single model whose parts are trained together. Compound systems route requests among separately built and deployed components.