Model Ensemble

A model ensemble combines the predictions of several trained models into a single output, by voting, averaging, or a learned combination, so that the errors of individual models cancel out. When the members make different mistakes, the ensemble is more accurate and more robust than its best member, which is why ensembles dominate machine learning competitions: the Netflix Prize was won in 2009 by a team that blended the predictions of many models, and the deep learning entry that won the 2012 ImageNet challenge averaged the predictions of seven convolutional networks to reach a 15.3% top-5 error. The cost is computational: every member runs on every input, so memory use and inference latency grow with the size of the ensemble.

How Model Ensembles Work#

An ensemble helps only when its members make different mistakes. Averaging the outputs of models whose errors are independent shrinks the variance of the combined prediction, while models that fail on the same inputs gain little from being combined. Bagging creates this diversity by training each member on a random resample of the training data, as in a random forest of decision trees, and boosting trains each new member to correct the errors of the ones before it, as in XGBoost and LightGBM. Deep ensembles train copies of one neural network from different random seeds, and the spread of their predictions also measures the model's uncertainty. Combining different architectures, such as a convolutional neural network and a vision transformer, gives members with different inductive biases and less correlated errors.

The combination rule depends on what the members output:

High-variance models such as unpruned decision trees benefit most. This scikit-learn example trains five trees that differ only in their random seed and averages their predicted probabilities:

import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

X, y = load_digits(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=0)

# Five decision trees that differ only in their random seed, which changes the features each split considers
trees = [DecisionTreeClassifier(max_features="sqrt", random_state=seed) for seed in range(5)]
for tree in trees:
    tree.fit(X_train, y_train)
print("Single trees:", ", ".join(f"{tree.score(X_test, y_test):.3f}" for tree in trees))

# Ensemble: average the predicted class probabilities and pick the most likely class
probabilities = np.mean([tree.predict_proba(X_test) for tree in trees], axis=0)
accuracy = (probabilities.argmax(axis=1) == y_test).mean()
print(f"Ensemble of 5: {accuracy:.3f}")

On 899 held-out digit images, the single trees score between 0.736 and 0.805, and the ensemble of all five scores 0.901, higher than any member, because each tree's mistakes fall on different images.

Ensembling Object Detectors#

Object detection ensembles must also merge boxes: several models usually find the same object at slightly different positions. The simplest method pools every model's boxes and applies non-maximum suppression (NMS), which keeps the highest-confidence box in each overlapping group. Weighted boxes fusion instead averages the coordinates of overlapping boxes, weighted by confidence, so every model contributes to the final position. Ultralytics YOLO26 builds a related ensemble into prediction: with augment=True, the detector runs on the image at full, 83%, and 67% scale, with the 83% copy flipped left to right, and merges the three sets of boxes with NMS. This example ensembles two YOLO26 models of different sizes the same way:

import torch
from torchvision.ops import batched_nms
from ultralytics import YOLO

# Two YOLO26 models of different sizes make different mistakes
models = [YOLO("yolo26n.pt"), YOLO("yolo26s.pt")]
results = [model("https://ultralytics.com/images/bus.jpg")[0] for model in models]

# Pool the boxes from both models
boxes = torch.cat([result.boxes.xyxy for result in results])
scores = torch.cat([result.boxes.conf for result in results])
classes = torch.cat([result.boxes.cls for result in results])

# Keep the highest-scoring box in each overlapping group of the same class
keep = batched_nms(boxes, scores, classes, iou_threshold=0.5)
print(f"{len(boxes)} boxes from both models, {len(keep)} after fusion")

Boxes that both models found collapse into one, and objects that only one model found survive the merge, which is how a detection ensemble raises recall.

Applications#

• Medical image analysis: Diagnostic models combine several networks to raise sensitivity, and the members' disagreement flags cases for a radiologist to review.
• Competitions and benchmarks: Winning solutions on Kaggle and similar leaderboards routinely average models trained with different architectures, seeds, and data folds, since the accuracy gain outweighs the extra compute.
• Autonomous vehicles: Perception systems combine models with different strengths, such as a camera-based detector and a lidar-based detector, so an object one of them misses can still be caught by the other.

Model Ensemble vs. Related Terms#

• Ensemble learning: A synonym for the overall technique. "Model ensemble" usually refers to the combined set of models that results.
• Model merging: Averages the weights of compatible models into one network, which then runs at the cost of a single model.
• Mixture of experts (MoE): A learned router sends each input to only a few expert sub-networks, which adds capacity without running every expert on every input.
• Knowledge distillation: Trains a single student model to imitate a teacher, which can be an entire ensemble. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean's 2015 distillation paper compressed an ensemble of speech recognition models into one model that kept most of the ensemble's improvement.