Ensemble

Ensemble learning trains several models for the same task and combines their predictions, by voting, averaging, or a learned combiner, into one output that is usually more accurate and more stable than any single member. It works because different models make different mistakes, and combining them cancels part of each one's error. Anders Krogh and Jesper Vedelsby made this exact for averaged regression models in 1995: the ensemble's squared error equals the average error of its members minus their average disagreement with the ensemble, so an averaged ensemble is never worse than its average member, and diversity among members is what makes it better.

Bagging, Boosting, and Stacking#

Three families cover most ensembles in machine learning, and they act on different sides of the bias-variance tradeoff.

Bagging, short for bootstrap aggregating, was introduced by Leo Breiman in 1996. It trains each member independently on a bootstrap sample, drawn from the training set with replacement, and averages or votes over the results, which mainly reduces variance. It suits unstable learners such as deep decision trees, and Breiman's random forest of 2001 adds a random subset of features at every split to make the trees even less alike.

Boosting trains members in sequence, each one focusing on what the ensemble so far gets wrong, which mainly reduces bias. AdaBoost, from Yoav Freund and Robert Schapire, raises the weight of misclassified examples before training the next weak learner. Jerome Friedman's gradient boosting fits each new tree to the gradient of the loss function, the approach behind XGBoost, LightGBM, and CatBoost, which remain the strongest models on many tabular datasets.

Stacking, or stacked generalization, was proposed by David Wolpert in 1992. A meta-model, often a logistic regression, learns how to combine heterogeneous base models from their predictions on held-out folds, giving more weight to the models whose held-out predictions are most reliable.

This scikit-learn example compares a single tree with one ensemble from each family on a breast cancer diagnosis dataset, using 5-fold cross-validation:

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import GradientBoostingClassifier, RandomForestClassifier, StackingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier

X, y = load_breast_cancer(return_X_y=True)  # 569 tumors, 30 measurements each

forest = RandomForestClassifier(n_estimators=200, random_state=0)
boosting = GradientBoostingClassifier(random_state=0)
models = {
    "Single decision tree": DecisionTreeClassifier(random_state=0),
    "Bagging (random forest)": forest,
    "Boosting (gradient boosting)": boosting,
    # Stacking: a logistic regression learns how to combine the forest and the boosted trees
    "Stacking": StackingClassifier([("forest", forest), ("boosting", boosting)], LogisticRegression()),
}

for name, model in models.items():
    print(f"{name}: {cross_val_score(model, X, y, cv=5).mean():.3f}")

The single tree scores 0.917, the random forest 0.960, gradient boosting 0.963, and the stacked model 0.963. Every ensemble cuts the tree's error roughly in half. Stacking matches its stronger member because 16 of the 23 tumors the forest misclassifies are also misclassified by the boosted trees, which leaves the meta-model little disagreement to exploit.

Ensembles in Deep Learning and Vision#

Deep ensembles train several neural networks from different random initializations and average their outputs. Balaji Lakshminarayanan and colleagues at DeepMind showed in 2017 that this improves both accuracy and uncertainty quantification, because members disagree most on inputs unlike the training data. In object detection, the bounding boxes from several detectors are pooled and merged with non-maximum suppression (NMS) or Weighted Boxes Fusion, which averages overlapping boxes instead of keeping only the most confident one. Ensembles of different architectures are common in medical image analysis and competition entries, where accuracy matters more than inference cost.

That cost grows with every member, which limits ensembles on edge AI devices. Knowledge distillation was introduced partly to solve it: Geoffrey Hinton, Oriol Vinyals, and Jeff Dean trained a single model to match the averaged outputs of an ensemble. Dropout is a cheaper relative, since training with it approximates averaging an exponential number of thinned subnetworks.

Ensembles, Model Soups, and Mixtures of Experts#

A model ensemble usually means combining separately trained networks at inference, the most common form of ensemble learning in deep learning. Model soups and other model merging methods average the weights instead of the predictions, which keeps the cost of one model but works only for models fine-tuned from the same pretrained weights. A mixture of experts (MoE) trains its experts jointly with a router that sends each input to a few of them, so experts specialize instead of each solving the whole task.

Averaging weights along a single training run approximates an ensemble of that run's checkpoints at the cost of one model, as Pavel Izmailov and colleagues showed for stochastic weight averaging in 2018. Ultralytics YOLO training applies this to every YOLO26 run: it keeps an exponential moving average of the weights, updated after every optimizer step with a decay that ramps up toward 0.9999, and validates and saves that average in best.pt and last.pt instead of the raw weights.