Random Forest

Random forest is a supervised learning algorithm that trains many decision trees on random variations of the same data and combines their predictions, by vote or averaged class probability for classification and by the mean for regression. Leo Breiman described the method in his 2001 paper Random Forests, building on his earlier bagging technique. A single deep tree memorizes its training rows, so its predictions swing with small changes in the data. Averaging hundreds of trees that each make different mistakes cancels much of that variance, which makes random forest one of the most reliable default models for tabular data. It needs no feature scaling, handles nonlinear interactions between features, and its test error levels off as more trees are added.

How Random Forest Works#

Random forest adds two sources of randomness to tree training, because an ensemble only helps when its members make different mistakes, and then combines the trees:

• Bootstrap aggregating (bagging): Each tree trains on a bootstrap sample, a set of rows drawn with replacement from the training data and equal in size to it. Some rows appear several times and about 37% of rows (a fraction of 1/e) are left out of any given sample.
• Random feature subsets: At every split, a tree considers only a random subset of the features. Scikit-learn's classifier defaults to the square root of the feature count, so a dataset with 100 features offers 10 candidates per split, and its regressor considers all features by default. A dominant feature therefore cannot sit at the top of every tree, which decorrelates the trees.
• Aggregation: Breiman's original method takes a majority vote of the trees. Scikit-learn's implementation averages the class probabilities of all trees instead, and regression forests average the trees' numeric predictions.

The trees are grown deep and left unpruned, so each one has low bias and high variance. Averaging many decorrelated trees lowers the variance while keeping the bias low, a favorable move along the bias-variance tradeoff. Because the trees are independent, they also train in parallel across CPU cores.

Out-of-Bag Error and Feature Importance#

The rows a tree never saw act as a built-in test set. The out-of-bag (OOB) score predicts each training row using only the trees whose bootstrap samples left it out, which estimates accuracy on unseen data without holding back a separate validation set. Forests also report feature importance. The default impurity-based importance measures how much each feature reduces impurity across all splits, and it favors features with many unique values, so permutation importance, the drop in score after shuffling one feature, is the more trustworthy choice when ranking features.

This example compares one fully grown tree with a scikit-learn random forest on the bundled breast cancer dataset using 5-fold cross-validation, then reads the forest's OOB score:

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier

# 569 tumors described by 30 numeric features, labeled malignant or benign
X, y = load_breast_cancer(return_X_y=True)

# One fully grown tree vs. a forest of 300 trees on bootstrap samples
tree = DecisionTreeClassifier(random_state=0)
forest = RandomForestClassifier(n_estimators=300, oob_score=True, random_state=0)

# Compare 5-fold cross-validated accuracy
print(f"Single tree: {cross_val_score(tree, X, y, cv=5).mean():.3f}")
print(f"Forest:      {cross_val_score(forest, X, y, cv=5).mean():.3f}")

# Out-of-bag score: each row is predicted only by trees that never trained on it
forest.fit(X, y)
print(f"OOB score:   {forest.oob_score_:.3f}")

The script prints 0.917 for the single tree, 0.963 for the forest, and an OOB score of 0.965. The forest cuts the tree's error rate from 8.3% to 3.7%, and the OOB estimate lands within 0.002 of the cross-validated accuracy from a single training run.

Real-World Applications#

• Credit risk and fraud detection: Banks score loan applications and transactions from tabular records such as income, account age, and spending patterns, and the forest's probability output sets the decision threshold.
• Healthcare risk prediction: Clinical models predict outcomes such as readmission from patient records, and permutation importance shows which measurements drive each prediction.
• Satellite image analysis: Random forests are a long-standing choice for land-cover mapping, classifying each pixel or field from its spectral band values and vegetation indices.
• Crop yield modeling: A YOLO26 detector counts fruit or plants in field images, one detection per object, and those per-plot counts join soil and weather readings as columns in a random forest that predicts yield.

Random Forest vs. Related Methods#

• Single decision tree: A tree is easy to read as a set of if-then rules, but small changes in the data can produce a completely different tree. A forest trades that readability for much lower variance.
• Gradient boosting: XGBoost, LightGBM, and CatBoost build shallow trees one after another, each correcting the errors of the trees before it. Boosting often scores higher on tabular benchmarks after tuning, while a random forest's independent trees train in parallel and give strong results with default settings.
• Tabular foundation models: Pretrained transformers predict on a new table in a single forward pass, without fitting a model per dataset, and now rival tuned tree ensembles on small tables.
• Extremely randomized trees: For each candidate feature, extra trees draw one random threshold instead of searching for the best one and keep the best of those random splits. Scikit-learn trains them on the full dataset without bootstrap sampling by default, and skipping the threshold search makes them faster to fit than a random forest of the same size.