Decision Tree

A decision tree is a supervised learning model that predicts by asking a sequence of yes-or-no questions about an input's features, such as "Is income above $50,000?", and following the answers from the root to a leaf that holds the prediction. Classification trees predict a class at each leaf, and regression trees predict a number. Every prediction is a readable chain of rules, which makes trees a standard tool for explainable AI (XAI), and trees are the building blocks of random forests and gradient-boosted ensembles, which still outperform deep learning on medium-sized tabular datasets. The CART algorithm of Breiman, Friedman, Olshen, and Stone (1984) and Ross Quinlan's ID3 (1986) and C4.5 (1993) established how trees are learned from data.

How a Decision Tree Is Built#

Trees grow top-down by recursive partitioning. At each node, the algorithm scores candidate splits, each a feature and a threshold for numeric features, keeps the split that makes the two child nodes most homogeneous, and repeats the search in each child. Classification trees measure homogeneity with Gini impurity, the chance of mislabeling a random sample from the node if labels were assigned at random in the node's class proportions, or with entropy, whose reduction is called information gain. Regression trees choose the split that most reduces squared error, and each of their leaves predicts the mean target of its training samples. Growth stops when a node is pure, holds too few samples, or reaches a maximum depth. Each split looks only one step ahead, so this greedy search builds a good tree quickly, while finding the provably optimal tree is NP-hard.

Scikit-learn's DecisionTreeClassifier implements an optimized version of CART. This example grows a depth-2 tree on the iris flower dataset and prints the rules it learned:

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier, export_text

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target, random_state=42)

# A depth limit keeps the tree small enough to read and less prone to overfitting
clf = DecisionTreeClassifier(max_depth=2, random_state=42)
clf.fit(X_train, y_train)

print(export_text(clf, feature_names=iris.feature_names))
print(f"Test accuracy: {clf.score(X_test, y_test):.2f}")

The tree uses petal length alone: flowers at or below 2.45 cm are setosa (class 0), and a second threshold at 4.75 cm separates versicolor (class 1) from virginica (class 2), rules that classify 97% of held-out flowers correctly.

Every split is a threshold rule of this form, which a tree learns from data and a person can also set by hand. The Ultralytics AIGym workout monitor sets two such rules by hand for YOLO26 pose estimation output: it labels a frame "up" when the elbow angle computed from shoulder, elbow, and wrist keypoints exceeds 145° and "down" when it drops below 90°, and counts a repetition at each change from up to down.

Overfitting and Pruning#

A tree grown until every leaf is pure memorizes its training data, noise included, a textbook case of overfitting. Pre-pruning stops growth early with limits such as a maximum depth or a minimum number of samples per leaf. Post-pruning grows the full tree and then removes branches that add little; scikit-learn implements minimal cost-complexity pruning through the ccp_alpha parameter, where larger values prune more. Both kinds of limits are hyperparameters chosen by cross-validation.

Even a pruned tree has high variance: a small change in the training data can change the top split and, with it, every split below. Ensembles address this by combining many trees, either averaging trees trained on bootstrap samples, as random forests do, or adding trees one at a time to correct the errors of the previous ones, as gradient boosting does.

Strengths and Limitations#

Trees handle numeric and categorical features together, need no feature scaling, capture interactions and nonlinear effects without manual feature engineering, and predict with a handful of comparisons. Their splits are axis-aligned, so a diagonal boundary such as "spending exceeds income" becomes a staircase of many splits unless a ratio feature is added. Regression trees predict a constant within each leaf, so unlike linear regression they cannot extrapolate a trend beyond the range of the training data.

Applications#

• Credit decisions: Lenders use shallow trees and tree-based scorecards to sort applications by risk, and every decision traces back to the income, credit history, and debt thresholds that produced it.
• Clinical triage: Clinical decision rules shaped as trees route patients by vital signs and symptoms, for example to decide whether a patient needs immediate imaging.
• Customer churn: Trees split customers by usage, tenure, and support history into segments with high and low cancellation risk, and the paths describe each segment in plain terms.
• Predictive maintenance: Gradient-boosted trees on vibration, temperature, and runtime features estimate which machines are likely to fail before their next service.

Decision Trees vs. Related Terms#

• Random forests: Average many trees trained on bootstrap samples and random subsets of features, trading a single tree's readability for higher accuracy and stability.
• XGBoost, LightGBM, and CatBoost: Gradient-boosting libraries that add trees one at a time to correct the errors of earlier trees, with LightGBM built for speed on large datasets and CatBoost for native categorical features.
• Deep learning: Neural networks learn features from raw images, audio, and text, while trees split on features that already exist as table columns.
• Decision analysis trees: In operations research and business planning, a decision tree is also a diagram that an analyst draws by hand, with decision nodes, chance nodes, and payoffs, to compare options by expected value; it is built from estimated probabilities instead of learned from training data.