XGBoost

XGBoost (eXtreme Gradient Boosting) is an open-source library for gradient-boosted decision trees and one of the most widely used tools for tabular data. Tianqi Chen started it as a research project at the University of Washington and released it in 2014, and the 2016 KDD paper XGBoost: A Scalable Tree Boosting System, written with Carlos Guestrin, described its regularized objective and systems design. The library, maintained by the DMLC community, handles classification, regression, and ranking from Python, R, the JVM, and C, runs on CPUs and GPUs, and scales out with Dask and Spark. Its paper reports that 17 of the 29 challenge-winning solutions published on Kaggle's blog in 2015 used XGBoost.

How XGBoost Works#

XGBoost builds an ensemble sequentially: it starts from a constant prediction and adds one tree per boosting round, each trained to reduce the loss left by the trees before it. The final prediction sums the trees' outputs, each shrunk by the learning rate (eta, 0.3 by default) so that no single tree dominates. Two ideas from the paper define how each tree is built:

• Second-order objective: For every row, XGBoost computes the first and second derivatives of the loss with respect to the current prediction, the gradient g and Hessian h. A leaf whose rows have gradient sum G and Hessian sum H gets the weight -G / (H + λ), the same Newton step that second-order optimizers take.
• Regularized trees: The objective adds a penalty γ per leaf and an L2 penalty λ on leaf weights, plus an optional L1 penalty α. A split is kept only when its gain, ½ [G_L²/(H_L + λ) + G_R²/(H_R + λ) − (G_L + G_R)²/(H_L + H_R + λ)] − γ, is positive, which builds regularization into tree construction and limits overfitting.

Trees are depth-limited (max_depth, 6 by default) and grow level by level unless grow_policy="lossguide" switches to leaf-wise growth. Training usually runs with early stopping on a validation set:

Missing Values, Speed, and Scale#

XGBoost's sparsity-aware split finding trains on tables with gaps without an imputation step. At every split it tries sending the rows with a missing value left and then right, and keeps the direction with the higher gain as that node's default branch, and the same mechanism lets it scan only the stored entries of sparse matrices such as one-hot encodings. This example hides 30% of all feature values in scikit-learn's breast cancer dataset and trains on the gaps directly:

import numpy as np
import xgboost as xgb
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)

# Hide 30% of all values; each split learns a default branch for missing ones
rng = np.random.default_rng(0)
X_train_nan = np.where(rng.random(X_train.shape) < 0.3, np.nan, X_train)
X_test_nan = np.where(rng.random(X_test.shape) < 0.3, np.nan, X_test)

for name, train, test in [("complete", X_train, X_test), ("30% missing", X_train_nan, X_test_nan)]:
    model = xgb.XGBClassifier(n_estimators=200, max_depth=4, learning_rate=0.1)
    model.fit(train, y_train)
    print(f"{name}: accuracy {model.score(test, y_test):.3f}")

It prints a test accuracy of 0.972 on the complete data and 0.951 with 30% of the values missing in both training and test sets.

For speed, the default hist tree method buckets each feature into at most 256 bins (max_bin) and searches splits over bin boundaries, the same approach LightGBM uses, while exact enumerates every sorted value. Setting device="cuda" moves training to an NVIDIA GPU, enable_categorical=True splits categorical columns natively, and external-memory mode trains on datasets larger than RAM. Since version 2.1, models save to the binary UBJSON format by default.

XGBoost vs. Related Methods#

• LightGBM: Microsoft's 2016 boosting framework, which grows trees leaf-wise by default and adds gradient-based row sampling and bundling of sparse features. Both libraries now use histograms by default and share most techniques.
• CatBoost: Yandex's boosting library, which builds symmetric trees and encodes categorical features with ordered target statistics.
• Random forest: A bagging method that trains deep trees independently on bootstrap samples and averages them, which lowers variance and needs little tuning.
• Deep learning: The standard for unstructured inputs such as images, audio, and text, where networks learn features from raw data. On tables of engineered features, boosted trees remain the usual baseline to beat, and tabular foundation models such as TabPFN now compete with them on small datasets.

Real-World Applications#

• Fraud detection and credit risk: Payment companies score each transaction from amount, merchant, location, device, and account history features, a core predictive modeling task where XGBoost's fast inference fits real-time decisions.
• Visual inspection: Manufacturing quality systems combine camera and sensor data. Predictions from Ultralytics YOLO26 supply the camera side: result.summary() returns each detected defect's class, confidence, and box coordinates, which a plant can aggregate per part and join with temperature and pressure readings as features for an XGBoost model that predicts failures.
• Ranking and click prediction: XGBRanker trains with the rank:ndcg objective by default to order search results, and binary classifiers predict ad click-through rates.
• Competition-winning problems: The KDD paper lists the tasks behind the 2015 winning solutions, including store sales prediction, high energy physics event classification, malware classification, and massive online course dropout prediction.