Tabular Foundation Models

Tabular foundation models are pretrained neural networks that make predictions on new tables of rows and columns without being trained on them. Given a labeled table and a set of rows to predict, they use in-context learning: the labeled rows enter the model as context, and a single forward pass returns predictions for every new row, with no gradient descent updates. A gradient-boosted tree model such as XGBoost, LightGBM, or CatBoost has to be fit and tuned for each dataset. A tabular foundation model is trained once and applies what it learned to any small table it receives.

How TabPFN Learns From Synthetic Tables#

The best-known example is TabPFN (Tabular Prior-data Fitted Network), created by Noah Hollmann, Samuel Müller, Frank Hutter, and colleagues at the University of Freiburg and now developed by the company Prior Labs. It is a transformer pretrained entirely on synthetic data. Pretraining draws millions of artificial datasets from a prior over data-generating processes, mostly random structural causal models that link features to targets through random functions and noise. For each synthetic dataset the network sees some labeled rows and learns to predict the held-out ones, a form of meta-learning. Across many such tasks it learns to approximate Bayesian inference over that prior, so at prediction time it behaves like a learned statistical procedure rather than a model of any particular domain.

The first version, presented at ICLR 2023, handled classification with up to 1,000 training rows, 100 numeric features, and 10 classes, performing on par with AutoML systems while running up to 230 times faster. TabPFN v2, published in Nature in January 2025, extended the approach to classification and regression on datasets with up to 10,000 rows and 500 features, including categorical features and missing values. In classification, 2.8 seconds of TabPFN v2 outperformed an ensemble of the strongest baselines tuned for four hours. Version 2 gives each table cell its own representation and alternates attention across columns and across rows, so it does not depend on a fixed column order. Later releases raise the limits further, and TabPFN-3, released in May 2026, accepts up to one million training rows. Other tabular foundation models, such as TabICL and Amazon's Mitra, follow the same recipe of pretraining on synthetic tables and predicting in context.

Where They Fit Next to Gradient-Boosted Trees#

• Small datasets: With hundreds to a few thousand rows, tabular foundation models often match or beat tuned trees without any hyperparameter search, and their predicted probabilities are generally well calibrated.
• Large datasets: Boosted trees remain the practical default for millions of rows. Attention cost grows with the number of context rows, so foundation models need a GPU and more memory, and each prediction batch reprocesses the full training context, which makes inference slower than evaluating a tree ensemble.
• AutoML: AutoML searches over algorithms and settings for every new dataset. A tabular foundation model replaces most of that search with knowledge learned once during pretraining.
• Feature engineering: Still useful. The model cannot recover information that is missing from the columns or repair labels that are wrong.

Running TabPFN#

The tabpfn package exposes a scikit-learn style classifier. Its newest checkpoints download only after signing in and accepting Prior Labs' model license, while the version 2 weights selected below download without that step:

from sklearn.datasets import load_breast_cancer
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split
from tabpfn import TabPFNClassifier
from tabpfn.constants import ModelVersion

# 569 tumors described by 30 numeric measurements
features, targets = load_breast_cancer(return_X_y=True)
x_train, x_test, y_train, y_test = train_test_split(features, targets, test_size=0.2, random_state=0)

# fit() stores the labeled rows as context; no weights are updated
model = TabPFNClassifier.create_default_for_version(ModelVersion.V2)
model.fit(x_train, y_train)

# One forward pass predicts every test row from that context
probabilities = model.predict_proba(x_test)[:, 1]
print(f"ROC AUC: {roc_auc_score(y_test, probabilities):.3f}")

With no hyperparameters set, the classifier reaches a ROC AUC of about 0.998 on this held-out split. Model comparisons use repeated cross-validation across many datasets, since a 114-row test split is too small to separate strong models.

Many useful tables are built from the output of other models. Many useful tables are built from the output of other models. In Predict mode, result.to_df() returns each Ultralytics YOLO26 result as a Polars DataFrame with one row per detection and columns for the class name, class ID, confidence, and box coordinates, the row-and-column format a tabular foundation model reads as context.