Synthetic Data

Synthetic data is information produced by a simulation, a set of rules, or a generative AI model instead of being measured from real events, built to reproduce the statistical patterns of the real data it stands in for. A rendered street scene, a table of invented patient records whose columns correlate like real ones, and a math exercise written by a language model are all synthetic data. Because the generator creates every example, it also knows the answer for each one, so a rendered image arrives with exact bounding boxes, segmentation masks, and depth values at no labeling cost. Synthetic data generation is the process that produces it, from game-engine rendering to diffusion models.

Types of Synthetic Data#

Statistician Donald Rubin proposed fully synthetic data in 1993 so that census agencies could publish detailed records without releasing any real respondent's answers, and the field still distinguishes three kinds:

• Fully synthetic data: Every record or image comes from a generator, as in a dataset rendered from 3D scenes or a table sampled from a model fit to real records.
• Partially synthetic data: Only sensitive fields are replaced, such as synthesized incomes in a survey file, while the rest of each record stays real.
• Hybrid datasets: Real and synthetic examples share one training set, while the validation and test sets stay real so that metrics reflect real-world accuracy.

Synthetic data spans every modality: images and video from simulators, LiDAR and radar streams, tabular records, time series, and text written by large language models.

Why Synthetic Data Is Used#

Synthetic data fills gaps that real collection cannot. Collisions, rare defects, and extreme weather are too dangerous or too rare to record in volume, so simulators render them on demand for automotive perception and robotics. Records that never belonged to a real person reduce the risk of exposing patients or customers, which matters under data privacy rules in healthcare and finance. Generators can also oversample underrepresented classes, skin tones, or viewpoints to counter dataset bias. Microsoft researchers showed the approach can replace real images entirely for some tasks: their ICCV 2021 paper Fake It Till You Make It trained face keypoint localization and face parsing models only on rendered faces and matched the accuracy of training on real photos.

Dense geometry is another common reason to go synthetic. Ultralytics YOLO26 depth estimation models are pretrained on a mix of about 2.19 million images that includes three synthetic sources: Apple's ray-traced indoor scenes in Hypersim, AirSim environments in TartanAir, and KITTI driving scenes re-rendered under varied weather and lighting in Virtual KITTI 2. Each supplies a depth value for every pixel, while the real KITTI LiDAR scans in the same mix cover only scattered points.

Measuring Synthetic Data Quality#

Synthetic datasets are judged on three properties. Fidelity measures how closely they match real distributions, from column means and correlations in tables to image realism scores such as Fréchet Inception Distance (FID). Utility measures whether a model trained on synthetic data works on real data, usually by training on synthetic and testing on real (TSTR). Privacy measures whether real records leak through, using distances to the nearest real record, membership inference attacks, or formal differential privacy guarantees on the generator.

This TSTR check fits a Gaussian to each class of a real medical table and trains the same classifier on real and on synthetic rows:

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

# Real breast tumor measurements, split into training rows and a held-out real test set
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)

# Fully synthetic training set: 500 rows per class sampled from a Gaussian fit to that class's real rows
rng = np.random.default_rng(0)
classes = [X_train[y_train == c] for c in (0, 1)]
X_syn = np.vstack([rng.multivariate_normal(rows.mean(0), np.cov(rows.T), 500) for rows in classes])
y_syn = np.repeat([0, 1], 500)

# Train the same classifier on real or on synthetic rows, then test both on the same real rows
for name, (X_fit, y_fit) in {"real": (X_train, y_train), "synthetic": (X_syn, y_syn)}.items():
    model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_fit, y_fit)
    print(f"trained on {name} data: {model.score(X_test, y_test):.1%} accuracy on real test data")

The model trained on real rows scores 97.7% and the one trained on 1,000 synthetic rows scores 95.9%. The 1.8-point gap measures the signal the simple Gaussian generator failed to capture, and none of the synthetic rows is a copy of a real patient's record.

Risks and Limits#

A model can learn the shortcuts of a renderer, such as perfect edges or repeated textures, and then fail on real images. This sim-to-real gap is narrowed by domain randomization, by mixing in real data, or by fine-tuning on a small real set. A generator also inherits the blind spots of the data it learned from, so it rarely produces the long-tail cases that motivated it unless they are specified explicitly. Training generative models repeatedly on their own outputs causes model collapse, which a 2024 Nature paper by Shumailov and colleagues showed erases the rare tails of the original distribution first.

Synthetic Data vs. Related Terms#

• Data augmentation: Transforms existing real examples with flips, crops, and color shifts, so every new sample is derived from a specific original.
• Digital twin: A virtual model of one specific real asset, kept in sync with its sensor data. Rendering a twin is one way to produce synthetic data of that asset.
• Anonymized data: Real records with names and identifiers removed or masked, so each row still describes a real person.