Synthetic Data Generation
Synthetic data generation creates artificial training data with simulators, procedural rules, generative models, and LLMs, often with exact labels.
Synthetic data generation is the process of producing artificial images, video, sensor readings, tables, or text with a simulator, procedural rules, or a generative model, so that a machine learning model can train or be tested on it. Its output is synthetic data; the generation process decides that data's quality through four choices: which method creates the samples, how labels are attached, how widely each scene varies, and how the result is checked against real data. In computer vision, the main attraction is that a generator that places every object can also write the data annotation for it, exact and free.
Main Generation Methods#
Simulators build a 3D scene from assets, set its lighting, materials, and camera, and render images together with exact masks, depth maps, and bounding boxes, because the scene description records where everything is. Open-source driving simulators such as CARLA produce camera, LiDAR, and radar streams for autonomous vehicles, and robotics simulators do the same for grasping and navigation. Domain randomization varies textures, lighting, and poses across renders so that models trained on them transfer to real cameras.
Procedural generation uses scripts and rules instead of a full scene. Cut-and-paste compositing, popularized by the 2017 "Cut, Paste and Learn" paper from Carnegie Mellon University, pastes segmented object crops onto new backgrounds and records each pasted box as a label. The simplest version draws shapes at random positions, and the generator writes each object's label as it draws:
import cv2
import numpy as np
rng = np.random.default_rng(0)
boxes = 0
for i in range(100):
# Random background color, then one to four contrasting discs in separate quadrants so none overlap
background = rng.integers(0, 256, 3)
image = np.full((320, 320, 3), background, dtype=np.uint8)
labels = []
for qx, qy in [(0, 0), (160, 0), (0, 160), (160, 160)][: rng.integers(1, 5)]:
r = int(rng.integers(10, 40))
x, y = qx + int(rng.integers(r, 160 - r)), qy + int(rng.integers(r, 160 - r))
color = (background + rng.integers(64, 193, 3)) % 256
cv2.circle(image, (x, y), r, color.tolist(), thickness=-1)
# The generator placed the disc, so its normalized box label is exact and free
labels.append(f"0 {x / 320:.4f} {y / 320:.4f} {2 * r / 320:.4f} {2 * r / 320:.4f}")
cv2.imwrite(f"synthetic_{i:03d}.jpg", image)
with open(f"synthetic_{i:03d}.txt", "w") as f:
f.write("\n".join(labels))
boxes += len(labels)
print(f"wrote 100 images with {boxes} exact box labels")It prints wrote 100 images with 256 exact box labels. Each line follows the YOLO label format and stores the class and the box center, width, and height as fractions of the image size, and none of them needed a human annotator.
Generative models learn the look of real data and sample new examples from it. Generative adversarial networks (GANs) and diffusion models produce photorealistic images from noise, a text prompt, or a reference image, and conditioning methods such as ControlNet fix the layout so that object positions stay known. Without that conditioning, generated images need labeling afterward. For tables, statistical synthesizers fit a joint distribution to real records with Gaussian copulas, Bayesian networks, or conditional GANs such as CTGAN (NeurIPS 2019) and then sample new rows.
Large language models generate text data: instructions, dialogues, code exercises, and reasoning traces. Microsoft trained phi-1, a 1.3B-parameter code model, on 6B tokens of filtered web text plus 1B tokens of textbooks and exercises written by GPT-3.5, and it reached 50.6% pass@1 on the HumanEval benchmark.
Choosing and Validating a Method#
The method follows from what the data must contain, and every method ends in the same check against real data:
Validation trains a model on the generated data, alone or mixed with real data, and evaluates it only on held-out real test data, as in Ultralytics' benchmarks of NVIDIA Cosmos synthetic data for defect detection. If accuracy on real images drops as the synthetic share grows, the generator has a gap in realism or coverage, and image-level scores such as Fréchet Inception Distance (FID) help locate it. Generators trained on real data can also memorize it, so privacy-sensitive pipelines check generated records against the originals, and some add differential privacy to training.
Ultralytics Platform adds generation to dataset management through its Generate Similar Images action. It takes an image from a dataset, writes descriptions of new scenes inspired by it, and generates one image per description with the Ultralytics Image 4B model by default, from 1 to 16 at a time, following optional instructions on what to vary, such as lighting, viewpoint, background, or objects. Kept images join the dataset unlabeled, ready for annotation.
Synthetic Data Generation vs. Related Terms#
- Data augmentation: Transforms existing real samples during training with flips, crops, or color shifts, so each output stays tied to one original.
- Generative AI: The broader field of models that create content. Generative models are one source of synthetic data, alongside simulators and procedural rules.
- Synthetic data: The dataset that generation produces, judged by its fidelity to real data, its usefulness for training, and the privacy it preserves.










