License YOLO
Back to Ultralytics Glossary

Synthetic Data Generation

Synthetic data generation creates artificial training data with simulators, procedural rules, generative models, and LLMs, often with exact labels.

Synthetic data generation is the process of producing artificial images, video, sensor readings, tables, or text with a simulator, procedural rules, or a generative model, so that a machine learning model can train or be tested on it. Its output is synthetic data; the generation process decides that data's quality through four choices: which method creates the samples, how labels are attached, how widely each scene varies, and how the result is checked against real data. In computer vision, the main attraction is that a generator that places every object can also write the data annotation for it, exact and free.

Main Generation Methods#

Simulators build a 3D scene from assets, set its lighting, materials, and camera, and render images together with exact masks, depth maps, and bounding boxes, because the scene description records where everything is. Open-source driving simulators such as CARLA produce camera, LiDAR, and radar streams for autonomous vehicles, and robotics simulators do the same for grasping and navigation. Domain randomization varies textures, lighting, and poses across renders so that models trained on them transfer to real cameras.

Procedural generation uses scripts and rules instead of a full scene. Cut-and-paste compositing, popularized by the 2017 "Cut, Paste and Learn" paper from Carnegie Mellon University, pastes segmented object crops onto new backgrounds and records each pasted box as a label. The simplest version draws shapes at random positions, and the generator writes each object's label as it draws:

import cv2
import numpy as np

rng = np.random.default_rng(0)
boxes = 0
for i in range(100):
    # Random background color, then one to four contrasting discs in separate quadrants so none overlap
    background = rng.integers(0, 256, 3)
    image = np.full((320, 320, 3), background, dtype=np.uint8)
    labels = []
    for qx, qy in [(0, 0), (160, 0), (0, 160), (160, 160)][: rng.integers(1, 5)]:
        r = int(rng.integers(10, 40))
        x, y = qx + int(rng.integers(r, 160 - r)), qy + int(rng.integers(r, 160 - r))
        color = (background + rng.integers(64, 193, 3)) % 256
        cv2.circle(image, (x, y), r, color.tolist(), thickness=-1)
        # The generator placed the disc, so its normalized box label is exact and free
        labels.append(f"0 {x / 320:.4f} {y / 320:.4f} {2 * r / 320:.4f} {2 * r / 320:.4f}")
    cv2.imwrite(f"synthetic_{i:03d}.jpg", image)
    with open(f"synthetic_{i:03d}.txt", "w") as f:
        f.write("\n".join(labels))
    boxes += len(labels)
print(f"wrote 100 images with {boxes} exact box labels")

It prints wrote 100 images with 256 exact box labels. Each line follows the YOLO label format and stores the class and the box center, width, and height as fractions of the image size, and none of them needed a human annotator.

Generative models learn the look of real data and sample new examples from it. Generative adversarial networks (GANs) and diffusion models produce photorealistic images from noise, a text prompt, or a reference image, and conditioning methods such as ControlNet fix the layout so that object positions stay known. Without that conditioning, generated images need labeling afterward. For tables, statistical synthesizers fit a joint distribution to real records with Gaussian copulas, Bayesian networks, or conditional GANs such as CTGAN (NeurIPS 2019) and then sample new rows.

Large language models generate text data: instructions, dialogues, code exercises, and reasoning traces. Microsoft trained phi-1, a 1.3B-parameter code model, on 6B tokens of filtered web text plus 1B tokens of textbooks and exercises written by GPT-3.5, and it reached 50.6% pass@1 on the HumanEval benchmark.

Choosing and Validating a Method#

The method follows from what the data must contain, and every method ends in the same check against real data:

Validation trains a model on the generated data, alone or mixed with real data, and evaluates it only on held-out real test data, as in Ultralytics' benchmarks of NVIDIA Cosmos synthetic data for defect detection. If accuracy on real images drops as the synthetic share grows, the generator has a gap in realism or coverage, and image-level scores such as Fréchet Inception Distance (FID) help locate it. Generators trained on real data can also memorize it, so privacy-sensitive pipelines check generated records against the originals, and some add differential privacy to training.

Ultralytics Platform adds generation to dataset management through its Generate Similar Images action. It takes an image from a dataset, writes descriptions of new scenes inspired by it, and generates one image per description with the Ultralytics Image 4B model by default, from 1 to 16 at a time, following optional instructions on what to vary, such as lighting, viewpoint, background, or objects. Kept images join the dataset unlabeled, ready for annotation.

  • Data augmentation: Transforms existing real samples during training with flips, crops, or color shifts, so each output stays tied to one original.
  • Generative AI: The broader field of models that create content. Generative models are one source of synthetic data, alongside simulators and procedural rules.
  • Synthetic data: The dataset that generation produces, judged by its fidelity to real data, its usefulness for training, and the privacy it preserves.

Explore solutions

Computer vision in sports

Computer vision in sports

Discover how Ultralytics YOLO powers sports vision with player and ball tracking, performance analytics, coaching insights, and athlete rehabilitation.
Learn more
Computer vision for aerial imagery

Computer vision for aerial imagery

Turn drone and aerial imagery into real-time insights for agriculture, aquaculture, conservation, and geospatial analysis with Ultralytics YOLO.
Learn more
Computer vision in aerospace

Computer vision in aerospace

Apply computer vision in aerospace with Ultralytics YOLO for aircraft inspection, runway FOD detection, airport ground operations, and composite defect detection.
Learn more
Computer vision in security

Computer vision in security

Protect every site with Ultralytics YOLO models. Vision AI powers perimeter monitoring, threat detection, license-plate recognition, and workplace safety.
Learn more
Computer vision in robotics

Computer vision in robotics

Power smarter machines with Ultralytics YOLO models. Vision AI in robotics drives autonomous navigation, perception, object tracking, and real-time control.
Learn more
Computer vision in logistics

Computer vision in logistics

Streamline logistics with Ultralytics YOLO models. Vision AI enables package inspection, sorting, vehicle tracking, and real-time warehouse safety monitoring.
Learn more
Computer vision in retail

Computer vision in retail

Reimagine retail with Ultralytics YOLO models. Vision AI powers inventory tracking, shelf monitoring, queue management, and smarter customer insights.
Learn more
Computer vision in healthcare

Computer vision in healthcare

Build healthcare solutions with Ultralytics YOLO models. Vision AI in healthcare powers faster medical imaging, smarter diagnostics, and patient monitoring.
Learn more
Computer vision in manufacturing

Computer vision in manufacturing

Optimize manufacturing with Ultralytics YOLO models. Vision AI drives quality control, defect detection, PPE compliance, and assembly line automation.
Learn more
Computer vision in automotive

Computer vision in automotive

Apply computer vision in automotive with Ultralytics YOLO models. Vision AI elevates road safety, driver assistance, and vehicle automation for smarter roads.
Learn more
Computer vision in agriculture

Computer vision in agriculture

Bring vision AI to smart agriculture with Ultralytics YOLO models. Power crop monitoring, livestock tracking, and precision farming for higher, smarter yields.
Learn more

Let's build the future of AI together!

Begin your journey with the future of machine learning