License YOLO
Back to Ultralytics Glossary

Video Generation

Video generation is the synthesis of new video clips from text, images, or existing footage by generative models that keep motion consistent over time.

Video generation is the use of generative AI models to synthesize new video: a sequence of frames whose objects, lighting, and motion stay consistent from one frame to the next. A model can start from a text prompt, a still image to animate, an existing clip to restyle or extend, or past frames plus actions, as in world models. Compared with a single generated image, a five-second clip at 24 frames per second holds 120 images that must agree with each other, and errors that would be invisible in one frame, such as a hand changing its number of fingers, show up as flicker or morphing when the frames play. Video understanding and object tracking work in the other direction, analyzing footage that already exists.

The field grew from tiny clips to minute-long ones in under a decade. Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba's 2016 VGAN, a generative adversarial network, produced about a second of 64×64 video by generating a static background and a moving foreground separately. Jonathan Ho and colleagues at Google extended diffusion models to video in April 2022, Meta's Make-A-Video and Google's Imagen Video followed that fall, and OpenAI's Sora, previewed in February 2024, generated clips up to a minute long. Open-weight models such as Tencent's HunyuanVideo (December 2024) and Alibaba's Wan 2.1 (February 2025) made the same architecture available to run and fine-tune locally.

How Video Generation Models Work#

Most current systems are latent diffusion models with three parts. A video autoencoder compresses the clip in both space and time into a much smaller latent tensor, because denoising raw pixels for hundreds of frames would be far too expensive. A denoising network, today usually a diffusion transformer (DiT), cuts that latent into spacetime patches, the units OpenAI's Sora operates on, and removes noise step by step, with attention linking every patch to patches in other frames so a subject keeps its shape as it moves. The autoencoder's decoder then turns the clean latent back into frames.

Earlier models took cheaper routes to temporal consistency. Many inserted temporal attention layers, which compare the same location across frames, into a pretrained text-to-image U-Net, so the network reused what it knew about appearance and learned only motion. Imagen Video chained a base model with separate spatial and temporal super-resolution models that raised resolution and frame rate in stages. Google's Lumiere (2024) generated the whole clip in one pass with a space-time U-Net, arguing that cascades that first generate distant keyframes struggle to keep motion globally coherent. Long or interactive video is often generated autoregressively, chunk by chunk, with each chunk conditioned on the frames before it, a setting that methods such as diffusion forcing train for.

Types of Video Generation#

  • Text-to-video: Generates a clip from a written prompt alone.
  • Image-to-video: Animates a still image, keeping it as the first frame, which gives control over composition and subject.
  • Video-to-video: Restyles, edits, or extends an existing clip while keeping its motion, a core tool in AI video editing.
  • Audio and video together: Google's Veo 3 (May 2025) and OpenAI's Sora 2 (September 2025) generate dialogue, sound effects, and music in sync with the picture.
  • World models: Predict the next frames from the current ones plus an action, such as a steering command or a robot arm movement, so the video responds to control.

Applications#

Film and advertising teams use generators such as Google DeepMind's Veo and Meta Movie Gen for previsualization and for shots that would be costly to film, part of a wider shift toward AI in filmmaking. Physical AI developers generate driving and robotics footage of rare events: NVIDIA's Cosmos world foundation models, released in January 2025, produce video for training autonomous vehicles and robots, and Wayve's GAIA-1 generates driving scenes conditioned on text and vehicle actions. Generated clips become synthetic data for perception models once their frames are labeled, and an Ultralytics Platform dataset that receives a generated clip samples it at 1 frame per second, up to 100 frames, and stores each frame as an image ready for annotation.

Limits and Evaluation#

Generated video still breaks physics and object permanence. OpenAI has stated that Sora does not accurately model interactions such as glass shattering, and generated objects can pass through each other, appear from nowhere, or vanish behind an occluder. Computation grows with resolution and duration, which is why most models generate clips of a few seconds and extend them in stages. Evaluation moved from a single distribution score, Fréchet Video Distance, the video counterpart of Fréchet Inception Distance (FID), to suites such as VBench, which scores 16 dimensions from subject consistency to prompt faithfulness. Realistic output also enables deepfakes, so providers mark generated video with AI watermarking and Content Credentials: Google embeds an invisible SynthID watermark in every Veo video.

Explore solutions

Computer vision in sports

Computer vision in sports

Discover how Ultralytics YOLO powers sports vision with player and ball tracking, performance analytics, coaching insights, and athlete rehabilitation.
Learn more
Computer vision for aerial imagery

Computer vision for aerial imagery

Turn drone and aerial imagery into real-time insights for agriculture, aquaculture, conservation, and geospatial analysis with Ultralytics YOLO.
Learn more
Computer vision in aerospace

Computer vision in aerospace

Apply computer vision in aerospace with Ultralytics YOLO for aircraft inspection, runway FOD detection, airport ground operations, and composite defect detection.
Learn more
Computer vision in security

Computer vision in security

Protect every site with Ultralytics YOLO models. Vision AI powers perimeter monitoring, threat detection, license-plate recognition, and workplace safety.
Learn more
Computer vision in robotics

Computer vision in robotics

Power smarter machines with Ultralytics YOLO models. Vision AI in robotics drives autonomous navigation, perception, object tracking, and real-time control.
Learn more
Computer vision in logistics

Computer vision in logistics

Streamline logistics with Ultralytics YOLO models. Vision AI enables package inspection, sorting, vehicle tracking, and real-time warehouse safety monitoring.
Learn more
Computer vision in retail

Computer vision in retail

Reimagine retail with Ultralytics YOLO models. Vision AI powers inventory tracking, shelf monitoring, queue management, and smarter customer insights.
Learn more
Computer vision in healthcare

Computer vision in healthcare

Build healthcare solutions with Ultralytics YOLO models. Vision AI in healthcare powers faster medical imaging, smarter diagnostics, and patient monitoring.
Learn more
Computer vision in manufacturing

Computer vision in manufacturing

Optimize manufacturing with Ultralytics YOLO models. Vision AI drives quality control, defect detection, PPE compliance, and assembly line automation.
Learn more
Computer vision in automotive

Computer vision in automotive

Apply computer vision in automotive with Ultralytics YOLO models. Vision AI elevates road safety, driver assistance, and vehicle automation for smarter roads.
Learn more
Computer vision in agriculture

Computer vision in agriculture

Bring vision AI to smart agriculture with Ultralytics YOLO models. Power crop monitoring, livestock tracking, and precision farming for higher, smarter yields.
Learn more

Let's build the future of AI together!

Begin your journey with the future of machine learning