Video Generation
Video generation is the synthesis of new video clips from text, images, or existing footage by generative models that keep motion consistent over time.
Video generation is the use of generative AI models to synthesize new video: a sequence of frames whose objects, lighting, and motion stay consistent from one frame to the next. A model can start from a text prompt, a still image to animate, an existing clip to restyle or extend, or past frames plus actions, as in world models. Compared with a single generated image, a five-second clip at 24 frames per second holds 120 images that must agree with each other, and errors that would be invisible in one frame, such as a hand changing its number of fingers, show up as flicker or morphing when the frames play. Video understanding and object tracking work in the other direction, analyzing footage that already exists.
The field grew from tiny clips to minute-long ones in under a decade. Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba's 2016 VGAN, a generative adversarial network, produced about a second of 64×64 video by generating a static background and a moving foreground separately. Jonathan Ho and colleagues at Google extended diffusion models to video in April 2022, Meta's Make-A-Video and Google's Imagen Video followed that fall, and OpenAI's Sora, previewed in February 2024, generated clips up to a minute long. Open-weight models such as Tencent's HunyuanVideo (December 2024) and Alibaba's Wan 2.1 (February 2025) made the same architecture available to run and fine-tune locally.
How Video Generation Models Work#
Most current systems are latent diffusion models with three parts. A video autoencoder compresses the clip in both space and time into a much smaller latent tensor, because denoising raw pixels for hundreds of frames would be far too expensive. A denoising network, today usually a diffusion transformer (DiT), cuts that latent into spacetime patches, the units OpenAI's Sora operates on, and removes noise step by step, with attention linking every patch to patches in other frames so a subject keeps its shape as it moves. The autoencoder's decoder then turns the clean latent back into frames.
Earlier models took cheaper routes to temporal consistency. Many inserted temporal attention layers, which compare the same location across frames, into a pretrained text-to-image U-Net, so the network reused what it knew about appearance and learned only motion. Imagen Video chained a base model with separate spatial and temporal super-resolution models that raised resolution and frame rate in stages. Google's Lumiere (2024) generated the whole clip in one pass with a space-time U-Net, arguing that cascades that first generate distant keyframes struggle to keep motion globally coherent. Long or interactive video is often generated autoregressively, chunk by chunk, with each chunk conditioned on the frames before it, a setting that methods such as diffusion forcing train for.
Types of Video Generation#
- Text-to-video: Generates a clip from a written prompt alone.
- Image-to-video: Animates a still image, keeping it as the first frame, which gives control over composition and subject.
- Video-to-video: Restyles, edits, or extends an existing clip while keeping its motion, a core tool in AI video editing.
- Audio and video together: Google's Veo 3 (May 2025) and OpenAI's Sora 2 (September 2025) generate dialogue, sound effects, and music in sync with the picture.
- World models: Predict the next frames from the current ones plus an action, such as a steering command or a robot arm movement, so the video responds to control.
Applications#
Film and advertising teams use generators such as Google DeepMind's Veo and Meta Movie Gen for previsualization and for shots that would be costly to film, part of a wider shift toward AI in filmmaking. Physical AI developers generate driving and robotics footage of rare events: NVIDIA's Cosmos world foundation models, released in January 2025, produce video for training autonomous vehicles and robots, and Wayve's GAIA-1 generates driving scenes conditioned on text and vehicle actions. Generated clips become synthetic data for perception models once their frames are labeled, and an Ultralytics Platform dataset that receives a generated clip samples it at 1 frame per second, up to 100 frames, and stores each frame as an image ready for annotation.
Limits and Evaluation#
Generated video still breaks physics and object permanence. OpenAI has stated that Sora does not accurately model interactions such as glass shattering, and generated objects can pass through each other, appear from nowhere, or vanish behind an occluder. Computation grows with resolution and duration, which is why most models generate clips of a few seconds and extend them in stages. Evaluation moved from a single distribution score, Fréchet Video Distance, the video counterpart of Fréchet Inception Distance (FID), to suites such as VBench, which scores 16 dimensions from subject consistency to prompt faithfulness. Realistic output also enables deepfakes, so providers mark generated video with AI watermarking and Content Credentials: Google embeds an invisible SynthID watermark in every Veo video.










