Text-to-Video

Text-to-video is the generation of a video clip from a natural-language prompt, such as "a red fox trotting through fresh snow at dawn, low tracking shot." It is the branch of video generation whose only required input is text, and it extends text-to-image generation with a time axis: the model must render the objects, style, and camera work the prompt names and also invent motion that the prompt only implies, such as how a fox's legs move at a trot. Early research models in 2022 included Tsinghua University's CogVideo, a 9.4-billion-parameter transformer, and Meta's Make-A-Video, which learned what things look like from text-image pairs and how they move from unlabeled video, so it needed no paired text-video data. OpenAI's Sora, previewed in February 2024, made minute-long, high-resolution clips, Meta Movie Gen followed in October 2024, and Google DeepMind's Veo family reached Veo 3 in May 2025, which, like OpenAI's Sora 2, generates synchronized dialogue and sound with the picture.

How Text Conditions a Video Model#

A frozen language model, such as a T5 encoder, turns the prompt into a sequence of token embeddings. The video model, usually a diffusion transformer working on a compressed latent video, reads those embeddings through cross-attention at every denoising step, so each spacetime patch of the clip can attend to the words that describe it. Generation starts from random noise and removes it step by step, as in other diffusion models, until a decoder turns the clean latent into frames.

Prompt adherence is tuned with classifier-free guidance, introduced by Jonathan Ho and Tim Salimans in 2022. During training, the prompt is dropped for a fraction of examples, so one network learns both conditional and unconditional predictions; at each sampling step, the model computes both and pushes the result away from the unconditional one by a guidance scale. Higher scales follow the prompt more literally at the cost of variety and natural color.

The text a model learns from matters as much as its architecture. Most training videos come with no description or a short, noisy one, so OpenAI re-captioned Sora's training clips with a dedicated captioning model that writes detailed descriptions, and it uses GPT to expand short user prompts into the same detailed style at generation time. This prompt enrichment closes the gap between how people write prompts and the captions the model was trained on.

Uses#

• Film previsualization: Directors and game designers turn a written scene into a rough clip to test camera angles, lighting, and pacing before a shoot.
• Marketing and advertising: Brands generate product shots and social clips in many variations from one script.
• Education and explainers: Teachers and publishers illustrate processes that are hard to film, such as historical scenes or cell division.
• Synthetic data: Prompts describing rare events, such as a deer crossing a highway at night, produce footage for training perception models once the frames are labeled.

Failure Modes and Evaluation#

Text-to-video models fail most often on the parts of a prompt that need precise binding. Counts drift ("three dogs" becomes two or four), attributes swap between objects ("a red car next to a blue truck"), spatial relations flip, and written text in the scene comes out garbled. Over time, subjects morph into other shapes, appear from nowhere, or vanish behind occluders, and physical interactions break: OpenAI has stated that Sora does not accurately model glass shattering. Such errors are a video form of hallucination.

VBench scores these failures separately, with dimensions for object class, multiple objects, color, spatial relationship, human action, and subject consistency, and the Fréchet Inception Distance (FID) family of distribution metrics measures overall realism. Object detection is the workhorse of the content checks: running YOLO26 in track mode over a generated clip gives every detected object a persistent ID, so a clip prompted with two dogs that accumulates five dog IDs has subjects that vanished and reappeared or changed form mid-shot.

Because generated clips can show real people doing things they never did, text-to-video raises the same concerns as deepfakes and AI ethics more broadly. Providers block prompts that name public figures, add visible watermarks and AI watermarking signals, and attach Content Credentials; OpenAI's Sora 2 videos carry both a visible moving watermark and C2PA metadata.