Video Understanding

Video understanding is the field of computer vision that interprets what happens in a video over time: which objects and people are present, what they do, when events start and end, and how one event leads to another. Image recognition analyzes each frame in isolation, while video understanding models motion and temporal order, which many questions depend on. A single frame shows a person halfway into a chair, and only the frames around it show whether they are sitting down or standing up. The field ranges from labeling short clips to answering open-ended questions about hours of footage.

Core Tasks#

• Action recognition: Assigns one label, such as "push-up" or "falling", to a short trimmed clip.
• Temporal action localization: Finds when each action starts and ends in a long, untrimmed video, and spatio-temporal action detection also boxes each actor in each frame.
• Object tracking: Follows each object across frames with a persistent ID, the basis for counting, speed measurement, and trajectory analysis.
• Video segmentation: Produces pixel masks for objects or scene regions that stay consistent from frame to frame.
• Video captioning and question answering: Describes a clip in text or answers questions such as "what did the driver do before the car stopped?"
• Temporal grounding: Finds the moment that matches a sentence, such as "the goalkeeper catches the ball", a task also called moment retrieval.
• Anomaly detection: Flags unusual events, such as a fall, a fight, or a vehicle driving the wrong way.

How Video Models Work#

Video models must capture appearance, which objects look like, and motion, how they change. Two-stream networks processed RGB frames and optical flow in separate branches; 3D convolutional neural networks (CNNs) convolve across height, width, and time together; and the Temporal Shift Module gives 2D CNNs temporal modeling at zero extra computation by shifting part of each layer's channels to neighboring frames. Video transformers apply self-attention across space-time patches, and TimeSformer made this affordable by attending over time and space in separate steps. Self-supervised pretraining on unlabeled video produces strong general-purpose encoders: Meta's V-JEPA 2 learned from over 1 million hours of internet video by predicting the representations of masked regions, and reached 77.3% top-1 accuracy on Something-Something v2, a benchmark of hand-object actions that differ mainly in their motion. Video language models sample frames, encode each into visual tokens, and pass them to a large language model (LLM) that answers questions in text.

The output a system needs largely decides which of these model families it uses:

Events from Object Tracks#

Many practical systems understand video by reasoning over tracks: a detector finds objects in each frame, a tracker links them into trajectories, and rules or models turn trajectories into events such as entering a zone, crossing a line, or stopping too long. The Ultralytics object counting solution works this way with YOLO26 tracking, counting each tracked object once when its path crosses a line and recording the direction from its positions in consecutive frames:

import cv2
from ultralytics import solutions

# Count people (COCO class 0) crossing a horizontal line through the middle of the 640x360 frame
counter = solutions.ObjectCounter(model="yolo26n.pt", region=[(0, 180), (640, 180)], classes=[0])

cap = cv2.VideoCapture("https://github.com/ultralytics/assets/releases/download/v0.0.0/solutions_ci_demo.mp4")
while True:
    success, frame = cap.read()
    if not success:
        break
    results = counter(frame)

# A crossing and its direction come from each person's track, so they exist only across frames
print(f"Crossed moving down: {results.in_count}, moving up: {results.out_count}")

The two counts describe events that no single frame contains, since telling a person walking down from one walking up takes at least two positions of the same tracked person.

Video Understanding vs. Related Terms#

• Image recognition: Classifies or detects content in single images. Running it on every frame of a video yields per-frame results with no notion of identity, motion, or order.
• Multi-object tracking (MOT): One building block of video understanding that keeps identities across frames without interpreting what the objects are doing.
• Video generation: Creates new video from text or images, the reverse direction, though both rely on models of how scenes change over time.
• Multimodal AI: Combines video with audio, subtitles, or other data, which helps when the soundtrack or speech carries the key information.

Applications#

• Sports analytics: Tracking players and the ball to measure speed, possession, and formations, and finding highlights in full-match recordings.
• Traffic and smart cities: Counting vehicles and pedestrians, measuring speeds, and detecting wrong-way driving or stalled vehicles.
• Retail and safety monitoring: Measuring queue lengths, flagging falls in care homes, and detecting people entering restricted zones.
• Media and content moderation: Indexing large video libraries for search and flagging policy-violating uploads.
• Robotics and autonomous vehicles: Predicting where pedestrians and other vehicles will move next.

Challenges#

Video multiplies the cost of every image: a minute at 30 frames per second contains 1,800 frames, so models sample sparsely, often 8 to 64 frames per clip, and still risk missing brief events. Long videos strain memory and context length, and labeling actions with precise start and end times takes far longer than labeling images. Google's Gemini API samples video at one frame per second and uses about 100 tokens per second of video at its default low media resolution, so its models with a one-million-token context window can process up to three hours of video, or one hour at high resolution.