Action Recognition

Action recognition is a computer vision task that classifies what a person, or sometimes an animal or robot, is doing in a video: walking, waving, falling, swinging a golf club, or doing a push-up. It extends image classification along time, because many actions cannot be told apart in a single frame. A still image of someone halfway down may show sitting or standing up, while a short clip shows which one is happening. The term is often used interchangeably with human activity recognition (HAR), although HAR also covers recognition from wearable sensors such as accelerometers.

How Action Recognition Works#

Models take a clip, typically 8 to 64 frames sampled from a few seconds of video, and learn two kinds of information: spatial features describing what the people and objects look like, and temporal features describing how they move. Early two-stream networks processed RGB frames and optical flow in separate branches and fused their predictions. 3D convolutional neural networks convolve across height, width, and time at once; I3D built them by inflating 2D image filters pretrained on ImageNet into 3D, and SlowFast pairs a low-frame-rate pathway for appearance with a high-frame-rate pathway for motion. Video transformers apply self-attention across space-time patches, and VideoMAE pretrains them by reconstructing heavily masked video.

A 3D CNN from the TorchVision video models classifies a clip in a few lines:

import cv2
import numpy as np
import torch
from torchvision.models.video import R3D_18_Weights, r3d_18

# Load a 3D CNN pretrained on the 400 action classes of Kinetics-400
weights = R3D_18_Weights.KINETICS400_V1
model = r3d_18(weights=weights).eval()

# Read 16 consecutive frames from a short exercise video
cap = cv2.VideoCapture("https://github.com/ultralytics/assets/releases/download/v0.0.0/solution_ci_pose_demo.mp4")
frames = [cv2.cvtColor(cap.read()[1], cv2.COLOR_BGR2RGB) for _ in range(16)]

# Stack the frames into one clip shaped (time, channels, height, width) and classify it
clip = torch.from_numpy(np.stack(frames)).permute(0, 3, 1, 2)
batch = weights.transforms()(clip).unsqueeze(0)
with torch.no_grad():
    probs = model(batch).softmax(dim=1)[0]

top = probs.argmax().item()
print(f"{weights.meta['categories'][top]}: {probs[top]:.2f}")

The model prints push up: 0.99: it labels the whole half-second clip with one action from its 400 classes.

Skeleton-based methods take a different route. A pose estimation model first finds body keypoints in every frame, and a temporal model such as the graph network ST-GCN classifies the resulting joint trajectories. Skeletons ignore clothing, lighting, and background, and they are small enough to process in real time on edge devices. Ultralytics YOLO26 pose models supply the 17 body keypoints per person for this route, and the workouts monitoring solution turns the angle between three of those keypoints into repetition counts for exercises such as push-ups and squats.

Action Recognition vs. Related Tasks#

• Pose estimation and object tracking: Pose estimation locates joints and tracking keeps each person's identity across frames. Both are often inputs to action recognition, which interprets the movement they describe; in a crowd, each tracked person gets their own clip and label.
• Temporal action localization: Finds when each action starts and ends in a long, untrimmed video. Spatio-temporal action detection goes further and boxes each actor in each frame, as in the AVA dataset.
• Video understanding: The broader field that also covers captioning, question answering, and reasoning about events and causes in video.

Real-World Applications#

• Healthcare and elder care: Fall detection systems recognize a fall from camera or wearable data and alert staff, and rehabilitation apps check that patients perform exercises correctly.
• Fitness and sports: Apps count repetitions and score form, and sports analytics classify shots, passes, and strokes in broadcast footage.
• Safety and security: Monitoring systems flag fights in public spaces and falls or unsafe lifting on factory floors.
• Human-robot interaction: Robots and smart devices respond to gestures such as waving or pointing.

Datasets and Challenges#

Labeling video takes far more effort than labeling images, so progress has followed large datasets. Kinetics-400 contains at least 400 YouTube clips of about 10 seconds for each of its 400 human action classes, and UCF101 has 13,320 clips in 101 classes. Something-Something V2 targets scene bias, the tendency of models to recognize swimming from the pool instead of the strokes: its roughly 220,000 clips show hands interacting with everyday objects in 174 classes that differ only in motion, such as "pushing something from left to right" and "pushing something from right to left".