Pose Estimation

Pose estimation is the computer vision task of locating a predefined set of keypoints on a body or object, such as the shoulders, elbows, and knees of a person, and connecting them into a skeleton that describes its posture. A model outputs the image coordinates of each keypoint, or its position in 3D, together with a confidence score, so applications can measure joint angles, count repetitions, or recognize actions. Pose estimation describes how each person is standing or moving, which neither a bounding box from object detection nor a mask from instance segmentation captures.

How Pose Estimation Works#

Multi-person 2D pose estimators follow one of three strategies, and the third mirrors one-stage object detectors:

• Top-down: A person detector finds everyone in the image, and a single-person model predicts the keypoints inside each crop, as in HRNet (2019) and ViTPose (2022). Accuracy is high for large, clearly visible people, and the cost grows with the number of people.
• Bottom-up: The network finds every keypoint in the image at once, then groups them into people. OpenPose (2017) links joints with part affinity fields, 2D vectors that point along each limb, so its cost stays nearly flat in crowds.
• Single-stage: The network predicts a person box and all of its keypoints from the same grid location in one pass, which needs neither crops nor a grouping step. YOLO-Pose (2022) brought this design to YOLO.

Keypoint locations come either from heatmaps, one probability map per keypoint whose peak marks the location, or from regressing the coordinates directly. Heatmaps led benchmarks for years but cost memory and are limited by the map's resolution. Residual Log-likelihood Estimation (RLE) narrowed the gap for regression in 2021 by learning the distribution of the prediction error with a normalizing flow. YOLO26 pose models are single-stage regressors trained with RLE: during training, an extra branch predicts each keypoint's uncertainty and a RealNVP flow scores the error, and both are removed when the model is fused for inference, so they add no inference cost.

from ultralytics import YOLO

# Load a YOLO26 nano pose model trained on the 17 COCO body keypoints
model = YOLO("yolo26n-pose.pt")

# Find every person and their keypoints in one pass
results = model("https://ultralytics.com/images/bus.jpg")
result = results[0]

# Shape is (people, keypoints, xy); save the image with the skeletons drawn
print(result.keypoints.xy.shape)
result.save(filename="pose_result.jpg")

The printed shape counts the people found by 17 keypoints by 2 pixel coordinates, and the saved image draws each skeleton over its person.

2D and 3D Pose Estimation#

2D estimators return pixel coordinates. 3D human pose estimation recovers joint positions in space by lifting a sequence of 2D keypoints with a second network, as in VideoPose3D, by triangulating keypoints seen from several calibrated cameras, or by fitting a parametric body mesh such as SMPL to the image. Object pose estimation is a separate task: it recovers the 3D rotation and translation of a rigid object, often from keypoints on its surface, for robot grasping and augmented reality.

Evaluating Pose Estimation#

The COCO Keypoints benchmark, with about 250,000 people labeled with 17 keypoints, scores predictions with object keypoint similarity (OKS), the keypoint counterpart of intersection over union (IoU). For each labeled keypoint, the distance d between prediction and label becomes a similarity exp(-d²/(2s²k²)), where s² is the person's segmented area and k is a per-keypoint constant measured from how consistently annotators placed that keypoint. OKS averages these terms over the labeled keypoints, and mean average precision (mAP) is averaged over OKS thresholds from 0.50 to 0.95.

import numpy as np

# COCO falloff constants: annotators place eyes far more consistently than hips
sigmas = {"eye": 0.025, "hip": 0.107}
area = 20_000  # person's segmented area in pixels (s squared)
error = 5.0  # predicted keypoint lies 5 pixels from its label

# Per-keypoint OKS term: exp(-d^2 / (2 s^2 k^2)) with k = 2 * sigma
for name, sigma in sigmas.items():
    similarity = np.exp(-(error**2) / (2 * area * (2 * sigma) ** 2))
    print(f"{name}: {similarity:.2f}")

The same 5-pixel miss scores 0.78 on an eye and 0.99 on a hip. Other benchmarks use other measures: MPII reports PCKh, the share of keypoints within half the head size of their label, and 3D benchmarks such as Human3.6M report mean per-joint position error in millimeters.

Real-World Applications#

• Sports analysis: Coaches measure joint angles, stride length, and swing mechanics from ordinary video, without the reflective markers of laboratory motion capture.
• Rehabilitation: Physical therapy apps track range of motion and exercise form at home, and fall-detection systems watch for a body's keypoints dropping to floor level.
• Action recognition: Sequences of skeletons are compact inputs for classifying actions such as waving, falling, or lifting, and they ignore clothing and background.
• Animal research: Custom keypoint sets track animal posture, such as the 24 keypoints per dog in the Dog-Pose dataset, to study gait and behavior.