Data Labeling

Data labeling is the process of attaching ground-truth information to raw data, such as a class name for an image, a bounding box around each car in a street photo, or a transcript for an audio clip, so that supervised learning models can learn from it. The labels define the task: a model learns to predict exactly what the labels describe, as precisely and consistently as they describe it. Data annotation is a synonym, and the people who do the work are called labelers or annotators. Labeling is usually the slowest and most expensive step in building training data, which is why many workflows have a model propose labels and people correct them.

Label Types in Computer Vision#

The label format follows the task the model must perform:

• Image classification: One class per image, such as "defective" or "ok" for a part on an inspection line.
• Object detection: An axis-aligned box and class for every object of interest.
• Oriented bounding boxes: Rotated boxes that fit elongated or angled objects, such as ships and vehicles in aerial images.
• Instance segmentation: A polygon or pixel mask outlining each object separately.
• Semantic segmentation: A class for every pixel, such as road, sidewalk, and sky, without separating individual objects.
• Pose estimation: Keypoints at fixed landmarks, such as COCO's 17 points from the nose and eyes down to the ankles, often with a visibility flag for each.

Effort grows with detail. Drawing one high-quality box for ILSVRC, the ImageNet challenge dataset, took about 35 seconds, and extreme clicking, where annotators click the top, bottom, left-most, and right-most points of an object instead of dragging corners, cut that to 7 seconds with boxes of the same quality. Polygons and pixel masks take longer still. Outside vision, labels mark entity spans for named entity recognition (NER), sentiment for text classification, and transcripts for speech recognition.

The Labeling Workflow#

Labeling projects start with written guidelines that define every class and settle the edge cases before work begins: whether to box a car that is 90 percent hidden, how tightly to draw around a cyclist's limbs, and what to do with reflections, tiny objects, and crowds. In model-assisted labeling, a model trained on the labels so far pre-labels new images, annotators correct its mistakes, reviewers check a sample, and each retrained model makes the next round of proposals more accurate. Active learning adds a selection step to this loop, so that people label the images the model is least sure about first.

The annotation editor in Ultralytics Platform builds this loop in: its Smart mode turns one click into a mask with Meta's Segment Anything models, or adds predictions from pretrained or your own fine-tuned YOLO models as draft annotations. The same pre-labeling step runs in Python with a YOLO26 detector, which writes its predictions in the label format used for training:

from pathlib import Path

from ultralytics import YOLO

# Load a pretrained YOLO26 nano detector to propose labels
model = YOLO("yolo26n.pt")

# Predict on an unlabeled image
results = model("https://ultralytics.com/images/bus.jpg")
result = results[0]

# Write one "class x_center y_center width height" row per object, normalized to 0-1
label_file = result.save_txt("bus.txt")

# Print the draft labels an annotator will review
print(Path(label_file).read_text())

Each printed row is a COCO class index followed by the box center, width, and height as fractions of the image size, the YOLO label format that training reads directly once an annotator has fixed any wrong or missing boxes.

Label Quality#

A model cannot be more consistent than its labels, and errors are common even in famous datasets. A study of label errors in 10 widely used benchmarks estimated that at least 3.3% of test labels are wrong on average, including at least 6% of the ImageNet validation set. Teams catch errors with several standard checks:

• Gold questions: Items with known answers mixed into each labeler's queue to measure their accuracy, often drawn from a golden dataset.
• Overlap labeling: Two or more annotators label the same items, and their agreement is scored with Cohen's kappa for class labels or with intersection over union (IoU) between their boxes and masks.
• Model-based auditing: A trained model flags labels it strongly disagrees with, and tools such as cleanlab rank the most likely label errors for human review.

Data Labeling vs. Related Terms#

• Data annotation: A synonym. Computer vision teams tend to say annotation for drawn geometry such as boxes and masks.
• Semi-supervised learning: A model labels unlabeled data itself, keeping its own confident predictions as pseudo-labels without human review.
• Active learning: Decides which items people label next, typically the ones the current model finds most uncertain.
• Data cleaning: Fixes problems in a dataset, such as duplicates, corrupt files, and inconsistent formats, including wrong labels found after labeling.
• Synthetic data: Rendered or generated images whose labels come free from the generator, because the scene layout is known exactly, at the cost of a gap between synthetic and real images.