Data Cleaning

Data cleaning, also called data cleansing or data scrubbing, is the process of finding and fixing or removing corrupt, duplicate, incomplete, inconsistent, and mislabeled records in a dataset. In machine learning (ML), it comes before data preprocessing and training, because a model learns whatever its training data contains, errors included. A detector trained on images where half the bicycles are labeled "motorcycle" learns that confusion, and a test set with wrong labels misreports how good the model is. Cleaning is a core practice of data-centric AI, which improves models by improving their data rather than their architecture.

Why Data Quality Limits Models#

Errors in data cost accuracy in several ways. Mislabeled examples teach wrong input-output mappings and push a model toward overfitting as it memorizes noise. Duplicates give some examples extra weight and, when copies land in both the training and evaluation splits, cause data leakage that inflates scores. Corrupt files crash data loaders or are silently skipped, and inconsistent class names split one category into several. Benchmarks are affected too: a study of 10 widely used test sets estimated label errors in at least 3.3% of examples on average and at least 6% of the ImageNet validation set. Scored on corrected labels, ResNet-18 outperforms ResNet-50 once the share of originally mislabeled test examples rises by just 6%.

Common Data Cleaning Tasks#

Tabular data is usually cleaned with libraries such as pandas, while vision datasets need checks on image files and their annotations, from the pixels to the bounding box coordinates. Near-duplicate images are found by comparing embeddings, with vector search scaling the comparison to millions of images.

• Missing values: Records with empty fields are dropped or filled by imputation from column statistics or nearest neighbors. In vision datasets, an image without a label file is ambiguous, since it may be a deliberate background image or one nobody annotated.
• Duplicates and near-duplicates: Byte-identical files share a hash. Resized or recompressed copies and consecutive video frames do not, so they need perceptual hashes or embedding similarity.
• Corrupt and invalid files: Truncated downloads, unsupported formats, unexpected color modes, and tiny images fail to decode or carry no usable signal.
• Label errors: Wrong classes, missing objects, and loose or misplaced boxes are the costliest errors in supervised vision data and the hardest to find by eye.
• Inconsistent categories: Spellings such as "Car" and "car", or overlapping classes such as "automobile" and "vehicle", are merged into one taxonomy so each concept has one label.
• Outliers: Impossible values, such as negative ages or boxes extending past the image border, are errors to fix. Rare but valid cases are kept, because those edge cases are often what the model most needs to learn.

Finding Label Errors With a Model#

Reviewing every label by hand does not scale, so cleaning pipelines use a trained model to rank which labels to inspect. Confident learning, implemented in the open-source cleanlab library, takes out-of-sample predicted probabilities from cross-validation and flags examples where the model confidently predicts a different class than the given label. People still make the final call: in the test-set study above, crowdsourced reviewers confirmed 51% of the algorithmically flagged candidates as real label errors.

Automating File Checks#

File-level checks are cheap enough to run on every new batch of data, alongside the label and class-balance audits covered in the Dataset Quality Control lesson. The following script builds a small folder with one image, an exact copy, and a truncated file, then flags the copy and the corrupt file using Pillow and a content hash:

import hashlib
from pathlib import Path

from PIL import Image

# A tiny image folder: one original, one exact copy, and one truncated download
folder = Path("images")
folder.mkdir(exist_ok=True)
Image.new("RGB", (640, 480), "orange").save(folder / "a.jpg")
(folder / "b.jpg").write_bytes((folder / "a.jpg").read_bytes())
(folder / "c.jpg").write_bytes((folder / "a.jpg").read_bytes()[:1000])

seen = {}
for path in sorted(folder.glob("*.jpg")):
    try:
        Image.open(path).load()  # decode every pixel; a truncated file raises OSError
    except OSError as error:
        print(f"{path.name}: corrupt ({error})")
        continue
    digest = hashlib.md5(path.read_bytes()).hexdigest()  # identical bytes give identical hashes
    if digest in seen:
        print(f"{path.name}: duplicate of {seen[digest]}")
    seen.setdefault(digest, path.name)

The script prints b.jpg: duplicate of a.jpg and c.jpg: corrupt (image file is truncated (5 bytes not processed)). Fully decoding each image matters, because opening a file reads only its header and a truncated file can pass that step. Ultralytics YOLO runs related checks whenever it builds a dataset's label cache: it re-saves JPEGs that lack an end-of-image marker, removes duplicate label rows, and skips image-label pairs with images under 10 pixels, coordinates outside the image, or class ids beyond those in the dataset YAML, then reports image, background, and corrupt counts in its scan summary.

Data Cleaning vs. Related Terms#

• Data preprocessing: Transforms data that is already correct into the form a model expects, such as resizing images and normalizing pixel values. Cleaning decides which data is correct.
• Data augmentation: Adds deliberate variation, such as flips and color shifts, to clean data during training.
• Data labeling: Creates labels for unlabeled data, while cleaning audits and corrects labels that already exist.
• Dataset bias: A dataset can be free of errors and still underrepresent some conditions or groups. Fixing bias means collecting different data, which cleaning alone cannot do.