Anomaly Detection

Anomaly detection is the task of identifying data points, events, or image regions that deviate from an established pattern of normal behavior. It is also called outlier detection, and in manufacturing vision it is often called defect detection or automated visual inspection. Anomalies are rare, varied, and often of a kind never seen before, so most methods learn what normal looks like from mostly normal data, assign each new input an anomaly score for how far it falls from that pattern, and flag scores above a threshold. Applications include spotting scratches and dents on production lines, fraudulent card transactions, network intrusions, failing machinery, and suspicious findings in medical image analysis.

Types of Anomalies#

• Point anomalies: A single value or image far from the rest, such as a transaction 50 times larger than a customer's usual spend.
• Contextual anomalies: Values that are normal in one context and abnormal in another. An outdoor temperature of 30 °C is ordinary in July in New York and alarming there in January.
• Collective anomalies: Sequences or groups that are abnormal together although each member looks normal, such as a burst of login attempts from many countries within one minute, each unremarkable alone.

How Anomaly Detection Works#

Methods differ in how they model normal data:

• Statistical rules: Flag values more than a set number of standard deviations from the mean, or outside a range set by the interquartile range.
• Distance and density: Flag points far from their nearest neighbors or in sparse regions, as the local outlier factor does.
• Isolation: An Isolation Forest splits the data with random cuts, and anomalies end up isolated after fewer cuts than normal points.
• One-class models: A one-class support vector machine (SVM) learns a boundary around the training data and flags points outside it.
• Reconstruction: An autoencoder trained only on normal data reconstructs normal inputs well and anomalies poorly, so the reconstruction error serves as the anomaly score.
• Feature-based visual methods: PatchCore stores patch features of defect-free images, extracted by an ImageNet-pretrained network, in a memory bank and scores each test patch by its distance to the nearest stored patch, reaching image-level AUROC of up to 99.6% on the MVTec AD benchmark.

scikit-learn distinguishes outlier detection, where the training data already contains some anomalies, from novelty detection, where the training data is clean and only new inputs are scored. The script below trains an Isolation Forest on readings from a healthy machine and scores four new ones:

import numpy as np
from sklearn.ensemble import IsolationForest

# 1,000 readings from a healthy machine: temperature (C) and vibration (mm/s)
rng = np.random.default_rng(0)
normal = rng.normal([60, 2.0], [3, 0.3], size=(1000, 2))

# Learn what normal looks like, then score new readings
detector = IsolationForest(random_state=0).fit(normal)
new = np.array([[61, 2.1], [59, 1.8], [75, 2.0], [60, 4.5]])
labels = detector.predict(new)  # 1 = normal, -1 = anomaly
scores = detector.decision_function(new)  # below 0 = anomaly

for (temp, vibration), label, score in zip(new, labels, scores):
    status = "anomaly" if label == -1 else "normal"
    print(f"{temp:.0f} C, {vibration:.1f} mm/s -> {status} (score {score:+.3f})")

The two readings near the training mean score +0.080 and +0.076 and pass. The overheating reading at 75 °C scores −0.139 and the reading with more than twice the usual vibration scores −0.152, so both are flagged, although the model never saw an example of either fault.

Visual Anomaly Detection#

Visual inspection follows one of two routes, depending on whether defects can be collected and labeled. When the defect types are known and examples exist, object detection or instance segmentation models trained on classes such as scratch, dent, and crack locate and name each defect. A YOLO26 detection or segmentation model fine-tuned on such classes returns a box or mask, a class, and a confidence score for every defect it finds in an image. When defects are rare or unpredictable, one-class methods train only on images of good parts and flag anything that differs, which also catches defect types no one anticipated. The MVTec AD dataset, with more than 5,000 images in 15 object and texture categories and only defect-free images in its training sets, is a standard benchmark for this setting, and Ultralytics Platform also hosts it as a detection dataset for the supervised route. Open-source libraries such as Anomalib implement PatchCore and similar methods.

Anomaly detectors are evaluated with threshold-free metrics such as the area under the ROC curve (AUC), and with precision and recall at the chosen threshold, which reflect the false alarms and misses operators will actually see. Because anomalies are rare, the threshold is often set from scores on normal validation images, such as the score that only 1% of good parts exceed.

Anomaly Detection vs. Related Terms#

• Object detection: Finds instances of a fixed set of labeled classes. An object detector trained on defects finds only the defect types it was trained on.
• Data drift: A shift in the distribution of a whole stream of inputs over time, such as a week of darker images. Anomaly detection scores individual inputs.
• Out-of-distribution (OOD) detection: Flags inputs unlike a model's training data so the model can abstain or defer to a person, which is novelty detection applied to the inputs of a deployed model.
• Change detection: Compares images of the same scene taken at different times, such as satellite images before and after a flood, to find what changed.