License YOLO
Back to Ultralytics Glossary

Kullback-Leibler Divergence (KL Divergence)

Kullback-Leibler divergence measures how two probability distributions differ. The order matters in training, drift detection, and INT8 calibration.

Kullback-Leibler divergence (KL divergence), also called relative entropy, measures how much extra information it takes, on average, to encode outcomes drawn from one probability distribution using a code designed for another. In machine learning, it measures the cost of using a model's probabilities in place of a reference distribution. The order matters: the first distribution determines how much weight each outcome carries. Solomon Kullback and Richard A. Leibler introduced the measure in their 1951 paper On Information and Sufficiency.

How to Calculate KL Divergence#

For discrete distributions P and Q, match the outcomes by label or bin. For each outcome, divide its probability in P by its probability in Q, take the logarithm, and multiply by its probability in P. Add those weighted terms. In the notation D_KL(P || Q), P is the distribution that supplies the weights. For continuous distributions, use probability densities and integrate the weighted log ratio over the outcome space. Natural logarithms give nats; base-2 logarithms give bits.

For P = (0.8, 0.2) and Q = (0.5, 0.5), the terms are 0.8 * ln(0.8 / 0.5) and 0.2 * ln(0.2 / 0.5). They add to 0.193 nats. Swapping P and Q gives 0.223 nats. Individual terms can be negative; the sum is always nonnegative.

KL Divergence in Python#

The handwritten digits sample bundled with scikit-learn contains 1,797 labeled images. Comparing its label frequencies with a uniform distribution shows how far the image classification sample departs from equal class balance. SciPy's entropy normalizes both inputs, so it accepts class counts as well as probabilities.

import numpy as np
from scipy.stats import entropy
from sklearn.datasets import load_digits

digits = load_digits()
labels = digits.target
counts = np.bincount(labels, minlength=10)
uniform = np.full(10, 0.1)

# Use the observed label frequencies as the first distribution.
kl = entropy(counts, uniform)
print(f"KL(observed || uniform): {kl:.6f} nats")

The result is about 0.00011 nats, reflecting the roughly equal number of images per digit.

How to Interpret KL Divergence Values#

Zero means the distributions match. Larger values indicate greater mismatch when the direction, units, and outcome definitions stay the same. In model monitoring, KL divergence can flag changes in predicted class frequencies between a live window and a baseline. There is no universal alert threshold: the score varies with binning and sample size. A shift in predicted class frequencies can trigger a review; performance metrics on labeled data quantify any drop in model quality.

Infinity occurs whenever a discrete outcome has positive probability in P and zero probability in Q. An outcome with zero probability in P contributes nothing. In an observed histogram, an empty bin can reflect limited sampling. Adding a small count to every bin before normalizing produces a finite score for the resulting smoothed distributions. The informal name KL distance refers to the same quantity, which is asymmetric and fails the triangle inequality required of a mathematical distance.

Where KL Divergence Is Used#

In knowledge distillation, a student can learn from the teacher's full class distribution. Soft-target distillation uses temperature-scaled softmax probabilities for both models. Putting the teacher first in KL divergence penalizes the student for giving too little probability to classes the teacher considers plausible. A variational autoencoder uses KL divergence to regularize its encoded latent distribution toward a chosen prior, often a standard normal distribution.

In post-training model quantization, entropy calibration compares activation histograms before and after simulated INT8 quantization. It selects the clipping threshold that yields the lowest KL divergence. The Ultralytics TensorRT exporter selects entropy calibration for YOLO26 INT8 post-training exports targeting the deep learning accelerator (DLA) on NVIDIA Jetson with a DLA-compatible TensorRT 7–10 release.

KL Divergence vs. Cross-Entropy and Jensen-Shannon Divergence#

Cross-entropy loss adds the inherent uncertainty of P to the extra cost measured by KL divergence. When P is fixed, both losses have the same optimum for Q. When P is a one-hot target, its uncertainty is zero, so both losses equal the negative logarithm of the probability Q assigns to the labeled class.

Jensen-Shannon divergence averages two KL values: one from P to the midpoint distribution and one from Q to that same midpoint. The midpoint assigns positive probability to every outcome that appears in either distribution. This makes Jensen-Shannon divergence symmetric and gives it a maximum of 1 bit, reached when the distributions have no outcomes in common.

Explore solutions

Computer vision in sports

Computer vision in sports

Discover how Ultralytics YOLO powers sports vision with player and ball tracking, performance analytics, coaching insights, and athlete rehabilitation.
Learn more
Computer vision for aerial imagery

Computer vision for aerial imagery

Turn drone and aerial imagery into real-time insights for agriculture, aquaculture, conservation, and geospatial analysis with Ultralytics YOLO.
Learn more
Computer vision in aerospace

Computer vision in aerospace

Apply computer vision in aerospace with Ultralytics YOLO for aircraft inspection, runway FOD detection, airport ground operations, and composite defect detection.
Learn more
Computer vision in security

Computer vision in security

Protect every site with Ultralytics YOLO models. Vision AI powers perimeter monitoring, threat detection, license-plate recognition, and workplace safety.
Learn more
Computer vision in robotics

Computer vision in robotics

Power smarter machines with Ultralytics YOLO models. Vision AI in robotics drives autonomous navigation, perception, object tracking, and real-time control.
Learn more
Computer vision in logistics

Computer vision in logistics

Streamline logistics with Ultralytics YOLO models. Vision AI enables package inspection, sorting, vehicle tracking, and real-time warehouse safety monitoring.
Learn more
Computer vision in retail

Computer vision in retail

Reimagine retail with Ultralytics YOLO models. Vision AI powers inventory tracking, shelf monitoring, queue management, and smarter customer insights.
Learn more
Computer vision in healthcare

Computer vision in healthcare

Build healthcare solutions with Ultralytics YOLO models. Vision AI in healthcare powers faster medical imaging, smarter diagnostics, and patient monitoring.
Learn more
Computer vision in manufacturing

Computer vision in manufacturing

Optimize manufacturing with Ultralytics YOLO models. Vision AI drives quality control, defect detection, PPE compliance, and assembly line automation.
Learn more
Computer vision in automotive

Computer vision in automotive

Apply computer vision in automotive with Ultralytics YOLO models. Vision AI elevates road safety, driver assistance, and vehicle automation for smarter roads.
Learn more
Computer vision in agriculture

Computer vision in agriculture

Bring vision AI to smart agriculture with Ultralytics YOLO models. Power crop monitoring, livestock tracking, and precision farming for higher, smarter yields.
Learn more

Let's build the future of AI together!

Begin your journey with the future of machine learning