Kullback-Leibler Divergence (KL Divergence)
Kullback-Leibler divergence measures how two probability distributions differ. The order matters in training, drift detection, and INT8 calibration.
Kullback-Leibler divergence (KL divergence), also called relative entropy, measures how much extra information it takes, on average, to encode outcomes drawn from one probability distribution using a code designed for another. In machine learning, it measures the cost of using a model's probabilities in place of a reference distribution. The order matters: the first distribution determines how much weight each outcome carries. Solomon Kullback and Richard A. Leibler introduced the measure in their 1951 paper On Information and Sufficiency.
How to Calculate KL Divergence#
For discrete distributions P and Q, match the outcomes by label or bin. For each outcome, divide its probability in P by its probability in Q, take the logarithm, and multiply by its probability in P. Add those weighted terms. In the notation D_KL(P || Q), P is the distribution that supplies the weights. For continuous distributions, use probability densities and integrate the weighted log ratio over the outcome space. Natural logarithms give nats; base-2 logarithms give bits.
For P = (0.8, 0.2) and Q = (0.5, 0.5), the terms are 0.8 * ln(0.8 / 0.5) and 0.2 * ln(0.2 / 0.5). They add to 0.193 nats. Swapping P and Q gives 0.223 nats. Individual terms can be negative; the sum is always nonnegative.
KL Divergence in Python#
The handwritten digits sample bundled with scikit-learn contains 1,797 labeled images. Comparing its label frequencies with a uniform distribution shows how far the image classification sample departs from equal class balance. SciPy's entropy normalizes both inputs, so it accepts class counts as well as probabilities.
import numpy as np
from scipy.stats import entropy
from sklearn.datasets import load_digits
digits = load_digits()
labels = digits.target
counts = np.bincount(labels, minlength=10)
uniform = np.full(10, 0.1)
# Use the observed label frequencies as the first distribution.
kl = entropy(counts, uniform)
print(f"KL(observed || uniform): {kl:.6f} nats")The result is about 0.00011 nats, reflecting the roughly equal number of images per digit.
How to Interpret KL Divergence Values#
Zero means the distributions match. Larger values indicate greater mismatch when the direction, units, and outcome definitions stay the same. In model monitoring, KL divergence can flag changes in predicted class frequencies between a live window and a baseline. There is no universal alert threshold: the score varies with binning and sample size. A shift in predicted class frequencies can trigger a review; performance metrics on labeled data quantify any drop in model quality.
Infinity occurs whenever a discrete outcome has positive probability in P and zero probability in Q. An outcome with zero probability in P contributes nothing. In an observed histogram, an empty bin can reflect limited sampling. Adding a small count to every bin before normalizing produces a finite score for the resulting smoothed distributions. The informal name KL distance refers to the same quantity, which is asymmetric and fails the triangle inequality required of a mathematical distance.
Where KL Divergence Is Used#
In knowledge distillation, a student can learn from the teacher's full class distribution. Soft-target distillation uses temperature-scaled softmax probabilities for both models. Putting the teacher first in KL divergence penalizes the student for giving too little probability to classes the teacher considers plausible. A variational autoencoder uses KL divergence to regularize its encoded latent distribution toward a chosen prior, often a standard normal distribution.
In post-training model quantization, entropy calibration compares activation histograms before and after simulated INT8 quantization. It selects the clipping threshold that yields the lowest KL divergence. The Ultralytics TensorRT exporter selects entropy calibration for YOLO26 INT8 post-training exports targeting the deep learning accelerator (DLA) on NVIDIA Jetson with a DLA-compatible TensorRT 7–10 release.
KL Divergence vs. Cross-Entropy and Jensen-Shannon Divergence#
Cross-entropy loss adds the inherent uncertainty of P to the extra cost measured by KL divergence. When P is fixed, both losses have the same optimum for Q. When P is a one-hot target, its uncertainty is zero, so both losses equal the negative logarithm of the probability Q assigns to the labeled class.
Jensen-Shannon divergence averages two KL values: one from P to the midpoint distribution and one from Q to that same midpoint. The midpoint assigns positive probability to every outcome that appears in either distribution. This makes Jensen-Shannon divergence symmetric and gives it a maximum of 1 bit, reached when the distributions have no outcomes in common.










