Differential Privacy

Differential privacy (DP) is a mathematical definition of privacy for algorithms that analyze data: an algorithm is differentially private if its output distribution barely changes when any one person's record is added to or removed from the input. Whatever an observer learns from the output, they would have learned almost the same without that person's data, so the result reveals almost nothing specific to them. It is used to publish data analytics results and to train machine learning models on sensitive data such as medical images and keyboard input. Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith introduced the definition in their 2006 paper Calibrating Noise to Sensitivity in Private Data Analysis, work that earned them the 2017 Gödel Prize. The guarantee holds against attackers with any amount of outside information, which is where traditional anonymization fails: stripped records can be re-identified by linking them to other databases.

Epsilon, Sensitivity, and the Privacy Budget#

A randomized algorithm M is ε-differentially private if, for any two datasets D and D′ that differ in one record and any set of outputs S, Pr[M(D) ∈ S] ≤ e^ε × Pr[M(D′) ∈ S]. The parameter epsilon (ε) bounds how much one person can shift the output: smaller values mean stronger privacy and noisier results. The relaxed (ε, δ) version allows the bound to fail with a tiny probability δ, typically far smaller than one divided by the number of records.

The noise is calibrated to sensitivity, the most one record can change a result. A count changes by at most 1, so the Laplace mechanism releases the true count plus Laplace noise with scale 1/ε. Privacy loss also accumulates: answering k queries at ε each spends up to k × ε in total, so organizations set a privacy budget for a dataset and stop answering once it is spent. The snippet below releases one count at three privacy levels and measures both the error and how often an attacker who sees the output can tell whether one person was in the data:

import numpy as np

rng = np.random.default_rng(0)

# A count over 1,000 records, and a neighboring dataset with one person removed
true_count = 412
neighbor_count = true_count - 1

for epsilon in (0.1, 1.0, 10.0):
    # Laplace mechanism: a count has sensitivity 1, so the noise scale is 1 / epsilon
    released = true_count + rng.laplace(scale=1 / epsilon, size=100_000)
    released_neighbor = neighbor_count + rng.laplace(scale=1 / epsilon, size=100_000)

    # Utility: average error. Privacy: accuracy of guessing whether the person is included
    error = np.abs(released - true_count).mean()
    threshold = true_count - 0.5
    attack = ((released > threshold).mean() + (released_neighbor <= threshold).mean()) / 2
    print(f"epsilon={epsilon:<4} mean error={error:6.2f}  attacker accuracy={attack:.1%}")

At ε = 0.1 the average error is 9.98 and the attacker is right 52.5% of the time, barely better than a coin flip. At ε = 1 the error falls to 1.00 and the attacker reaches 69.5%, and at ε = 10 the error is 0.10 while the attacker succeeds 99.7% of the time. The best accuracy any attacker can reach against an ε-differentially private release is e^ε / (1 + e^ε), which is 52.5% at ε = 0.1 and 73.1% at ε = 1.

Differentially Private Training#

DP-SGD, introduced by Martín Abadi and colleagues at Google in 2016, makes deep learning differentially private by changing stochastic gradient descent. It clips each training example's gradient to a maximum norm, so no single example can move the model weights too far, adds Gaussian noise to the summed gradients of each batch, and uses a privacy accountant to track the total ε spent over training. The result bounds how much the trained model can memorize any one example, which directly limits membership inference attacks. The cost is lower accuracy and more compute, so DP models usually need larger batches and datasets. Google's VaultGemma, released in 2025, is a 1-billion-parameter large language model pretrained entirely with DP-SGD at ε ≤ 2.0 and δ ≤ 1.1e-10.

Libraries such as Meta's Opacus for PyTorch compute the per-example gradients DP-SGD requires, and they reject layers that mix information across examples. Batch normalization is the common case, because each example's normalized value depends on the rest of its batch. A convolutional detector such as Ultralytics YOLO26, whose convolution blocks apply batch normalization after every convolution, needs those layers swapped for group normalization before DP-SGD can train it, the replacement Opacus applies automatically.

Where Differential Privacy Is Used#

• Official statistics: The 2020 US Census was the first decennial census to publish its results under differential privacy.
• Local differential privacy: Google's RAPPOR in Chrome (2014) and Apple's iOS features (2016) add noise on the device before data leaves it, so the company learns aggregate trends, such as which emoji are popular, without collecting any user's true values.
• Federated learning: Keeps data on devices, and adding DP to the shared model updates limits what they reveal about any one user's data.
• Synthetic data: Generators trained with DP produce shareable datasets that carry the same ε guarantee for the people in the original data.

Differential privacy is one technique within the broader field of data privacy, and it complements data security and confidential AI, which protect data from unauthorized access during storage and processing. NIST's SP 800-226, finalized in March 2025, gives guidelines for evaluating differential privacy guarantees and warns that an ε claim is meaningful only together with the unit of privacy it protects, such as one event or one person.