Fairness in AI

Fairness in AI is the requirement that a model's decisions and errors do not systematically disadvantage people because of sensitive attributes such as race, sex, age, or disability, together with the metrics and methods used to measure and enforce that requirement. Bias in AI names the problem: systematic skew that can enter through unrepresentative training data (dataset bias) or through modeling and deployment choices (algorithmic bias). Fairness is the criterion a system is tested against, one of the core principles of AI ethics alongside transparency and accountability, and it has several formal definitions that cannot all hold at once.

Measuring Fairness#

Group fairness metrics compare a model's behavior across groups defined by a sensitive attribute:

• Demographic parity: Each group receives positive decisions, such as loan approvals or interview invitations, at the same rate.
• Equalized odds: Each group has the same true positive rate and the same false positive rate. Moritz Hardt, Eric Price, and Nathan Srebro defined it in 2016, together with the weaker equal opportunity, which equalizes only the true positive rate.
• Calibration within groups: A predicted score of 0.7 corresponds to the same 70% outcome rate in every group.
• Individual fairness: Similar individuals receive similar predictions, a definition Cynthia Dwork and colleagues formalized in 2012.

The snippet below scores a perfectly accurate model on two groups whose base rates differ:

import numpy as np

# Outcomes and a sensitive attribute for 10 applicants; group A has a higher base rate
y_true = np.array([1, 1, 1, 1, 0, 1, 1, 0, 0, 0])
group = np.array(["A"] * 5 + ["B"] * 5)

# A perfectly accurate model predicts every outcome correctly
y_pred = y_true.copy()

for g in ("A", "B"):
    in_group = group == g
    selection_rate = y_pred[in_group].mean()
    tpr = y_pred[in_group & (y_true == 1)].mean()  # true positive rate
    fpr = y_pred[in_group & (y_true == 0)].mean()  # false positive rate
    print(f"group {g}: selection rate {selection_rate:.0%}, TPR {tpr:.0%}, FPR {fpr:.0%}")

The model satisfies equalized odds, with a 100% true positive rate and a 0% false positive rate in both groups, and still violates demographic parity, approving 80% of group A and 40% of group B. Forcing equal selection rates would require wrong decisions in at least one group.

Why Fairness Definitions Conflict#

Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan in 2016 and Alexandra Chouldechova in 2017 proved that when base rates differ between groups, a classifier cannot be calibrated within each group and also have equal false positive and false negative rates, except in degenerate cases such as perfect prediction. The COMPAS recidivism tool shows the conflict in practice. ProPublica reported in 2016 that Black defendants who did not reoffend were nearly twice as likely as white defendants to be labeled high risk, while the tool's developer showed that its scores were equally calibrated for both groups. Both analyses were correct, so choosing a fairness metric for a given application is a policy decision as much as a technical one.

Fairness in Computer Vision#

The Gender Shades study by Joy Buolamwini and Timnit Gebru (2018) audited three commercial gender classifiers and found error rates up to 34.7% for darker-skinned women against at most 0.8% for lighter-skinned men. Similar gaps appear wherever groups are underrepresented in training images, from facial recognition to medical image analysis models for skin conditions trained on dermatology datasets dominated by lighter skin. A computer vision audit therefore reports precision and recall separately for each relevant subgroup and records them in a model card, since a single overall accuracy figure can hide a subgroup that fails.

Mitigation Strategies#

Interventions are grouped by the stage of the machine learning pipeline they change. Pre-processing methods collect representative data, fill gaps with targeted data augmentation or synthetic data, and reweight examples so underrepresented groups count more. Ultralytics YOLO offers the same reweighting for classes: setting the cls_pw training argument above its default of 0 weights each class's loss by its inverse frequency raised to that power, so a rare class carries more weight in training.

In-processing methods add fairness constraints or penalties to the loss function, or use distributionally robust optimization to minimize the loss of the worst-performing group. Post-processing methods adjust decision thresholds per group after training, the approach Hardt and colleagues proposed for reaching equalized odds.

Open-source toolkits such as Fairlearn and IBM's AI Fairness 360 implement these metrics and mitigations. Regulation increasingly requires the measurements: the EU AI Act requires providers of high-risk systems to examine training data for possible biases, and New York City's Local Law 144 has required annual independent bias audits of automated hiring tools since July 2023.