Algorithmic Bias

Algorithmic bias is systematic and repeatable error in a computer system that produces unfair outcomes, such as favoring one group of people over another in hiring, lending, policing, or healthcare. The term covers any automated decision procedure, from a hand-written credit-scoring formula to a deep neural network, and because one flawed rule runs on every case, the same error repeats across thousands or millions of decisions. In discussions of machine learning (ML), algorithmic bias and bias in AI are synonyms. Algorithmic bias is the older and broader term, and it usually refers to the decisions a deployed system makes about people.

Three Origins of Algorithmic Bias#

Batya Friedman and Helen Nissenbaum's 1996 paper "Bias in Computer Systems" sorted bias into three kinds by where it comes from, a framework still used today:

• Preexisting bias: The system absorbs prejudice that already existed in society or in historical records.
• Technical bias: A design choice, such as which variable stands in for the quantity of interest, creates the unfairness.
• Emergent bias: Bias arises after deployment, when the population, the context, or the system's own influence on its future data changes.

Amazon scrapped an experimental recruiting model after it learned to penalize resumes containing the word "women's", a case of preexisting bias: it had been trained on ten years of resumes submitted mostly by men. A widely used US healthcare algorithm shows technical bias. It predicted patients' future medical costs as a stand-in for their health needs, and since less money had been spent on Black patients with the same level of illness, it referred them for extra care less often; a 2019 study in Science found that correcting the bias would raise the share of Black patients flagged for extra help from 17.7% to 46.5%. Predictive policing is the standard example of emergent bias, through a feedback loop:

Researchers showed in 2018 that this loop can run away: patrols find more incidents where they are sent, the new records confirm the ranking, and the model sends patrols back to the same neighborhoods even when crime rates elsewhere are equal.

Why Fair Decisions Involve Trade-Offs#

In 2016, ProPublica analyzed COMPAS, a risk score used in US courts, and found that Black defendants who did not reoffend were nearly twice as likely as white defendants to be labeled higher risk (45% versus 23%). The vendor answered that the score was equally calibrated for both groups, meaning a given score corresponded to the same reoffense rate regardless of race. Both claims held. Researchers including Kleinberg, Mullainathan, and Raghavan then proved that when two groups have different base rates, an imperfect score cannot be calibrated for both groups and also give them equal false positive and false negative rates. The following simulation shows the conflict with perfectly calibrated scores and one shared threshold:

import numpy as np

rng = np.random.default_rng(0)

# True risk per person; group B has a higher base rate than group A
risks = {"A": rng.beta(2, 5, 100_000), "B": rng.beta(3, 4, 100_000)}

for name, risk in risks.items():
    # Outcomes follow the risk exactly, so the score is perfectly calibrated in both groups
    outcome = rng.random(risk.size) < risk
    flagged = risk > 0.5  # one shared decision threshold

    # False positive rate: share of people without the outcome who were flagged anyway
    fpr = flagged[~outcome].mean()
    print(f"group {name}: base rate {outcome.mean():.2f}, false positive rate {fpr:.2f}")

Group A has a 0.29 base rate and a 0.06 false positive rate, and group B has a 0.43 base rate and a 0.23 false positive rate: the same calibrated score and threshold flag people in group B who never have the outcome almost four times as often. Every system that turns scores into decisions faces this choice. Ultralytics YOLO26 applies one confidence threshold (conf, 0.25 by default for prediction) to every detection, so any group whose detections score lower, such as pedestrians who are rare in the training data, loses more detections at the same setting.

Algorithmic Bias vs. Related Terms#

• Dataset bias: Flaws in the data itself, such as unrepresentative sampling. Dataset bias is one cause of algorithmic bias, alongside design choices and deployment effects.
• Fairness in AI: The field that defines fairness criteria, such as demographic parity or equal error rates, and the methods for meeting them.
• Data drift: A change in input data after deployment, which can create emergent bias in a model that tested fair at launch.

Auditing and Reducing Algorithmic Bias#

Audits compare outcomes and error rates across groups, using open-source toolkits such as Fairlearn and IBM's AI Fairness 360. Fixes come at three stages: pre-processing that reweights or resamples the training data, in-processing that adds fairness constraints to the training objective, and post-processing that adjusts decision thresholds per group. Ongoing model monitoring catches emergent bias that pre-launch audits miss. Regulation increasingly requires such audits: New York City's Local Law 144, enforced since July 5, 2023, bars employers from using automated hiring tools unless an independent auditor has checked them for bias within the past year and the employer publishes a summary with impact ratios comparing selection rates by sex and by race or ethnicity.