Bias in AI

Bias in AI, also called AI bias, is systematic error in an artificial intelligence (AI) system that produces less accurate or less fair outcomes for some groups of people, often along lines such as skin tone, gender, age, dialect, or income. It differs from random error because it is consistent: the same group receives worse results again and again. In most usage, AI bias is a synonym for algorithmic bias; algorithmic bias is the older term and also covers automated decisions made by fixed formulas and scoring rules that involve no learning. Because bias can enter at every stage of building a machine learning (ML) system, AI ethics frameworks treat it as a property of the whole lifecycle.

Where Bias Comes From#

NIST's report Towards a Standard for Identifying and Managing Bias in Artificial Intelligence (SP 1270) sorts AI bias into three categories: systemic bias from institutions and history, statistical and computational bias from data and algorithms, and human bias from the people who build and use the system. In a typical ML pipeline, those categories show up at specific stages:

• Representation and dataset bias: Groups that are rare in the training data get worse predictions. A computer vision model trained mostly on photos from North America and Europe recognizes objects and scenes from other regions less reliably.
• Measurement: A feature or label measures one group less accurately. Pulse oximeters overestimate blood oxygen in patients with darker skin, and a 2020 study in the New England Journal of Medicine found that hidden low oxygen went undetected nearly three times as often in Black patients as in white patients.
• Labeling: Annotators bring their own judgments to the labels a model learns from. A 2019 study of hate speech detection found that widely used datasets labeled tweets in African American English as offensive far more often, and models trained on them flagged such tweets up to twice as often as others.
• Modeling and evaluation: Minimizing average error favors the majority, and a single overall score can hide a group that fails. Models can also learn shortcut features that happen to correlate with group membership.
• Deployment: A model used on a population or in a context it was never tested on, or whose outputs shape the data it later learns from, drifts into new bias over time.

Real-World Examples#

• Gender Shades: This 2018 audit found commercial gender classifiers misclassified darker-skinned women up to 34.7% of the time and lighter-skinned men at most 0.8%.
• Facial recognition: NIST's 2019 evaluation of face recognition algorithms found false positive rates 10 to 100 times higher for some demographic groups in many of them.
• Pedestrian detection: A 2019 study that annotated pedestrians in a driving dataset by skin tone found object detection models less accurate on darker-skinned pedestrians, a gap that time of day and occlusion did not explain.
• Dermatology AI: Skin-lesion classifiers trained mostly on images of light skin performed worse on dark skin when tested on a diverse clinical image set, and fine-tuning on diverse images narrowed the gap.

Measuring Bias#

Bias becomes visible through disaggregated evaluation: computing the same metric separately for each group, as model cards recommend. In the following example, a sensor reads 1.0 units too high for group B, which makes up 5% of patients, mirroring the pulse oximeter problem:

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(0)
group_b = rng.random(20000) < 0.05  # 5% of patients belong to group B
oxygen = rng.normal(size=20000)  # true blood oxygen, standardized
y = oxygen < 0  # condition to detect: low oxygen

# The sensor reads 1.0 too high for group B (measurement bias)
reading = (oxygen + 1.0 * group_b).reshape(-1, 1)
x_tr, x_te, y_tr, y_te, _, g_te = train_test_split(reading, y, group_b, random_state=0)

model = LogisticRegression()
model.fit(x_tr, y_tr)
pred = model.predict(x_te)

# Disaggregated evaluation: score each group separately
print(f"overall accuracy: {np.mean(pred == y_te):.3f}")
print(f"group A accuracy: {np.mean(pred[~g_te] == y_te[~g_te]):.3f}")
print(f"group B accuracy: {np.mean(pred[g_te] == y_te[g_te]):.3f}")

The model scores 0.967 overall and 0.981 on group A but only 0.706 on group B, whose low-oxygen cases the inflated readings hide. Group-level fairness metrics, such as equal false negative rates across groups, formalize these checks in fairness in AI.

Mitigating Bias#

Mitigation starts with data: collecting more examples from underrepresented groups, fixing faulty measurements and labels, and adding data augmentation or synthetic data where real examples are scarce. Training can reweight or resample rare groups or add fairness constraints, and post-processing can adjust decision thresholds. Ultralytics Platform charts the annotation count of every class in a dataset, and the cls_pw argument for YOLO26 training weights the classification loss by inverse class frequency, raised to that power, so rare classes count more. The NIST AI Risk Management Framework asks organizations to measure bias throughout a system's life, and Article 10 of the EU AI Act requires providers of high-risk AI systems to examine their training, validation, and test data for possible biases.

Bias in AI vs. Other Meanings of Bias#

• Bias-variance tradeoff: Statistical bias is error from a model too simple to capture the pattern in the data, which causes underfitting for every group alike.
• Bias parameters: In a neural network, each neuron's bias is a learnable offset added to its weighted inputs, the intercept term of a linear equation.