When AI makes headlines, all too often it is because of problems with bias and fairness. Some of the most infamous issues involve facial recognition, policing, and healthcare. Across industries and applications, machine learning contributes to creating outcomes where some groups or individuals are disadvantaged.
How do we develop AI systems that lead to fair and equitable outcomes? It starts with understanding what bias and fairness actually mean in the context of AI.
What Is Bias in AI
Bias can exist in many forms and can be introduced at any stage of the model development pipeline. At a fundamental level, bias is inherently present in the world around us and encoded into our society. We cannot directly solve the bias in the world. On the other hand, we can take measures to address bias in our data, our models, and our human review processes.
Bias in Data
Bias in data shows up in several forms.
Historical bias is already existing bias in the world that has seeped into our data. This bias can occur even given perfect sampling environments and feature selection. It tends to show up for groups that have been historically disadvantaged or excluded. Research has shown that word embeddings trained on news articles exhibit and perpetuate gender-based stereotypes from society.
Representation bias happens from how we define and sample a population to create a dataset. When training data is mostly based on one demographic group, the model performs poorly on others. Datasets collected through smartphone apps can underrepresent lower-income or older demographics.
Measurement bias occurs when choosing or collecting features or labels for predictive models. Data that is easily available is often a noisy proxy for actual features of interest. Measurement processes and data quality often vary across groups. In predictive policing, proxy measurements in predicting recidivism have led to harsher sentences for Black defendants than white defendants for the same crime.
Bias in Modeling
Even with perfect data, modeling methods can introduce bias.
Evaluation bias occurs during model iteration and evaluation. A model is optimized using training data, but quality is measured against benchmarks. Bias arises when benchmarks do not represent the general population or are not appropriate for how the model will be used.
Aggregation bias arises when distinct populations are inappropriately combined. Many AI applications have heterogeneous populations where a single model is unlikely to suit all groups. In healthcare, models for diagnosing diabetes have historically used metrics that differ in complicated ways across ethnicities. A single model for all populations is bound to exhibit bias.
Bias in Human Review
Even if your model makes correct predictions, human reviewers can introduce their own biases when deciding whether to accept or disregard a prediction. A reviewer might override a correct model prediction based on systemic bias, saying something like "I know that demographic, and they never perform well."
What Is Fairness
Bias and fairness in AI are two sides of the same coin. While there is no universally agreed upon definition for fairness, we can broadly define fairness as the absence of prejudice or preference for an individual or group based on their characteristics.
A Fairness Example
Consider a binary classification model where we believe we have accurate predictions. Fairness can come into question if the data actually includes two different underlying groups. These groups could represent different ethnicities, genders, or even geographical or temporal differences.
This fairness problem could result from aggregation bias. Using a single threshold for different groups leads to poor outcomes for some. In healthcare, this might mean inadequate care for certain patient populations.
Calibration and Group Fairness
One best practice is ensuring predictions are calibrated for each group. If model scores are not calibrated for each group, you are likely systemically overestimating or underestimating the probability of outcomes for one group.
Beyond group calibration, you may decide to create separate models and decision boundaries for each group. This approach is fairer than a single threshold applied universally.
The Individual Fairness Tension
Creating additional thresholds can lead to a new problem: individual fairness. Two individuals from different groups might share almost all the same characteristics but be treated completely differently by the AI system.
There is often tension between group and individual fairness. Optimizing for one can create problems for the other. The right balance depends on context, values, and the specific application.
Building Fair AI Systems
We see fairness challenges manifest in real systems regularly. Credit card algorithms have faced allegations of gender discrimination. Healthcare algorithms have been investigated for racial bias in patient care decisions. These incidents share a common pattern: organizations deployed AI without adequately testing for discriminatory outcomes.
Detecting bias is difficult. No universal metrics exist for quantifying fairness. Different definitions of fairness can conflict with each other. A model might appear fair on one measure while being unfair on another.
Machine learning models are particularly likely to obscure bias. They can learn complex interactions that produce discriminatory outcomes without explicitly discriminatory inputs. They can create localized bias, treating specific subgroups unfairly while appearing fair in aggregate.
The Path Forward
Organizations need systematic approaches to bias detection and fairness monitoring. This includes:
Testing models against multiple fairness metrics before deployment. Examining outcomes for protected groups and their intersections. Monitoring fairness measures over time in production through AI observability systems.
When bias is detected, processes should exist for response. This might mean adjusting thresholds, retraining models, or in severe cases taking models offline until issues are resolved.
Bias and fairness in AI is a developing field. More companies, especially in regulated industries, are investing in well-governed practices. The organizations that take this work seriously will build AI systems worthy of trust. Those that do not will face regulatory scrutiny, reputational damage, and real harm to the people their systems are meant to serve.
