Key Takeaways: Bias and Fairness
Core Ideas
-
Bias enters at every pipeline stage. AI bias is not a single bug that can be fixed at a single point. It can originate in problem formulation, data collection, data labeling, feature selection, model training, and deployment. Addressing bias requires examining the entire pipeline.
-
Four types of bias require different responses. Historical bias (the world is unequal and data reflects it), representation bias (some groups are missing from data), measurement bias (what you measure is not what you care about), and aggregation bias (one-size-fits-all models fail subgroups) each call for distinct mitigation strategies.
-
Fairness is not a single metric. The impossibility theorem proves that demographic parity, equalized odds, and calibration cannot all be achieved simultaneously when base rates differ between groups. Choosing between them is a moral and political decision, not a technical one.
-
"Fairness through unawareness" does not work. Removing protected characteristics (race, gender, age) from a model does not eliminate bias, because proxy variables — features correlated with protected characteristics — carry the same information through different channels.
-
Feedback loops amplify bias over time. AI predictions can generate the data that confirms them, creating self-reinforcing cycles. Predictive policing is a textbook example: predicting crime in over-policed areas leads to more policing, more arrests, and more data that "confirms" the prediction.
-
Technical fixes are necessary but not sufficient. Pre-processing, in-processing, and post-processing techniques can reduce measurable bias, but they cannot address the structural inequalities that produce biased data and biased institutions. Organizational approaches (diverse teams, independent audits, community engagement, ongoing monitoring) are equally important.
-
AI systems distribute power. Bias in AI is not just a technical problem — it is a question of who has the power to define fairness, who makes trade-off decisions, and who bears the consequences. These are civic questions that require democratic deliberation.
Key Terms
| Term | Definition |
|---|---|
| Algorithmic bias | Systematic errors in an AI system that produce unfair outcomes for specific groups |
| Historical bias | Bias that arises because training data reflects real-world patterns of inequality |
| Representation bias | Bias that arises because certain groups are underrepresented or absent from training data |
| Measurement bias | Bias that arises because the measured variable is a poor proxy for the actual concept of interest |
| Aggregation bias | Bias that arises when a model uses one-size-fits-all assumptions across diverse populations |
| Demographic parity | A fairness criterion requiring that outcomes be distributed equally across demographic groups |
| Equalized odds | A fairness criterion requiring that error rates (false positives and false negatives) be equal across groups |
| Calibration | A fairness criterion requiring that predicted probabilities mean the same thing across groups |
| Fairness through unawareness | The (usually insufficient) approach of removing protected characteristics from an AI model |
| Proxy variable | A feature so correlated with a protected characteristic that it effectively encodes that characteristic |
| Protected class | A demographic group protected from discrimination by law (e.g., race, gender, age, disability) |
| Disparate impact | When a facially neutral policy or system produces disproportionately negative outcomes for a specific group |
| Bias audit | A structured evaluation of an AI system's performance across demographic groups |
| Fairness-accuracy trade-off | The common (though not universal) tension between maximizing overall accuracy and achieving equitable performance across groups |
Frameworks Introduced
The Bias Pipeline Framework
Evaluate AI bias by examining each stage: 1. Problem formulation — What question is being asked? What assumptions are embedded? 2. Data collection — Who is in the data? Who is missing? 3. Labeling/annotation — Whose judgments are encoded in the labels? 4. Feature selection — Are any features proxies for protected characteristics? 5. Training/optimization — Is the model optimized in ways that disadvantage minorities? 6. Deployment/feedback — Do feedback loops amplify bias over time?
The Fairness Definition Comparison
When evaluating an AI system's fairness: - Ask which definition of fairness is being used - Identify what was traded away - Determine who made the trade-off decision - Assess whether affected communities had input
The Three Layers of Bias
- Technical — Addressable through data and model fixes
- Institutional — Addressable through organizational practices and requirements
- Structural — Addressable through policy reform and societal change
Common Misconceptions
| Misconception | Reality |
|---|---|
| "If we remove race/gender from the model, it will be fair" | Proxy variables carry the same information through other channels |
| "High overall accuracy means the system is fair" | Overall accuracy can mask severe disparities for specific subgroups |
| "Bias is a bug that can be fixed with better engineering" | Many forms of bias reflect structural inequalities that no algorithm can resolve |
| "There is one correct definition of fairness" | Multiple valid definitions exist, and they are mathematically incompatible |
| "AI bias is always unintentional" | While often unintentional, the failure to test for and address bias can be a deliberate choice |
| "Bias only enters through the data" | Bias can enter at problem formulation, feature selection, optimization, deployment, and through feedback loops |