Key Takeaways: Chapter 14 — Bias in Data, Bias in Machines

Core Takeaways

1. Algorithmic bias is systematic, disadvantaging, and unjustified. It is not random error; it is patterned failure that tracks lines of social inequality. The pattern is the point — one denied application is a data point; a thousand denied applications clustered along racial lines is a system.

2. Bias does not require intent. The most dangerous algorithmic biases are structural — embedded in data, features, and optimization objectives by the same social forces that produce inequality. Nobody at Amazon told the hiring algorithm to penalize women. Nobody at VitraMed told the risk model to underserve Black patients. The algorithms learned these patterns from data that reflected an unjust world, and they reproduced that injustice with mathematical precision.

3. There are six types of bias, and each requires a different intervention. Historical bias exists in the world before data is collected. Representation bias occurs when training data excludes populations. Measurement bias arises from flawed proxies. Aggregation bias comes from treating distinct subgroups as one population. Evaluation bias occurs when benchmarks are unrepresentative. Deployment bias emerges when a model is used in a context it was not designed for. Identifying the type of bias is the first step toward addressing it.

4. Bias can enter at every stage of the machine learning pipeline. From problem formulation through data collection, feature engineering, model training, evaluation, and deployment — each stage is a potential entry point. Bias introduced early propagates and compounds through later stages. Effective bias mitigation requires a pipeline-wide approach, not a single-stage fix.

5. Proxy variables allow bias to persist even when protected attributes are removed. Zip code proxies for race (because of residential segregation). Healthcare spending proxies for race (because of access disparities). Resume formatting may proxy for socioeconomic status. Removing race or gender from a model's features does not eliminate the bias, because the model reconstructs the pattern from correlated proxies. This phenomenon — redundant encoding — makes bias resilient to naive "blinding" approaches.

6. The COMPAS case reveals that fairness has multiple definitions — and they conflict. ProPublica measured fairness by equal false positive rates. Northpointe measured fairness by calibration. Both were mathematically correct. This is not a failure of analysis; it is a fundamental insight about the nature of fairness, which Chapter 15 will formalize.

7. The Obermeyer healthcare study demonstrates the "proxy trap." A seemingly reasonable variable (healthcare spending) used as a proxy for a concept that is harder to measure (health need) can embed devastating bias when the proxy's relationship to the target concept differs across groups. The lesson is not to avoid a specific proxy but to interrogate every proxy for structural contamination.

8. Amazon's hiring algorithm proves that "fairness through unawareness" does not work. Removing gender as a feature did not eliminate gender bias, because gender was encoded in dozens of other features — institutional affiliations, organizational memberships, linguistic patterns, formatting conventions. Bias is not located in any single feature; it is distributed across the correlational structure of the data.

9. Feedback loops transform one-time biases into self-reinforcing cycles. A biased prediction leads to biased action, which generates biased data, which confirms the biased prediction. In predictive policing, this means that initially over-policed neighborhoods appear increasingly "high-crime" even if actual crime rates are unchanged — because more policing detects more crime, which justifies more policing. Feedback loops are among the most dangerous properties of deployed algorithmic systems because they make bias self-perpetuating.

10. Intersectional analysis is essential because single-axis analysis masks the worst disparities. A system may perform acceptably for Black people as a group and for women as a group, while failing dramatically for Black women specifically. Buolamwini and Gebru's "Gender Shades" study showed error rates of 34.7% for dark-skinned women vs. less than 1% for light-skinned men. Only intersectional analysis — examining subgroups defined by multiple characteristics — reveals the full scope of bias.

Key Concepts

Key Debates

1. Is algorithmic bias the algorithm's problem or society's problem? If algorithms reflect social reality, is fixing algorithms treating a symptom? Or do algorithms actively worsen inequality through feedback loops and scale? The chapter argues: both. Algorithms reflect and amplify.

2. Can historical bias be corrected at the algorithmic level? If training data encodes centuries of discrimination, can we build models that predict what would have happened in a fair world? Counterfactual and causal modeling approaches attempt this, but they require defining "fair" — which is a political, not technical, task.

3. Should risk assessment be used in criminal justice at all? The COMPAS debate is not just about calibration vs. equalized odds. It is about whether quantifying human risk with a numerical score is compatible with justice, dignity, and the presumption of innocence.

4. What obligations do companies have when they discover bias in internal tools? Amazon scrapped its hiring algorithm. But should companies be required to disclose such discoveries publicly? To notify affected applicants? To submit to independent audit?

Looking Ahead

Chapter 14 established that algorithmic systems can be biased — and biased in patterned, predictable ways that track existing social inequalities. The natural next question is: how do we fix it? Chapter 15, "Fairness — Definitions, Tensions, and Trade-offs," reveals that "fixing bias" first requires defining "fairness" — and that fairness has multiple competing definitions that are often mathematically incompatible. The impossibility theorem awaits.

Use this summary as a study reference and a quick-access card for key vocabulary. The bias taxonomy (six types) and the bias pipeline (six stages) are frameworks you will apply throughout the remainder of this textbook.