Chapter 25 Further Reading: Bias in AI Systems

Foundational Texts

1. Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy — Cathy O'Neil (2016)

The book that brought algorithmic bias to a mainstream audience. O'Neil, a former quantitative analyst, examines how opaque, unregulated algorithms — in education, criminal justice, lending, and insurance — systematically disadvantage the poor and reinforce inequality. Readable, passionate, and thoroughly researched, this is the essential starting point for anyone who wants to understand why algorithmic fairness matters. O'Neil's concept of "weapons of math destruction" — models that are widespread, opaque, and destructive — provides a useful framework for identifying high-risk AI systems.

2. Algorithms of Oppression: How Search Engines Reinforce Racism — Safiya Umoja Noble (2018)

Noble's examination of how Google Search results reflect and amplify racial and gender stereotypes was groundbreaking when published and remains essential. The book demonstrates that seemingly neutral information retrieval systems encode societal biases in ways that shape public perception. Particularly relevant to the chapter's discussion of representation bias and proxy variables. Required reading for anyone involved in search, recommendation, or content curation systems.

3. Race After Technology: Abolitionist Tools for the New Jim Code — Ruha Benjamin (2019)

Benjamin, a Princeton sociologist, introduces the concept of the "New Jim Code" — the idea that technology can perpetuate racial hierarchies under the guise of innovation and objectivity. The book connects AI bias to broader patterns of structural racism and argues that technical fixes alone are insufficient without addressing the social and political contexts in which technologies are developed and deployed. Pairs well with the chapter's discussion of historical bias and proxy variables.

4. Artificial Unintelligence: How Computers Misunderstand the World — Meredith Broussard (2018)

Broussard coins the term "technochauvinism" — the belief that technology is always the best solution — and demonstrates through case studies in journalism, education, and criminal justice how AI systems fail in predictable ways. Her analysis of algorithmic bias in standardized testing and school assignment systems complements the chapter's coverage of hiring and lending bias.

The Gender Shades Study and Facial Recognition

5. "Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification" — Joy Buolamwini and Timnit Gebru (2018)

Proceedings of the Conference on Fairness, Accountability, and Transparency (FAccT) The landmark study discussed in the chapter. Buolamwini and Gebru's intersectional analysis of commercial facial recognition systems — revealing error rates up to 34.7% for darker-skinned women — set the standard for how AI systems should be evaluated across demographic subgroups. The paper's methodology (balanced benchmark, intersectional categories, Fitzpatrick scale for skin type) is a model for rigorous fairness evaluation. Essential primary source.

6. Unmasking AI: My Mission to Protect What Is Human in a World of Machines — Joy Buolamwini (2023)

Buolamwini's memoir expands on the personal and professional journey behind the Gender Shades research. The book combines technical rigor with storytelling, documenting her experiences as a Black woman computer scientist who discovered that the facial recognition systems she was building could not see her face. Provides important context on how individual experiences of exclusion can motivate systemic research.

7. "Actionable Auditing: Investigating the Impact of Publicly Naming Biased Performance Results of Commercial AI Products" — Inioluwa Deborah Raji and Joy Buolamwini (2019)

Proceedings of AAAI/ACM Conference on AI, Ethics, and Society A follow-up study showing that public disclosure of bias in facial recognition systems led to significant improvements by the companies named. Microsoft reduced its error rate for darker-skinned women from 20.8% to 1.5% within a year of the Gender Shades publication. Demonstrates that external accountability can drive internal change — a finding directly relevant to the chapter's discussion of organizational responsibility.

The COMPAS Debate and Criminal Justice

8. "Machine Bias" — Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner (2016)

ProPublica, May 23, 2016. The investigative article that launched the modern algorithmic fairness debate. Essential primary source for Case Study 2. ProPublica's analysis of COMPAS risk scores in Broward County, Florida, revealed racial disparities in false positive and false negative rates that forced a reckoning with what "fair" means in algorithmic prediction.

9. "Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments" — Alexandra Chouldechova (2017)

Big Data, 5(2), 153-163. The paper that proved the impossibility result: when base rates differ across groups, calibration, equal false positive rates, and equal false negative rates cannot all be satisfied simultaneously. This mathematical proof is the intellectual foundation of the fairness tradeoffs discussed throughout the chapter. Accessible to readers with basic probability knowledge.

10. "Inherent Trade-Offs in the Fair Determination of Risk Scores" — Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan (2016)

Proceedings of Innovations in Theoretical Computer Science (ITCS) An independent proof of the impossibility result, arrived at simultaneously with Chouldechova's work. Kleinberg et al. approach the problem from a computer science perspective, complementing Chouldechova's statistical framing. Reading both papers provides a complete picture of why the fairness impossibility is robust and fundamental.

11. "The Measure and Mismeasure of Fairness: A Critical Review of Fair Machine Learning" — Sam Corbett-Davies and Sharad Goel (2018)

arXiv preprint The most comprehensive survey of algorithmic fairness definitions, their mathematical relationships, and their limitations. Corbett-Davies and Goel organize the landscape of fairness metrics into three families (anti-classification, classification parity, and calibration), explain the tradeoffs between them, and argue for a framework based on equalizing utility across groups. Essential reference for Chapter 26 as well.

Healthcare Bias

12. "Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations" — Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan (2019)

Science, 366(6464), 447-453. The study that revealed how a widely used healthcare algorithm discriminated against Black patients by using healthcare costs as a proxy for health needs. One of the most cited papers in algorithmic fairness and the basis for the chapter's discussion of proxy variables in healthcare. The paper estimates that eliminating the bias would increase the percentage of Black patients identified for additional care from 17.7% to 46.5%. Landmark work that every healthcare AI practitioner should read.

13. "Racial Bias in Pulse Oximetry Measurement" — Michael W. Sjoding et al. (2020)

New England Journal of Medicine, 383(25), 2477-2478. The study documenting that pulse oximeters systematically overestimate oxygen saturation in patients with darker skin pigmentation. During the COVID-19 pandemic, this measurement bias contributed to delayed treatment for Black patients. Demonstrates how bias in measurement devices — not just algorithms — can cascade through AI systems that depend on those measurements.

14. "The Coded Gaze: Bias in Artificial Intelligence" — Joy Buolamwini (2017)

TED Talk A 9-minute talk that introduces the concepts of algorithmic bias and the "coded gaze" to a general audience. Buolamwini's demonstration of facial recognition failure on her own face is one of the most powerful illustrations of AI bias ever produced. Excellent for sharing with non-technical colleagues or executives who need to understand why bias matters.

Fairness Tools and Technical Resources

15. Fairlearn — Microsoft

URL: fairlearn.org The open-source Python library used in this chapter's BiasDetector implementation. Fairlearn provides MetricFrame for disaggregated metric analysis, reductions-based algorithms for in-processing mitigation (ExponentiatedGradient), and ThresholdOptimizer for post-processing threshold adjustment. The documentation includes tutorials and case studies. Essential for any practitioner implementing fairness audits.

16. AI Fairness 360 (AIF360) — IBM

URL: aif360.readthedocs.io IBM's open-source toolkit for detecting and mitigating bias in AI systems. AIF360 provides over 70 fairness metrics and 11 bias mitigation algorithms across pre-processing, in-processing, and post-processing categories. More comprehensive than Fairlearn in terms of metric coverage, with excellent Jupyter notebook tutorials. The "bias and fairness" tutorial is a good companion to this chapter's code.

17. "A Survey on Bias and Fairness in Machine Learning" — Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan (2021)

ACM Computing Surveys, 54(6), 1-35. The most comprehensive survey of bias sources, fairness definitions, and mitigation strategies in the literature. Covers the Suresh and Guttag taxonomy used in this chapter, plus additional bias sources, fairness criteria, and mitigation techniques. Dense but thorough — serves as a reference encyclopedia for the field.

18. "Auditing Employment Algorithms for Discrimination" — Pauline Kim (2023)

Northwestern University Law Review, 117(1). A legal analysis of how US employment discrimination law applies to AI hiring systems. Kim examines the interaction between Title VII disparate impact doctrine and algorithmic decision-making, and evaluates proposals for mandatory algorithmic auditing. Directly relevant to the chapter's discussion of Lena Park's legal framework.

19. EU AI Act — Official Text (2024)

URL: eur-lex.europa.eu (search "Artificial Intelligence Act") The full text of the world's first comprehensive AI regulatory framework. Annex III lists high-risk AI use cases, including employment (recruitment, screening, evaluation) and access to essential services (credit, insurance, education). For business leaders operating in or serving the EU market, this is the primary compliance reference.

20. "Uniform Guidelines on Employee Selection Procedures" — US Equal Employment Opportunity Commission (1978)

URL: govinfo.gov The regulatory document that established the four-fifths rule (80% threshold) for disparate impact analysis. While written before AI existed, its principles apply directly to algorithmic hiring tools. Section 4D defines adverse impact; Section 3A establishes the four-fifths guideline. Surprisingly readable for a government document.

AI Ethics and Governance

21. The Alignment Problem: Machine Learning and Human Values — Brian Christian (2020)

A deeply researched narrative exploration of how AI systems learn human values — and how they fail to. Christian interviews leading researchers in machine learning, fairness, interpretability, and AI safety, weaving their work into a coherent story about the challenge of building AI systems that do what we actually want. The chapters on fairness and bias draw on many of the same cases discussed in this chapter. Excellent for readers who want the broader intellectual context.

22. Atlas of AI: Power, Politics, and the Planetary Costs of Artificial Intelligence — Kate Crawford (2021)

Crawford, a leading AI ethics researcher, maps the full supply chain of AI — from the lithium mines that produce battery materials to the data centers that house models to the content moderators who clean up their outputs. The book broadens the definition of AI "bias" beyond algorithmic fairness to include labor exploitation, environmental damage, and the concentration of power. Challenges readers to think about AI ethics at a systems level, not just a model level.

23. NIST AI Risk Management Framework (AI RMF 1.0) — National Institute of Standards and Technology (2023)

URL: nist.gov/ai The US federal government's framework for managing AI risks, including bias. The framework organizes AI risk management into four functions: Govern, Map, Measure, and Manage. Section 2.6 specifically addresses fairness and bias. While not legally binding, the framework is increasingly referenced in procurement requirements and industry standards. Directly relevant to Chapter 27's coverage of AI governance frameworks.

Additional Research

24. "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?" — Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell (2021)

Proceedings of FAccT 2021 A widely discussed paper examining the risks of large language models, including the encoding and amplification of bias in training data. The paper argues that LLMs trained on internet text inevitably absorb the biases present in that text — racism, sexism, ableism — and present them with the authority of a "knowledgeable" system. Relevant to the chapter's discussion of historical bias and the LLM applications covered in Part 4.

25. "Datasheets for Datasets" — Timnit Gebru et al. (2021)

Communications of the ACM, 64(12), 86-92. Proposes a standardized documentation framework for datasets — analogous to datasheets for electronic components — that would require dataset creators to document the data's composition, collection process, intended uses, and known biases. Implementing datasheets is a practical step toward the transparency and accountability advocated in this chapter. Pairs with the model card concept introduced in Chapter 26.

Reading Path Recommendation

If you read three things: Start with O'Neil's Weapons of Math Destruction (accessible framing of why algorithmic bias matters), then Buolamwini and Gebru's Gender Shades paper (the gold standard for intersectional bias evaluation), then Obermeyer et al.'s Science paper (the most impactful demonstration of proxy variable bias in healthcare).

If you read one thing: O'Neil's Weapons of Math Destruction. It is the most accessible and comprehensive introduction to AI bias available, and it will give you the conceptual vocabulary to engage with every other source on this list.

If you want to build: Install Fairlearn (resource 15), work through the tutorials, and apply the BiasDetector class from this chapter to a dataset from your own domain. Then compare your implementation with IBM's AIF360 toolkit (resource 16) to see a production-grade alternative.

If you want to go deep on the mathematics: Read Chouldechova (resource 9) and Kleinberg et al. (resource 10) for the impossibility results, then Corbett-Davies and Goel (resource 11) for the comprehensive survey of fairness definitions and their relationships.