Chapter 7: Understanding Algorithmic Bias

Opening: The Algorithm That Learned From Discrimination

In 2014, Amazon assembled a team of engineers in Edinburgh, Scotland, with an ambitious mandate: build an artificial intelligence system that could automate the company's hiring process. The goal was a tool that could ingest hundreds of thousands of résumés and identify the top candidates with minimal human involvement — a kind of mechanical recruiter that would operate at scale impossible for any human hiring team. Amazon was, at the time, processing millions of job applications per year. The efficiency case seemed obvious.

The engineers trained their model on ten years of Amazon's own hiring data: résumés submitted between 2004 and 2014, combined with records of who had been hired and, presumably, who had performed well. The system would learn, in other words, from the accumulated hiring wisdom of a decade's worth of Amazon recruiters. If it worked, it would capture the patterns that distinguished Amazon's most successful employees from the rest and apply those patterns automatically to every future application.

By 2015, early warning signs had appeared. The system was not rating candidates in a gender-neutral way. Female applicants were consistently rated lower than male applicants with comparable credentials. When Amazon's engineers dug into the mechanism, they found something both obvious in retrospect and alarming in implication: the model had learned that male applicants were more likely to be hired, because male applicants had been more likely to be hired throughout the decade of training data. The system had absorbed the historical bias of Amazon's recruiters and encoded it into an automated process that would apply it at scale, millions of times, with no individual decision-maker to appeal to and no moment of human reflection to interrupt it.

The bias was not a simple matter of the model noticing gender and penalizing women. Amazon's engineers had explicitly removed gendered language and direct gender markers from the input features. But the system had learned to use proxies. It downgraded résumés that included the word "women's" — as in "women's chess club" or "women's leadership initiative." It penalized graduates of historically women's colleges. It identified patterns in language style that correlated with female authorship in the training data and rated those patterns negatively. The algorithm had effectively reconstructed gender from indirect signals, because gender had been a powerful predictor in the underlying data.

Amazon quietly shut down the tool in 2017. The story did not become public until Reuters reported it in October 2018, citing five unnamed current and former Amazon employees. Amazon's response was that the tool had never been used to make final hiring decisions and that it had been shut down before any candidates were harmed. Critics noted that hundreds of thousands of résumés had been processed by the system, that Amazon had not disclosed the issue to any regulatory body, and that the company had not reached out to any candidates who might have been screened out unfairly.

The Amazon hiring algorithm is now one of the canonical examples in the field of AI ethics — a case study assigned in business schools, cited in regulatory guidance, and referenced in congressional testimony. This chapter uses it as a through-line to explore a fundamental challenge in artificial intelligence: the tendency of AI systems to absorb, amplify, and automate the biases embedded in the data they learn from, the humans who design them, and the social systems they are embedded within.

Understanding algorithmic bias is not an optional concern for business professionals who deploy or depend on AI systems. It is a strategic, legal, and ethical imperative. The costs of getting it wrong — in litigation, in regulatory action, in reputational damage, and most importantly in harm to real human beings — are severe and growing. The chapters that follow will address measurement, mitigation, and domain-specific applications in detail. This chapter builds the foundation: a precise vocabulary, a clear taxonomy of causes, and a structural understanding of why algorithmic bias is so hard to detect and so easy to perpetuate.

Learning Objectives

By the end of this chapter, you should be able to:

1. Define algorithmic bias with precision, distinguishing its technical, emergent, and sociotechnical dimensions.
2. Identify the major sources of bias across the machine learning pipeline, from data collection through deployment.
3. Explain the concept of disparate impact and how facially neutral AI systems can produce discriminatory outcomes.
4. Describe the feedback loop problem and why biased AI systems tend to self-reinforce.
5. Apply the concept of intersectionality to bias analysis and explain why single-axis fairness testing is insufficient.
6. Identify the major legal frameworks governing algorithmic discrimination in employment, housing, credit, and criminal justice.
7. Assess algorithmic bias as a multidimensional business risk spanning reputational, legal, and operational dimensions.
8. Describe early-detection organizational practices that reduce bias risk before deployment.

Section 7.1: What Is Algorithmic Bias? A Precise Definition

The word "bias" carries different meanings in different contexts, and those differences matter considerably when thinking about AI systems. In everyday language, bias often connotes prejudice — a conscious or unconscious predisposition against a particular group. In statistics, bias has a precise technical definition: systematic error, a deviation between a model's predictions and the true values it is trying to predict. In law, discrimination involves the differential treatment of individuals on the basis of protected characteristics such as race, sex, or national origin. Algorithmic bias sits at the intersection of all three meanings, which is both why the concept is rich and why it is frequently misunderstood.

Bias as Statistical Error vs. Bias as Discrimination

A helpful starting point is to distinguish between bias-as-error and bias-as-discrimination. A model that consistently overestimates credit risk for all applicants by 15 percent exhibits statistical bias — it is systematically wrong — but if it overestimates equally for all demographic groups, it does not produce discriminatory outcomes in the legally or ethically relevant sense. Conversely, a model that is highly accurate on average may still exhibit discriminatory bias if its errors are distributed unequally across groups — if, for instance, it is accurate for white applicants but systematically inaccurate for Black applicants.

This distinction is crucial because it reveals that overall accuracy is an insufficient metric for fairness. A facial recognition system that achieves 98 percent accuracy overall can simultaneously exhibit deeply discriminatory performance if that 98 percent masks dramatically lower accuracy for faces that are darker-skinned, female, or older. As we will examine in Section 7.4, this is not a hypothetical: NIST's systematic evaluation of commercial facial recognition systems found exactly this pattern.

Three Levels of Algorithmic Bias

Researchers in this field have developed taxonomies that capture the multi-layered nature of algorithmic bias. One of the most useful distinguishes three levels:

Technical bias refers to errors and inaccuracies in AI systems that arise from flawed algorithms, insufficient or unrepresentative training data, or poor implementation choices. Technical bias can often be identified through systematic testing and, in principle, corrected through technical means. A model that performs less well on minority-class examples because those examples were underrepresented in training data exhibits technical bias in this sense. It is real and harmful, but it is also, at least in principle, addressable by improving the data or the model.

Emergent bias refers to bias that arises not from flaws in the system itself but from the interaction between the system and the social world it operates within. A system designed and tested for one population may exhibit bias when deployed to another. A hiring algorithm designed and validated in one cultural context may embed assumptions that are inapplicable or harmful when the algorithm is used globally. A recommendation system designed to maximize engagement may, through the actions of users, come to amplify divisive or extremist content even though neither the designers nor the algorithm ever targeted those outcomes. Emergent bias is often harder to anticipate because it depends on the social environment that did not fully exist when the system was designed.

Sociotechnical bias refers to bias that is embedded in the assumptions, purposes, and social function of the system itself, independent of any technical flaw. When Amazon trained its hiring algorithm to predict who would be hired at Amazon, it was operationalizing a social judgment — that past hiring decisions reflected genuine merit — that was itself shaped by decades of gender discrimination in the technology industry. The problem was not merely a data problem or a model problem; it was a problem with the framing of the task. Sociotechnical bias is the deepest form of algorithmic bias and the hardest to address because fixing it requires not just better data or better models but a willingness to question the social assumptions built into the system's design.

Intentional vs. Structural Discrimination

One of the most important conceptual moves in understanding algorithmic bias is separating intent from outcome. The engineers who built Amazon's hiring algorithm did not intend to discriminate against women. They were, by all accounts, trying to build an efficient and accurate system. The bias that emerged was not a product of malice but of structure — of the way historical discrimination was encoded in the training data, and the way the system learned from that data without any mechanism to recognize or resist it.

This matters enormously for how we think about accountability. If harm required intent, algorithmic discrimination would be difficult to address — it would require proving that someone, somewhere, intended to discriminate, which is rarely the case with AI systems built by large engineering teams across complex organizational structures. The law has grappled with this problem in the employment context through the doctrine of disparate impact.

Disparate Impact vs. Disparate Treatment

Disparate treatment refers to the intentional differential treatment of individuals on the basis of a protected characteristic. Under Title VII of the Civil Rights Act of 1964 and its subsequent interpretations, it is unlawful for an employer to treat an applicant differently because of their race, sex, national origin, or religion. Proving disparate treatment typically requires demonstrating intent.

Disparate impact is a different and in some respects more powerful doctrine. Under the disparate impact framework, established by the Supreme Court in Griggs v. Duke Power Co. (1971) and subsequently codified in the Civil Rights Act of 1991, a facially neutral employment practice can be unlawful if it has a disproportionate adverse effect on a protected class and cannot be justified by business necessity. The employer need not have intended to discriminate; the statistical outcome is sufficient to establish a prima facie case.

The disparate impact doctrine has direct application to algorithmic hiring tools. An AI résumé screener that systematically disadvantages women, even without explicitly considering gender, can violate Title VII under the disparate impact framework. The employer who uses it — not just the vendor who built it — may bear legal liability. The EEOC's 2023 technical assistance document on AI and employment discrimination made this explicit, confirming that employers cannot escape Title VII liability by outsourcing discriminatory decisions to an AI system.

Legal Context

The legal landscape governing algorithmic discrimination is evolving rapidly. In the United States, the primary frameworks are: Title VII of the Civil Rights Act (employment), the Fair Housing Act (housing), the Equal Credit Opportunity Act (credit), and the Fair Labor Standards Act. These laws were written before AI existed and are being applied to it through regulatory guidance and litigation. The EEOC, CFPB, FDIC, and DOJ have jointly issued guidance affirming that existing civil rights laws apply fully to AI systems.

In the European Union, the General Data Protection Regulation (GDPR) provides individuals with rights relating to automated decision-making, including the right not to be subject to decisions based solely on automated processing that produce legal or similarly significant effects. The EU AI Act, finalized in 2024, goes further, designating AI systems used in employment, credit, and certain other high-stakes contexts as "high-risk" systems subject to mandatory conformity assessments, documentation requirements, and human oversight obligations.

The global variation in legal frameworks is itself a significant challenge for multinational companies, which must navigate different rules in different jurisdictions. An AI hiring tool that is legally compliant in one country may violate anti-discrimination law in another. The EU's requirements are among the most stringent, and many companies are treating EU compliance as a global baseline — though this approach has its own complications, as the categories of protected characteristics and the precise legal standards differ across jurisdictions.

Vocabulary Builder

• Algorithmic bias: Systematic and unfair differences in how an AI system treats individuals or groups, arising from flaws in data, design, or social context.
• Disparate impact: The doctrine that a facially neutral practice can be unlawful if it has a disproportionate adverse effect on a protected class, regardless of intent.
• Disparate treatment: The intentional differential treatment of an individual on the basis of a protected characteristic.
• Protected class: A group of individuals protected by anti-discrimination law, defined by characteristics such as race, sex, national origin, age, or disability.
• Proxy variable: A variable that serves as a substitute measure for a characteristic that is not directly observed; in the context of algorithmic bias, variables that correlate with protected characteristics (e.g., zip code as a proxy for race).
• Feedback loop: A dynamic in which the outputs of a system influence the inputs to that same system, such that biased outputs can produce biased future training data.

Section 7.2: Why AI Systems Become Biased

Algorithmic bias does not emerge from a single source. It accumulates across the entire lifecycle of an AI system, from the initial framing of the problem through the collection of data, the training of models, and the deployment and operation of the resulting system in the real world. Understanding the specific mechanisms through which bias enters AI systems is essential for anyone seeking to prevent or mitigate it. This section provides a systematic taxonomy.

Training Data Bias: The Mirror Problem

The most fundamental source of algorithmic bias is training data that reflects historical discrimination. An AI system learns the patterns present in its training data. If those patterns encode discriminatory outcomes — if the historical data shows that certain groups were systematically denied opportunities, resources, or fair treatment — the AI will learn to replicate those patterns. It is not learning what is right or fair; it is learning what has happened.

Amazon's hiring algorithm is the paradigm case. The tech industry's workforce in the decade from 2004 to 2014 was overwhelmingly male. Women were systematically underrepresented in technical roles. Amazon's historical hiring decisions reflected this pattern. When the model learned from those decisions, it learned that successful Amazon engineers were usually male. Not because male candidates were more qualified — but because the human hiring processes that generated the training data had, for a complex mixture of historical, cultural, and structural reasons, selected men at higher rates. The AI faithfully reproduced what the data showed.

This is the mirror problem: AI systems reflect the world as it has been, not the world as it should be. When the historical record is contaminated by discrimination, AI systems trained on that record will discriminate. The machine does not know this is wrong. It has no conception of fairness or justice. It knows only that certain patterns predicted certain outcomes in the past, and it applies those patterns to the future.

Feature Selection Bias: Choosing the Wrong Variables

Even with clean training data, the choice of input variables — features — can introduce bias. Feature selection bias occurs when the variables chosen to represent candidates, customers, or individuals are themselves correlated with protected characteristics, and when that correlation produces differential outcomes.

Consider a credit scoring model that includes zip code as a feature. Zip code is a powerful predictor of creditworthiness in historical data, because it correlates with income, employment stability, and numerous other economically relevant variables. But zip code also correlates with race, because of decades of racially discriminatory housing policy — redlining, racially restrictive covenants, and systematic exclusion of Black Americans from wealth-building homeownership. An AI credit model that uses zip code as a feature will therefore tend to produce lower credit scores for applicants in predominantly Black neighborhoods, not because of any characteristic of those individuals, but because of the historical discrimination embedded in residential segregation.

The variable is not race. It is zip code. But it functions as a proxy for race, importing racial discrimination into the model under a technically neutral label. This is the proxy variable problem, and it is one of the most common and consequential mechanisms of algorithmic bias.

Label Bias: Garbage In, Garbage Out

Many supervised machine learning systems are trained on labeled data — data in which each example is tagged with a "correct" outcome. In content moderation, human reviewers label content as harmful or acceptable. In medical AI, clinicians label images or records as indicating a particular diagnosis. In hiring, recruiters label applicants as promising or unpromising. If the humans who produce these labels are themselves biased, the AI trained on their labels will learn and amplify that bias.

Research has documented consistent racial disparities in how human reviewers label social media content. Studies have found that content moderation systems disproportionately flag African American Vernacular English (AAVE) as toxic or violating, because the human labelers who created the training data were disproportionately from demographic groups unfamiliar with AAVE and were more likely to misclassify it as problematic. The AI learned from biased human judgments and applied those judgments at scale.

Label bias is particularly insidious because it is difficult to detect by examining the model in isolation. The model is doing exactly what it was trained to do. The problem is in the training labels themselves, which are often assumed to be ground truth when they are actually the product of fallible and potentially biased human judgment.

Feedback Loops: How Bias Compounds Over Time

Algorithmic bias is often not static. Biased outputs can influence future data, creating self-reinforcing cycles that entrench and amplify initial bias. These feedback loops are among the most difficult aspects of algorithmic bias to address, because they cause the problem to grow over time rather than remaining constant.

The canonical example is predictive policing. A predictive policing algorithm uses historical crime data to identify geographic areas where future crimes are most likely to occur. Police departments then concentrate patrols in those areas. More patrols in those areas produce more arrests in those areas — because police are present to observe and respond to incidents that might go undetected in less-patrolled areas. Those additional arrests flow back into the crime database, which now shows even more criminal activity in those areas. The algorithm recommends even more patrol concentration. The cycle continues, with the algorithmic designation of "high-crime" areas becoming increasingly detached from the actual distribution of crime and increasingly reflective of the distribution of policing.

This example will be examined in greater depth in Section 7.5. For now, the key point is that feedback loops can cause relatively modest initial biases to grow substantially over time, making early detection and intervention critical.

Proxy Variables: Race by Another Name

The proxy variable problem deserves additional attention because it is both extremely common and counterintuitive to many practitioners. Organizations frequently believe they have avoided race- or gender-based discrimination by simply removing race and gender from their input features. Amazon's engineers did exactly this — they removed gender from the model's inputs. But the model reconstructed gender from other features that correlated with gender in the training data.

Common proxy variables include: zip code and neighborhood (proxy for race and socioeconomic status), school name and type (proxy for race and socioeconomic status), employment gaps (proxy for caretaking responsibilities, which correlate with gender), word choice and writing style (proxy for gender, cultural background, and education), social network characteristics (proxy for race, class, and geography), and extracurricular activities (proxy for class and cultural background).

The proxy variable problem reveals a fundamental tension in anti-discrimination law and ethics: genuine neutrality requires not just the absence of protected characteristics as explicit features, but the absence of features that reliably predict protected characteristics. In a world where historical discrimination has shaped education, employment, geography, and social networks, this is an extraordinarily demanding standard. Almost any feature of a person's background carries information about their demographic characteristics, because demographic characteristics have shaped that background.

Representation Gaps: Who Isn't in the Data

Many AI systems are trained on data that underrepresents certain populations. Medical AI systems are frequently trained on data from research hospitals that serve predominantly white, middle-class patients, and perform less well on patients from other demographic groups. Speech recognition systems trained predominantly on standard American English perform less well for speakers with different accents. Image classification systems trained on images from one part of the world may fail to generalize to another.

Representation gaps produce technical bias — the model is genuinely less accurate for underrepresented groups — but they also produce a kind of structural neglect. The populations excluded from training data are often precisely those who are most vulnerable and most in need of effective AI tools. Ensuring adequate representation in training data requires deliberate effort and often involves oversampling minority groups, collecting new data from underrepresented populations, and explicitly testing performance across demographic subgroups.

Optimization Target Bias: What Are We Maximizing?

Every AI system is optimized to maximize some objective function — to predict accurately, to maximize clicks, to minimize cost, to maximize revenue. The choice of optimization target can itself introduce bias. A clinical prediction model optimized to minimize cost may end up deprioritizing expensive treatments for groups that have historically received less care, because the training data shows that those groups "do not need" those treatments — not because they actually need them less, but because historical access disparities mean they received them less.

A famous example is the Optum health insurance algorithm studied by Obermeyer et al. (2019). The algorithm used healthcare costs as a proxy for health needs — a reasonable choice, since sicker patients tend to require more expensive care. But Black patients faced significant barriers to healthcare access, meaning they had historically incurred lower costs even when facing equivalent health conditions. The algorithm therefore systematically underestimated the health needs of Black patients, recommending them for fewer care management programs at every level of actual health risk.

Section 7.3: Types of Bias by Stage of the ML Pipeline

Bias does not enter AI systems at a single moment. It accumulates across every stage of the machine learning development pipeline, from the initial framing of the problem through deployment and long-term monitoring. Understanding this staged accumulation is essential for knowing where to intervene.

Stage 1: Problem Formulation — Whose Problem Is Being Solved?

Every AI project begins with a decision about what problem to solve. That decision is rarely neutral. The choice of problem reflects the priorities of those with the power and resources to commission AI systems — which typically means organizations with existing market power, access to capital, and entrenched interests. The questions that get automated are those that serve the interests of these institutions; the questions that would serve marginalized groups often do not get asked.

Amazon wanted to automate hiring to serve its own efficiency interests. No one commissioned an AI system to help job seekers identify discriminatory employers. The predictive policing systems deployed widely in American cities were commissioned by police departments seeking operational efficiency; no corresponding AI systems were built to predict or detect police misconduct.

Problem formulation also involves consequential choices about what to optimize and how to measure success. These choices embed value judgments that are often invisible because they seem technical. Defining "successful hire" as someone who stays at Amazon for more than two years is a value judgment that may systematically disadvantage candidates with caregiving responsibilities who might need more flexibility. Defining "high-risk" in a recidivism prediction tool is a value judgment with enormous consequences for individuals whose freedom depends on where they fall on the scale.

Stage 2: Data Collection — Who Is Represented?

Once a problem is framed, data must be collected or assembled. Data collection choices determine whose experiences are reflected in the AI system. Data that systematically excludes certain populations — because those populations lack internet access, have lower rates of smartphone use, are underrepresented in the records of formal institutions, or have historical reasons to distrust data-collection initiatives — will produce systems that serve those populations poorly.

Historical data is particularly problematic, as Amazon's case illustrates. When historical processes were discriminatory, historical data encodes that discrimination. Using historical data to train future-oriented AI systems perpetuates past injustice into future decisions.

Stage 3: Data Labeling and Annotation — Who Labels? What Are Their Assumptions?

Human labelers produce the "ground truth" on which supervised learning systems depend. Those labelers bring their own backgrounds, assumptions, and biases to their work. When labeling workforces are homogeneous — as they often are, because labeling is frequently crowdsourced through platforms whose workers are drawn from particular demographic pools — their shared assumptions can systematically shape the labels in ways that disadvantage groups outside that pool.

The standard of care requires explicit attention to labeler diversity, clear annotation guidelines that surface and mitigate known sources of bias, inter-rater reliability measurement, and regular audits of labels for systematic patterns that might reflect labeler bias.

Stage 4: Feature Engineering — What Variables Are Included and Why?

Feature engineering — the process of selecting, transforming, and constructing the input variables for an AI model — is a technically intensive process that is rarely examined through a fairness lens. Yet the choice of features is one of the most consequential determinants of whether a model will exhibit bias.

The key questions are: Is this feature correlated with protected characteristics? If so, what is the mechanism? Does including it constitute the use of a proxy variable that effectively imports protected-characteristic discrimination into the model? Does the feature perform differently for different demographic groups in ways that could introduce bias?

Stage 5: Model Training — What Is Being Optimized?

During training, choices about model architecture, hyperparameters, regularization, and loss functions shape how the model learns from data. Some of these choices have systematic effects on fairness. Models that optimize for overall accuracy can sacrifice accuracy for minority-class examples in order to improve performance on the majority class. Regularization techniques that prevent overfitting may paradoxically harm performance on underrepresented groups, because the model is pushed toward simpler patterns that capture majority-group dynamics well. The decision about which optimization objective to use, and how to handle class imbalance, directly affects which groups bear the cost of errors.

Stage 6: Model Evaluation — Which Population? Which Metrics?

A model may appear to perform well on overall metrics while concealing dramatic disparities at the group level. Standard evaluation practices often aggregate performance across all test examples, masking differential error rates across demographic subgroups. Disaggregated evaluation — reporting performance metrics separately for each relevant demographic group — is not yet standard practice in many organizations, though it is required by the EU AI Act for high-risk systems and is increasingly recommended by regulatory guidance in the United States.

Stage 7: Deployment — Context Changes Everything

When a model is deployed into the real world, the social context in which it operates can differ substantially from the context in which it was developed and tested. A model trained and evaluated in one city may perform very differently when deployed in another with different demographic composition. A model designed for one use case may be repurposed for another in ways that introduce new bias risks. The history of algorithmic bias is full of systems deployed in contexts for which they were not designed and for which their bias properties were not evaluated.

Stage 8: Monitoring — Is Post-Deployment Performance Being Tracked?

Even a model that was fair at deployment can become biased over time, as the social world changes and as feedback loops reshape the data distributions the model operates on. Ongoing monitoring of model performance across demographic subgroups is essential. Yet post-deployment monitoring is among the least commonly implemented practices in the field. Many organizations deploy models and treat them as finished products rather than ongoing systems that require continuous oversight.

Table 7.1: Bias Risk by Pipeline Stage

Section 7.4: High-Stakes Domains and Their Specific Bias Patterns

Algorithmic bias appears across virtually every domain in which AI systems are deployed. But it is not uniformly distributed in its consequences. In some applications, biased outputs cause inconvenience — a miscalibrated recommendation, an irrelevant advertisement. In others, they determine whether someone gets a job, a loan, medical care, or their freedom. This section surveys the major high-stakes domains, with attention to the specific patterns of bias that have been documented in each.

Employment

The employment domain is where algorithmic bias has received the most attention, in part because the Amazon case made it concrete and legible to a broad audience. AI tools are now used at various stages of the hiring process: to screen résumés, to analyze video interviews, to administer and score cognitive assessments, and to predict job performance and attrition. They are also used post-hire, to evaluate performance, predict promotion suitability, and identify flight risks.

Amazon's tool was designed to screen résumés and identify top candidates. As detailed in the opening hook, it learned to penalize female applicants because the training data — ten years of actual Amazon hiring decisions — reflected the gender imbalances of the tech industry. The mechanism involved both direct proxies (references to women's organizations and colleges) and more subtle language patterns.

The Amazon case is not isolated. Researchers have documented bias in automated video interview analysis tools, which evaluate factors including facial expressions, tone of voice, and word choice to predict candidate quality. These tools have been found to exhibit racial disparities in accuracy and scoring. Several large employers have quietly stopped using them following scrutiny. The EEOC's 2023 guidance explicitly flagged these tools as high-risk for Title VII violations.

The employment domain is particularly consequential for bias analysis because employment decisions are both high-stakes (affecting income, housing, healthcare access, and social status) and high-volume (millions of decisions per year, many now automated). A small percentage bias rate, applied millions of times, produces enormous aggregate harm.

Criminal Justice

The criminal justice system has become one of the most discussed arenas for algorithmic bias, driven primarily by investigative journalism and research on risk assessment instruments — tools used to predict an individual's likelihood of reoffending, which inform decisions about bail, sentencing, and parole.

The most scrutinized of these tools is COMPAS (Correctional Offender Management Profiling for Alternative Sanctions), developed by Northpointe and used in numerous US jurisdictions. In 2016, ProPublica published an investigation titled "Machine Bias" that analyzed COMPAS scores and criminal records for more than 7,000 people in Broward County, Florida. The investigation found that the tool falsely flagged Black defendants as future criminals at nearly twice the rate it falsely flagged white defendants. It also falsely flagged white defendants as low-risk at a higher rate. The same score meant different things for defendants of different races.

Northpointe disputed the analysis, arguing that the tool was calibrated — that is, a given score corresponded to the same probability of reoffending regardless of race. This sparked a vigorous academic debate about the mathematical properties of fairness. As researchers Chouldechova (2017) and Kleinberg et al. (2016) demonstrated formally, several common fairness criteria are mathematically incompatible when base rates differ across groups — a finding with profound implications for any domain in which historical discrimination has produced different outcome distributions across demographic groups.

The criminal justice domain raises the stakes of bias to their maximum: a biased risk assessment tool does not just sort résumés — it affects whether a person is detained before trial, how long their sentence is, and when they are eligible for parole.

Financial Services

Credit scoring, loan approval, and insurance pricing all depend heavily on algorithmic systems, and all have documented bias patterns. The proxy variable problem is particularly acute in credit, where zip code, credit history, and banking relationship are all powerful predictors of creditworthiness that also correlate with race because of historical discrimination in financial inclusion.

The CFPB and other regulators have documented persistent racial disparities in mortgage loan approval rates, auto loan rates, and credit limit assignment. The Equal Credit Opportunity Act prohibits discrimination in credit on the basis of race, sex, and other protected characteristics. The Act applies to algorithmic systems as well as human decision-makers. In 2022, the CFPB issued guidance noting that lenders cannot avoid ECOA liability by delegating credit decisions to algorithmic models.

The "digital redlining" problem — in which algorithmic credit systems replicate the geographic exclusion of traditional redlining — remains an active enforcement concern. Several major financial institutions have faced regulatory action for algorithmic credit discrimination in recent years.

Healthcare

Medical AI presents perhaps the most consequential manifestation of representation gaps. AI tools are now used for diagnosis support, treatment recommendation, risk stratification, and resource allocation across healthcare systems. When these tools are developed on data that underrepresents certain patient populations, they perform less well — sometimes dangerously less well — for those populations.

The Optum algorithm example (Section 7.2) is the best-documented case: a risk-stratification tool used by major health systems to identify patients needing intensive care management systematically underestimated the health needs of Black patients, because it used healthcare costs as a proxy for health need, and Black patients had lower historical costs due to access barriers rather than lower need.

Dermatology AI provides another example. Several widely cited AI diagnostic systems for skin conditions were trained predominantly on images of lighter-skinned patients. Their diagnostic accuracy for darker-skinned patients is substantially lower — a significant concern given that certain skin conditions, including some forms of skin cancer, present differently on darker skin tones and are already subject to diagnostic disparities.

Image Recognition and Facial Analysis

Facial recognition technology has produced some of the most clearly documented and publicly visible examples of algorithmic bias, driven largely by the work of researcher Joy Buolamwini and her collaborators. In a 2018 study known as the Gender Shades project, Buolamwini and Timnit Gebru evaluated commercial facial analysis systems from Microsoft, IBM, and Face++ for accuracy in classifying faces by gender. The results revealed dramatic disparities: overall accuracy was high, but accuracy for darker-skinned women was dramatically lower than for lighter-skinned men. The most accurate systems had an error rate of 0.8 percent for light-skinned men and 34.7 percent for dark-skinned women — a 43-fold difference.

The subsequent NIST Face Recognition Vendor Technology (FRVT) evaluation, examined in detail in the accompanying case study, extended these findings across dozens of commercial systems and multiple task types. NIST found false positive rates 10 to 100 times higher for Black and Asian faces than for white faces in one-to-one matching tasks. These disparities have direct consequences when facial recognition is used in law enforcement: an algorithm with much higher false positive rates for Black faces will produce false identifications — leading to wrongful investigation and, as documented cases show, wrongful arrest — at much higher rates for Black individuals.

Language Models

Large language models — including GPT-family models, Claude, and their competitors — exhibit their own bias patterns, including stereotype amplification, differential toxicity generation, and cultural bias. Research has documented that language models are more likely to associate certain professions with particular genders, to generate toxic content when prompted with certain demographic groups' names, and to perform less well in languages that were underrepresented in their training corpora.

Language model bias is addressed in depth in later chapters. For present purposes, the key point is that the same fundamental mechanisms — biased training data, representation gaps, optimization for majority-class performance — produce bias in language models as in other AI systems, but the outputs take the form of text rather than decisions, making the bias simultaneously more diffuse and more legible to lay audiences.

Section 7.5: The Feedback Loop Problem

Among the most consequential and least understood aspects of algorithmic bias is its tendency toward self-reinforcement. Many biased AI systems do not merely perpetuate historical discrimination at a constant rate; they generate conditions that cause the discrimination to intensify over time. This self-reinforcing dynamic is the feedback loop problem, and understanding it is essential for understanding why early detection and intervention are so critical.

The Mechanism

A feedback loop in the context of algorithmic bias occurs when the outputs of a biased system influence the data that the system — or a future version of it — will be trained on. The biased output creates a biased environment, which generates biased new data, which reinforces the system's initial bias. Over successive cycles, what began as a modest initial disparity can grow into a severe structural one.

Predictive Policing: A Feedback Loop in Detail

Predictive policing algorithms are designed to help police departments allocate patrol resources more efficiently by predicting where crimes are likely to occur. They are trained on historical crime data — records of where arrests and reported crimes have occurred in the past. This seems reasonable: past crime patterns should, in principle, provide useful information about future crime risk.

But historical crime data does not measure where crimes actually occur. It measures where crimes are detected and recorded. In American cities, crime detection rates have historically been much higher in heavily policed neighborhoods, which tend to be poorer and more heavily populated by Black and Latino residents, than in wealthier and whiter neighborhoods. Crimes that go unobserved go unrecorded. The historical crime data therefore reflects the distribution of policing at least as much as the distribution of crime.

When a predictive policing algorithm is trained on this data, it identifies heavily policed neighborhoods as "high-crime" and recommends concentrating patrol resources there. The concentrated patrols observe and record more crimes — including low-level offenses that might go undetected in less-patrolled areas. These additional recorded crimes flow back into the crime database. The algorithm, retrained on this expanded database, recommends even more concentration in the already heavily patrolled neighborhoods. The system is not discovering crime; it is generating the conditions for its own predictions to appear accurate.

The feedback loop operates independently of any individual officer's bias. The algorithm and the patrol allocations it drives could be implemented by officers with no conscious prejudice whatsoever, and the loop would still tighten. Structural discrimination does not require discriminatory intent.

The Credit Feedback Loop

Credit scoring operates similarly. Credit bureaus and lenders use AI models to assess credit risk. Applicants with lower predicted risk receive credit; applicants with higher predicted risk are denied. Credit history — the record of past borrowing and repayment — is one of the most important inputs to credit scoring models. But credit history requires access to credit. Applicants who are denied credit cannot build credit history. Their next application to a credit scoring model will again predict high risk, because their thin credit file is an indicator of risk, regardless of the underlying reason for it.

This creates a persistent disadvantage for first-time credit seekers and for populations that have been historically excluded from formal credit markets — populations that, because of discriminatory lending practices, include a disproportionate share of Black, Latino, and Native American individuals. The feedback loop perpetuates financial exclusion across generations.

The Recruitment Feedback Loop

Amazon's case illustrates a hiring feedback loop. The algorithm was trained to identify candidates who resembled Amazon's historical successful employees. Those employees were overwhelmingly male, because the tech industry had for decades hired predominantly male engineers. The algorithm learned this pattern and applied it in screening. Candidates who were rated highly by the algorithm and ultimately hired were disproportionately male — which generated another cohort of successful Amazon employees who were disproportionately male, which would have reinforced the algorithm's pattern on the next training run.

If Amazon had continued using the algorithm and periodically retraining it on its expanding employee base, the male skew would have become self-sustaining. The algorithm would have kept recommending men because the data kept showing that successful Amazon engineers were men, because the algorithm kept recommending men.

Breaking Feedback Loops

Interrupting feedback loops requires both technical and organizational intervention. Technically, it requires preventing biased outputs from flowing back into training data — which means implementing mechanisms to identify and quarantine outputs that may be biased, or to supplement algorithmic outputs with data collected by other means. It also requires periodic retraining on data that is explicitly collected to correct historical imbalances, rather than simply to reflect historical patterns.

Organizationally, breaking feedback loops requires the explicit recognition that the system's outputs are not neutral observations of reality but interventions in reality that shape future outcomes. This requires a level of institutional self-awareness that is uncommon in organizations that have adopted algorithmic systems primarily for efficiency reasons.

The Compounding Problem

Real-world environments often involve multiple AI systems operating simultaneously and interacting with one another. A biased hiring algorithm produces a biased workforce. That biased workforce makes biased promotion decisions. A biased credit scoring model produces a biased population of creditworthy individuals. A biased healthcare risk stratification tool produces biased patterns of care delivery. The accumulated effect of multiple interacting biased systems is greater than the sum of their individual biases. Individuals who are disadvantaged by one system are often disadvantaged by several simultaneously — a compounding effect that intensifies harm for those already most vulnerable.

Section 7.6: Intersectionality and Multiple Bias Dimensions

Legal scholar Kimberlé Crenshaw introduced the concept of intersectionality in 1989 to describe the way that systems of discrimination based on race and gender interact and overlap in ways that cannot be captured by analyzing either dimension alone. The concept was developed in the context of employment discrimination law — Crenshaw showed that Black women faced discrimination that was neither reducible to racial discrimination (which Black men also faced) nor to gender discrimination (which white women also faced), but was a distinct form of harm arising from the combination of both identities.

Applied to algorithmic bias, intersectionality reveals a critical gap in standard fairness analysis. Most fairness testing considers one dimension of potential bias at a time — evaluating whether the model performs equally well for men and women, or for Black and white individuals, but not considering how race and gender interact. This single-axis analysis can give a false picture of overall fairness.

The Gender Shades Demonstration

Joy Buolamwini's Gender Shades study provides the clearest empirical demonstration of this principle. When Buolamwini and Gebru evaluated commercial facial analysis systems, they found that the bias was concentrated not in either race or gender alone but at their intersection. Systems that appeared reasonably accurate when broken down by race alone — grouping all women together and all men together — or by gender alone concealed dramatic disparities for dark-skinned women specifically.

Consider the implications. A company evaluating its facial recognition vendor by testing accuracy across racial groups and accuracy across gender groups, separately, might observe acceptable aggregate performance. Only disaggregating by both race and gender simultaneously — examining the four cells of the matrix: light-skinned men, dark-skinned men, light-skinned women, dark-skinned women — would reveal the extreme underperformance for the most vulnerable subgroup. Single-axis testing is not just incomplete; it can actively mislead.

Why Intersectional Analysis Is Harder

Intersectional fairness analysis requires disaggregating performance across combinations of characteristics, which means needing sufficient sample sizes in each combination — sufficient numbers of dark-skinned women, or young Black men, or elderly Latinas — to draw statistically meaningful conclusions. As the number of characteristics considered grows, the number of intersectional subgroups grows exponentially, and the available sample in each cell shrinks. For rare intersections of characteristics, there may not be enough data to evaluate performance at all.

This creates a genuine methodological challenge. But it does not excuse ignoring the problem. It means that organizations must be thoughtful about which intersections are most likely to matter for their specific application, must invest in collecting sufficient data to enable intersectional analysis for the groups most likely to be affected, and must acknowledge the limits of their testing with appropriate humility.

The Business Implication

The business implication of intersectionality for algorithmic bias is straightforward but often overlooked: single-axis fairness testing is insufficient and can produce false confidence. An organization that has tested its AI system for racial bias and gender bias separately, and found acceptable aggregate performance on both, cannot conclude that the system is fair. The worst outcomes may be concentrated precisely at intersections that neither test examined.

The groups most likely to be harmed by intersectional bias are also, by definition, often the smallest and most marginalized — the groups with least political voice and least access to remedies. This makes intersectional bias both harder to detect and more morally urgent to address.

Section 7.7: Distinguishing Bias from Discrimination — Legal Implications

Not all algorithmic bias is illegal. Some of it is a civil rights violation. Understanding the distinction is essential for business professionals who bear responsibility for the AI systems their organizations deploy.

When Algorithmic Bias Becomes Illegal Discrimination

An AI system that systematically underperforms for Black users may be ethically troubling without being illegal. An AI system that systematically produces adverse employment decisions for Black applicants is likely to violate Title VII of the Civil Rights Act. The line between the two is not always obvious, and it has been the subject of significant regulatory and judicial attention in recent years.

The key legal frameworks in the United States are:

Title VII of the Civil Rights Act (1964) prohibits employment discrimination on the basis of race, color, religion, sex, and national origin. It applies to hiring, firing, compensation, and other terms and conditions of employment. Under the disparate impact doctrine established in Griggs v. Duke Power (1971), a facially neutral employment practice that has a disproportionate adverse effect on a protected class is presumptively unlawful unless the employer can demonstrate that the practice is job-related and consistent with business necessity.

The Fair Housing Act (1968) prohibits discrimination in the sale, rental, and financing of housing on the basis of race, color, national origin, religion, sex, familial status, and disability. It applies to algorithmic tools used in tenant screening, mortgage approval, and home valuation.

The Equal Credit Opportunity Act (1974) prohibits discrimination in credit transactions on the basis of race, color, religion, national origin, sex, marital status, age, and other characteristics. The CFPB has affirmed that ECOA applies to algorithmic credit scoring systems.

The Americans with Disabilities Act (1990) has implications for AI systems used in employment that may discriminate against individuals with disabilities, including some neurodivergent conditions.

The EEOC's 2023 AI Guidance

In May 2023, the EEOC issued technical assistance titled "Artificial Intelligence and Algorithmic Fairness Initiative," which made several important clarifications. First, it confirmed that employers who use algorithmic tools in employment decisions remain responsible for any discriminatory impacts of those tools under Title VII — they cannot outsource liability to the AI vendor. Second, it identified specific AI tools as high-risk for disparate impact claims, including résumé screening tools, video interview analysis tools, and personality assessment algorithms. Third, it noted that employers who rely on AI tools should conduct adverse impact analyses to identify any discriminatory patterns before deploying those tools.

The EU Legal Framework

The European Union's approach to algorithmic discrimination is both broader and more prescriptive than the US framework. The EU's non-discrimination directives prohibit discrimination on the basis of race, ethnic origin, sex, religion or belief, age, sexual orientation, and disability across employment, access to goods and services, and other domains. The GDPR provides rights relating to automated decision-making, including the right to explanation and the right to human review of automated decisions.

The EU AI Act, which entered into force in 2024, establishes a risk-based framework for AI regulation. AI systems used in employment, credit, education, law enforcement, and certain other high-stakes domains are classified as "high-risk" and subject to mandatory requirements including fundamental rights impact assessments, transparency documentation, data governance standards, human oversight mechanisms, and registration in a public EU database.

Liability: Who Answers for the Algorithm?

The question of liability — who is legally responsible when an AI system discriminates — is not fully settled but is increasingly being answered in ways that hold deploying organizations accountable. Under US civil rights law, the employer who uses an AI hiring tool is generally the responsible party, not the AI vendor. The employer selects the tool, deploys it in their hiring process, and bears the consequences of its outcomes.

This creates strong incentives for employers to conduct due diligence on AI vendors — to demand documentation of fairness testing, to conduct their own adverse impact analyses, and to maintain oversight mechanisms that can detect discriminatory patterns before they result in regulatory action or litigation.

The Loomis Case

State v. Loomis (Wisconsin, 2016) presented one of the first major judicial examinations of algorithmic bias in criminal sentencing. Eric Loomis challenged his sentence, arguing that the use of the COMPAS recidivism prediction tool violated his due process rights because the algorithm was proprietary and he could not examine the basis for his risk score. The Wisconsin Supreme Court upheld the sentence, ruling that COMPAS had been used as one factor among several and that Loomis had been given adequate procedural protections. But the case raised fundamental questions about the right to confront algorithmic decisions that affect one's liberty — questions that courts and legislatures continue to grapple with.

Section 7.8: The Business Imperative — Bias as Business Risk

For business leaders, algorithmic bias is not primarily a philosophical concern — it is a concrete and multidimensional business risk. Organizations that deploy biased AI systems face potential harm across several distinct risk dimensions.

Reputational Risk

Reputational damage from AI bias incidents can be severe and lasting. The press coverage of Amazon's hiring algorithm, the ProPublica investigation of COMPAS, the reporting on biased facial recognition leading to wrongful arrests, and the exposure of health insurance algorithms that systematically undervalued Black patients' health needs — each of these stories attracted broad attention and sustained scrutiny. In the age of algorithmic accountability journalism, investigative reporters are actively seeking and publishing AI bias stories. A bias incident that becomes public can damage brand reputation with customers, employees, and investors.

Several major companies have withdrawn AI products under reputational pressure. IBM and Microsoft voluntarily halted sales of facial recognition to law enforcement following the 2020 civil rights protests and the attendant attention to racial bias in these systems. These withdrawals, while framed as ethical choices, were also strategic responses to reputational risk.

Legal Risk

The regulatory environment for algorithmic discrimination is tightening. The EEOC, CFPB, FTC, and DOJ have all issued guidance and initiated enforcement actions related to algorithmic bias. Several state and local governments have enacted laws requiring algorithmic impact assessments, transparency, and in some cases independent auditing of high-risk AI systems. New York City's Local Law 144, which took effect in 2023, requires employers to conduct annual bias audits of automated employment decision tools and to disclose the results to job candidates.

Class action litigation over algorithmic discrimination is growing. Cases have been filed against employers using biased résumé screening tools, against financial institutions using discriminatory credit algorithms, and against insurers using biased underwriting models. Regulatory fines and litigation costs from these actions can be substantial.

Operational Risk

Biased AI systems often perform poorly as measured by their own stated objectives. A hiring algorithm that systematically screens out women from consideration for technical roles is also screening out a large portion of the qualified candidate pool — women who could perform the job at the level the company requires. A credit model that systematically underestimates creditworthiness for certain groups is making bad lending decisions, not just discriminatory ones. A health risk stratification model that underestimates needs for certain patients is not just unfair; it is inaccurate. Bias and poor performance often coincide, because both stem from the same root causes: incomplete data, flawed assumptions, and inadequate testing.

The Diversity Dividend

Research consistently finds that teams with greater cognitive and experiential diversity — including demographic diversity — are better at identifying assumptions and blind spots that homogeneous teams miss. In the context of AI development, this means that diverse development teams are more likely to notice when a system performs poorly for groups different from the majority of the team. They are more likely to ask whether the training data represents all affected populations, to flag proxy variables as problematic, and to test performance across demographic subgroups.

The diversity dividend is not a soft, feel-good argument — it is a concrete mechanism for risk reduction. Organizations with homogeneous AI development teams are at higher risk of deploying biased AI systems and at higher risk of failing to detect that bias before it causes harm.

Section 7.9: Early Detection and the Culture of Bias Awareness

The most important insight about algorithmic bias is also the most actionable: it is far easier to detect and address before deployment than after. An organization that builds a culture of bias awareness — that systematically incorporates fairness analysis at every stage of the AI development pipeline — will consistently produce less biased AI systems than one that treats bias as an afterthought or a regulatory compliance checkbox.

Building Organizational Practices

Early detection requires that fairness analysis be institutionalized, not discretionary. This means:

Diverse development teams. As noted in Section 7.8, diverse teams catch more bias. This is not a pipeline problem that can be deferred; it is a hiring, retention, and inclusion challenge that must be addressed with the same urgency as any other strategic capability gap.

Fairness specifications during problem formulation. Before any data is collected or model trained, the development team should explicitly specify what fairness means for this particular system: which demographic groups matter, which fairness metrics are relevant, and what performance thresholds are acceptable across groups. These decisions should be made before seeing the data, not after — making them afterward creates strong temptations to define fairness in whatever way the existing system already achieves.

Pre-deployment fairness testing. Disaggregated evaluation across relevant demographic subgroups should be a mandatory step in the model evaluation process, with explicit pass/fail criteria for each group. This requires having demographic information in the test dataset — which requires deliberate effort to collect and maintain.

Red-teaming for bias. Red-teaming — the practice of deliberately trying to find failure modes in a system — should explicitly target bias. A dedicated group should attempt to find demographic subgroups for which the system fails, to construct inputs that expose differential treatment, and to simulate deployment in contexts the development team may not have considered.

Post-Deployment Reporting Channels

Even the best pre-deployment testing will miss some bias patterns, because the deployment environment is inevitably more complex and varied than the test environment. Organizations need reliable channels through which users, employees, and affected community members can report suspected bias and receive genuine responses. These channels must be trustworthy — reporters must believe their reports will be taken seriously and acted upon, not dismissed or retaliated against — or they will go unused.

The Danger of "We Tested It and It Was Fine"

One of the most common and most dangerous responses to algorithmic bias concerns is the claim that the system was tested before deployment and found to be fair. This claim deserves careful scrutiny. The critical questions are: Tested on what population? Using which fairness metrics? At what level of disaggregation? By whom? With what standard for "fine"?

A test conducted on the same population used for training, using aggregate accuracy as the sole metric, with no disaggregation by demographic subgroup, and by the team that built the system, does not establish fairness. It establishes that the system performs adequately on average for the population it was designed for. This is a very different and much weaker claim.

The standard of care is evolving, but increasingly the expectation is: disaggregated evaluation across all relevant demographic groups; independent auditing or at least internal red-teaming; explicit documentation of known limitations and performance disparities; and ongoing post-deployment monitoring. Meeting this standard is not simple or cheap. But the cost of not meeting it — in harm to affected individuals, in regulatory action, in reputational damage — can be far higher.

The Role of Affected Communities

Perhaps the most important and most frequently neglected element of bias detection is the involvement of affected communities — the people who will actually experience the system's outputs. Affected communities often know things that development teams do not: they know which classifications are experienced as stigmatizing, which proxy variables are particularly harmful in their context, which use cases the developers did not anticipate, and which harms are most acute. Their knowledge is irreplaceable.

Engaging affected communities in bias detection is not just instrumentally valuable; it is ethically required by basic norms of democratic accountability. Systems that affect people's lives should be built with input from those people — not just the organizations that deploy them and the engineers who build them. This principle is increasingly recognized in regulatory frameworks: the EU AI Act's fundamental rights impact assessment explicitly requires consideration of the effects on affected groups, and engagement with civil society is increasingly expected as a component of responsible AI development.

The chapters that follow will build on this foundation, exploring specific methods for measuring bias (Chapter 9), strategies for mitigation (Chapters 10–12), domain-specific applications in employment (Chapter 13), criminal justice (Chapter 14), financial services (Chapter 15), and healthcare (Chapter 16), and the organizational and governance structures needed to sustain responsible AI development over time (Chapters 17–20). The concepts developed in this chapter — the taxonomy of bias sources, the pipeline-stage analysis, the feedback loop problem, intersectionality, the legal framework, and the organizational practices for early detection — will provide the analytical vocabulary for all that follows.

Discussion Questions

1. Amazon's engineers removed gender from the hiring algorithm's input features and the system still developed gender bias. What does this tell us about the limitations of "removing the sensitive variable" as a bias mitigation strategy? What would a more comprehensive approach have required?

2. The predictive policing feedback loop demonstrates that biased AI systems can be self-reinforcing. How does this change the moral urgency of early detection? If you were advising a city considering a predictive policing system, what safeguards would you require before it was deployed?

3. The COMPAS case revealed a mathematical tension: Northpointe argued the tool was "calibrated" (scores meant the same probabilities for all races), while ProPublica argued it produced unfair false positive rates. These two fairness criteria are mathematically incompatible when recidivism base rates differ across groups. Who do you think was right, and why? What does this tell us about the limits of purely technical definitions of fairness?

4. Consider the disparate impact doctrine. Should a facially neutral AI system that produces statistically different outcomes for different demographic groups be presumptively unlawful, even without any evidence of discriminatory intent? What are the strongest arguments on each side?

5. Joy Buolamwini's Gender Shades research found that the worst-performing demographic group — dark-skinned women — was also the smallest group in the training data. Is there a general relationship between data representation and bias concentration? What are the organizational implications?

6. Section 7.9 argues that affected communities should be involved in bias detection and AI system design. What practical mechanisms would allow this? What challenges — power imbalances, confidentiality, scale — must be addressed for such participation to be genuine rather than performative?

7. Amazon was criticized not just for building a biased system but for discovering the bias and failing to disclose it publicly or reach out to affected candidates. What disclosure obligations, if any, should organizations have when they discover their AI systems have produced biased outcomes? How should those obligations differ depending on the domain and severity of harm?

Chapter 7 continues in the accompanying case studies. See Case Study 7.1 (Amazon's Hiring Algorithm) and Case Study 7.2 (NIST Facial Recognition Findings) for detailed examinations of the primary examples introduced in this chapter. Proceed to Chapter 8 for in-depth treatment of bias sources in training data.