Chapter 7: Quiz

Understanding Algorithmic Bias

Instructions: This quiz covers the material in Chapter 7, including the main chapter text and both case studies. Allow approximately 45–60 minutes. Answer all questions. For applied scenario questions, write at least 150 words per response.

Part I: Multiple Choice (8 questions, 2 points each)

Question 1 Amazon's hiring algorithm was trained on résumés submitted between 2004 and 2014. Its gender bias most directly resulted from:

a) Engineers intentionally designing the algorithm to prefer male candidates b) Gender being explicitly included as an input feature in the model c) The model learning from historical hiring decisions made in an industry with male-dominated hiring patterns d) A technical error in the machine learning code that caused incorrect weighting of features

Correct Answer: C Explanation: The model was not designed to discriminate; it learned gender-correlated patterns from historical hiring data that reflected the tech industry's male-dominated hiring practices over the preceding decade. Gender was not an explicit input — the bias emerged from the correlation between gender and historical hiring outcomes embedded in the training labels.

Question 2 Which of the following best defines "disparate impact" in the context of anti-discrimination law?

a) An employer intentionally treating a job applicant differently because of their race b) A facially neutral practice that produces statistically disproportionate adverse effects on a protected class, regardless of intent c) An AI system that explicitly uses race as an input variable in its decision-making d) Differential accuracy in AI system performance that is detected during pre-deployment testing

Correct Answer: B Explanation: Disparate impact doctrine, established in Griggs v. Duke Power (1971), holds that a facially neutral practice can be unlawful if it disproportionately harms a protected class without business justification, regardless of discriminatory intent. This is the key legal framework for assessing facially neutral AI systems that produce unequal outcomes.

Question 3 In the NIST FRVT 2019 evaluation, which of the following was a primary finding about commercial facial recognition systems?

a) All major commercial systems performed within acceptable accuracy tolerances across all demographic groups b) Only systems developed outside the United States showed demographic accuracy disparities c) False positive rates were 10 to 100 times higher for Black and Asian faces than for white faces in one-to-one verification tasks d) The demographic disparities identified were primarily attributable to image quality differences rather than algorithmic bias

Correct Answer: C Explanation: NIST found that the vast majority of the more than 100 algorithms tested exhibited substantially higher false positive rates for Black and Asian faces compared to white faces in one-to-one verification. This was a systematic finding across the industry, not limited to particular vendors or non-US developers.

Question 4 The concept of "emergent bias" in AI systems refers to:

a) Bias that results from errors in the algorithm's mathematical formulation b) Bias that arises from the interaction between a system and the social context in which it operates, which may differ from the context in which it was developed c) Bias that emerges gradually in a model as it processes more data over time d) Bias that is detected only after the system has been deployed and is processing real-world inputs

Correct Answer: B Explanation: Emergent bias is distinct from technical bias. It arises when a system designed for one context is deployed in another, or when users interact with the system in ways that produce biased outcomes even if the system itself contains no technical flaws. A hiring algorithm designed for one cultural context may exhibit emergent bias when used globally.

Question 5 In the predictive policing feedback loop described in Chapter 7, the mechanism of self-reinforcement works primarily because:

a) Police officers are consciously biased against certain neighborhoods b) The algorithm makes direct recommendations that are always followed without human review c) Increased police presence in flagged areas produces more recorded arrests, which flows into crime databases, causing the algorithm to recommend even more concentration in those areas d) The algorithm is regularly retrained on data that has been deliberately falsified by vendors

Correct Answer: C Explanation: The feedback loop operates through the mechanism of differential crime detection: more police in an area produce more arrests, which the crime database records as higher crime rates, which the algorithm interprets as evidence supporting further patrol concentration. The loop operates even without individual bias or deliberate manipulation.

Question 6 Kimberlé Crenshaw's concept of intersectionality, applied to algorithmic bias, means that:

a) Multiple separate AI systems can interact to compound bias across different life domains b) Bias operating simultaneously along multiple dimensions (e.g., race and gender) can produce forms of harm that single-axis analysis fails to detect c) Intersecting data sources from multiple organizations increase the risk of privacy violations d) The most accurate fairness metric is always one that considers the intersection of all possible protected characteristics simultaneously

Correct Answer: B Explanation: Crenshaw developed intersectionality to describe how discrimination based on race and gender interacts to produce unique harms for Black women that cannot be captured by analyzing either dimension alone. Applied to AI, this means that testing a system separately for racial bias and gender bias may miss severe disparities concentrated at their intersection — as demonstrated by the Gender Shades research.

Question 7 Under the EU AI Act, AI systems used in employment are classified as:

a) Prohibited AI practices requiring immediate cessation b) Limited-risk systems requiring transparency disclosures only c) High-risk systems subject to conformity assessments, documentation requirements, and human oversight obligations d) Unregulated systems pending further guidance from the European Commission

Correct Answer: C Explanation: The EU AI Act classifies AI systems used in employment contexts — including recruitment, selection, promotion, and performance evaluation — as high-risk systems. This classification triggers mandatory requirements including fundamental rights impact assessments, data governance standards, transparency documentation, human oversight mechanisms, and registration in the EU database.

Question 8 The Optum health algorithm studied by Obermeyer et al. (2019) systematically underestimated the health needs of Black patients primarily because:

a) The algorithm explicitly used race as a negative weighting factor b) Healthcare cost data was used as a proxy for health need, and Black patients had lower historical costs due to access barriers rather than lower medical need c) The algorithm was trained predominantly on data from hospitals serving predominantly white patient populations d) The algorithm's developers failed to conduct any pre-deployment fairness testing

Correct Answer: B Explanation: The algorithm used healthcare costs as a proxy for health need — a seemingly reasonable design choice. But Black patients had historically incurred lower healthcare costs due to systemic barriers to access, not lower levels of illness. The algorithm therefore interpreted lower historical costs as lower health need and recommended Black patients for fewer intensive care management programs.

Part II: True / False (5 questions, 2 points each)

Question 9 An AI system that achieves overall accuracy of 97 percent can still exhibit significant discriminatory bias if its error rate is dramatically higher for some demographic groups than others.

Answer: TRUE Explanation: Overall accuracy metrics aggregate performance across all groups and can mask severe disparities at the subgroup level. A 97 percent overall accuracy that conceals 99 percent accuracy for one group and 80 percent accuracy for another represents significant discriminatory performance disparity. This is why disaggregated evaluation is essential.

Question 10 Under US law, an employer can avoid liability for employment discrimination by demonstrating that the AI system producing discriminatory outcomes was developed by a third-party vendor, not by the employer itself.

Answer: FALSE Explanation: The EEOC has confirmed that employers cannot escape Title VII liability by outsourcing discriminatory decisions to an AI system developed by a third-party vendor. The employer who deploys the tool and uses its outputs in employment decisions remains the responsible party under applicable civil rights law.

Question 11 The Amazon hiring algorithm's gender bias was completely eliminated once Amazon's engineers removed explicit references to gender and women's organizations from the input features.

Answer: FALSE Explanation: Removing explicit gender-correlated features was insufficient. The model found new proxy patterns in the remaining features — including writing style and linguistic patterns correlated with gender in the training data. The bias persisted because the training labels themselves (historical hiring decisions) were correlated with gender; removing features did not change the fundamental structure of the prediction problem.

Question 12 The Gender Shades study by Buolamwini and Gebru found that commercial facial analysis systems were least accurate for dark-skinned women — the group at the intersection of two dimensions of underperformance.

Answer: TRUE Explanation: This is a central finding of the Gender Shades study and an empirical demonstration of the intersectionality principle applied to AI. The accuracy gap between the best-performing subgroup (light-skinned men) and the worst-performing subgroup (dark-skinned women) was as large as 43-fold in some systems — a disparity that would not be visible in single-axis analysis.

Question 13 The mathematical incompatibility between certain fairness criteria — such as calibration and equal false positive rates — only arises when a technical error is present in the model design and can always be resolved by improving the algorithm.

Answer: FALSE Explanation: Researchers Chouldechova (2017) and Kleinberg et al. (2016) demonstrated that certain pairs of fairness criteria are mathematically incompatible when base rates differ across groups — a condition that arises not from technical error but from the social reality that historical discrimination has produced different outcome distributions across demographic groups. Improving the algorithm cannot resolve an incompatibility that is mathematical rather than technical.

Part III: Short Answer (4 questions, 10 points each)

Question 14 Explain the "proxy variable problem" using zip code as your example. Why does removing zip code from a credit scoring model not fully solve the problem?

Model Answer: A proxy variable is a variable that carries information about a protected characteristic and thereby serves as a functional substitute for it in algorithmic decision-making. Zip code is a classic example: because of decades of racially discriminatory housing policy — including redlining, racially restrictive covenants, and the systematic exclusion of Black Americans from wealth-building suburban homeownership — residential geography in the United States is strongly correlated with race. Including zip code in a credit scoring model therefore tends to produce lower credit scores for applicants in predominantly Black neighborhoods, importing racial discrimination under a technically neutral label.

Removing zip code from the model does not fully solve the problem for two reasons. First, zip code correlates with many other variables that would remain in the model — neighborhood type, nearby school quality, distance to commercial centers — that collectively carry similar information. Second, and more fundamentally, the same historical discrimination that produced the zip code correlation has shaped many other features in an applicant's financial profile: credit history length, banking relationship history, and prior loan approvals are all influenced by the same geographic and systemic exclusion. The discrimination is distributed across many features, not isolated in zip code, and removing one feature does not eliminate the distributed signal.

Question 15 Describe two stages of the ML pipeline where bias risk is particularly high and explain what a practitioner should do at each stage to detect and reduce that risk.

Model Answer (one acceptable version): Two particularly high-risk stages are data collection and model evaluation.

At the data collection stage, the primary risk is representation gaps — the systematic exclusion of certain demographic groups from the training data. When a model is trained on data that overrepresents some groups and underrepresents others, it will perform less accurately for the underrepresented groups. This is not a theoretical risk; it is documented in healthcare AI, speech recognition, and facial recognition systems. A practitioner should conduct a demographic audit of all data sources before training, document which groups are represented and at what rates, actively seek out additional data from underrepresented populations (including oversampling when appropriate), and establish minimum representation thresholds for all groups the system will be used on.

At the model evaluation stage, the risk is that aggregate performance metrics mask severe disparities at the group level. A model with 95 percent overall accuracy can simultaneously have 99 percent accuracy for the majority group and 75 percent accuracy for a minority group — a disparity that disappears in aggregate statistics. A practitioner should require disaggregated evaluation reporting performance metrics separately for each relevant demographic subgroup, establish minimum performance thresholds that must be met for each group rather than just in the aggregate, and use intersectional analysis (examining performance at combinations of characteristics, not just each individually) for the groups most likely to be affected.

Question 16 What is a feedback loop in the context of algorithmic bias? Give one example from Chapter 7 and explain why feedback loops make the case for early detection especially important.

Model Answer: A feedback loop occurs when the outputs of a biased AI system influence the inputs or training data for the same or a future version of the system, causing the initial bias to reinforce and intensify over time. Unlike static bias — which produces discrimination at a constant rate — feedback loops cause bias to grow across successive cycles, potentially producing severe structural discrimination from modest initial disparities.

The predictive policing example illustrates this clearly. A predictive policing algorithm trained on historical crime data identifies certain neighborhoods — typically those that have been most heavily policed historically — as high-crime and recommends concentrating patrol resources there. The concentrated patrols detect and record more crimes in those areas, including low-level offenses that would go undetected in less-patrolled areas. These additional arrests flow into the crime database, which now shows even higher crime rates in the algorithm-flagged areas. On the next training cycle, the algorithm identifies those areas as even higher-crime and recommends even more patrol concentration. The cycle continues indefinitely.

The feedback loop mechanism makes early detection especially important because the longer a biased system operates, the more entrenched the bias becomes. A bias that might have been addressed with moderate intervention when first deployed may require dramatic structural change — or become effectively impossible to undo — after years of feedback loop operation. The historical data generated by years of biased policing cannot simply be discarded; it becomes part of the social and institutional fabric. This is why the window for effective intervention is widest in the earliest stages of a system's design and deployment.

Question 17 Explain the key legal difference between "disparate treatment" and "disparate impact" in employment discrimination law. How does this distinction apply to an AI hiring tool that produces worse outcomes for female candidates without explicitly considering gender?

Model Answer: Disparate treatment and disparate impact represent two distinct legal theories of employment discrimination under Title VII of the Civil Rights Act.

Disparate treatment is intentional discrimination: an employer treats an individual differently because of a protected characteristic such as race or sex. Establishing disparate treatment typically requires proving that the employer's actions were motivated by discriminatory intent — that the employer acted differently toward a protected-class member than it would have acted toward a similarly situated non-member. Disparate treatment is the "classic" form of discrimination law, focused on individual decisions and subjective motivation.

Disparate impact is a broader doctrine established in Griggs v. Duke Power Co. (1971). Under disparate impact, a facially neutral employment practice — one that does not explicitly consider protected characteristics — can be unlawful if it produces a statistically significant disproportionate adverse effect on a protected class, without adequate business justification. Intent is not required; the statistical outcome alone is sufficient to establish a prima facie case.

An AI hiring tool that produces worse outcomes for female candidates without explicitly considering gender is most naturally analyzed under disparate impact. The tool appears facially neutral — it does not have a "female penalty" coded into its logic, and it does not ask applicants to identify their gender. But if its outputs systematically disadvantage female candidates at a statistically significant rate, that differential outcome triggers the disparate impact framework. The employer using the tool would then bear the burden of demonstrating that the tool is job-related and consistent with business necessity. The EEOC has explicitly confirmed that employers cannot escape this liability by attributing the discriminatory outcome to an AI vendor — the employer who deploys the tool and uses its outputs in employment decisions remains responsible.

Part IV: Applied Scenarios (3 questions, 15 points each)

Question 18 The Loan Underwriting Algorithm

A regional bank has deployed an AI system to automate loan underwriting decisions. An independent analysis of 12 months of lending data finds that the system approves loans for 73 percent of white applicants but only 51 percent of Black applicants with similar income, credit score, and debt-to-income ratios. The bank's legal team argues that the algorithm does not use race as an input and is therefore legally compliant. The algorithm's vendor states that the system has been "thoroughly tested for accuracy."

a) Evaluate the bank's legal team's argument. Is the absence of race as an explicit input variable sufficient to establish legal compliance under applicable US law? b) Identify at least three specific mechanisms from Chapter 7 that could explain how the racial disparity in approval rates arose despite race not being an explicit input. c) What steps should the bank take immediately? What steps should it take over the following 6–12 months?

Model Answer Guidelines:

Part a: The bank's legal team's argument is incorrect. The Equal Credit Opportunity Act prohibits discrimination in credit on the basis of race regardless of whether race is an explicit input. The disparate impact doctrine applies: if a facially neutral system produces a 22-percentage-point racial disparity in approval rates among applicants with similar financial profiles, this constitutes a substantial adverse effect on a protected class that creates a prima facie case of disparate impact. The CFPB has explicitly confirmed that ECOA applies to algorithmic credit systems and that lenders cannot avoid liability by pointing to neutral inputs. The vendor's "thoroughly tested for accuracy" claim does not address the disparate impact question; overall accuracy is not the relevant standard.

Part b: Multiple mechanisms from Chapter 7 could explain the disparity. Proxy variables: features like zip code, neighborhood, or employment history may serve as proxies for race due to residential segregation and historical employment discrimination. Training data bias: if the model was trained on historical approval decisions that reflected prior discriminatory lending practices, it will have learned patterns associated with those decisions. Feedback loops: historical credit denials in predominantly Black communities reduced credit history development, causing current credit scores to be lower for applicants in those communities independent of actual creditworthiness. Optimization target bias: if the model was optimized to minimize default rates based on historical data, and historical default rates were higher in communities with less access to financial support services, the model may have learned to treat community-level features as predictive in ways that proxy for race.

Part c: Immediately, the bank should halt the algorithm's decision-making for new applications or impose human review of all decisions, notify its primary banking regulator of the finding, and retain outside counsel to assess potential liability. It should document all decisions made by the algorithm during its deployment period. Over 6–12 months, the bank should conduct a comprehensive audit of the algorithm's features and training data to identify bias mechanisms; commission an independent fairness audit; develop and implement a remediation plan that may include retraining the model with fairness constraints, changing the feature set, or replacing the algorithm; establish ongoing disaggregated monitoring; and consider whether individuals who were denied credit by the algorithm should receive reconsideration under a fairer process.

Question 19 The Facial Recognition Deployment Decision

A major airport is considering deploying facial recognition technology for passenger boarding at all domestic gates, replacing manual ID checks. The system would compare each passenger's face against their driver's license or passport photo. Airport management notes that the system achieves 99.2 percent accuracy in vendor testing and argues this is sufficient for deployment. An advocacy organization raises concerns about the NIST findings on demographic accuracy disparities. The airport serves approximately 30 million passengers per year.

a) Using the NIST findings from Case Study 7.2, explain why aggregate accuracy of 99.2 percent may not be sufficient information to assess the fairness of this deployment. b) If false positive rates are 10 times higher for Black passengers than for white passengers, calculate approximately how many Black passengers would experience a false match per year, assuming that Black passengers represent 15 percent of the 30 million annual passengers. c) What additional information should the airport require from the vendor before making a deployment decision? What safeguards would a responsible deployment require?

Model Answer Guidelines:

Part a: Aggregate accuracy masks demographic disparities. If the system achieves 99.2 percent accuracy overall but has dramatically different accuracy rates for different demographic groups — as the NIST findings indicate is common in commercial facial recognition — the overall figure is misleading. A system with 99.8 percent accuracy for white passengers and 98 percent accuracy for Black passengers achieves a high aggregate figure while imposing the costs of errors disproportionately on Black passengers. The NIST findings showed false positive rates 10 to 100 times higher for Black and Asian faces than for white faces across the majority of tested systems. At airport scale, even small percentage differences in error rates translate into thousands of affected passengers.

Part b: 30 million passengers × 15 percent = 4.5 million Black passengers annually. A false positive rate means the system incorrectly flags a passenger who does not match. If the overall false positive rate produces a certain number of errors for white passengers, and the rate is 10 times higher for Black passengers, then approximately 10 times as many errors will occur per Black passenger as per white passenger. The exact calculation requires knowing the baseline false positive rate; students should demonstrate the logic. If the system has a 0.5 percent false positive rate for white passengers and 5 percent for Black passengers, then 4.5 million × 0.05 = 225,000 false positives would affect Black passengers annually.

Part c: The airport should require: disaggregated accuracy data broken down by race, gender, and age group (not just aggregate accuracy); independent verification of vendor claims using a dataset representative of the airport's actual passenger demographic mix; documentation of the training data's demographic composition; performance data for real-world deployment in similar settings; and specification of the false positive rate at the operating threshold the vendor proposes. Responsible deployment safeguards include: a clear process for resolving false positive matches without disproportionate burden or stigma on the affected passenger; an opt-out option; ongoing monitoring of outcomes by demographic group; a maximum performance disparity threshold that triggers review; and human oversight of all flagged cases with trained personnel who understand the system's known limitations.

Question 20 The Ethics of Discovery — What Should Organizations Do?

A consumer electronics company has deployed an AI content moderation system that automatically removes posts flagged as violating community guidelines. An internal data science team member runs an analysis and discovers that posts written in African American Vernacular English (AAVE) are flagged and removed at a rate 3.4 times higher than posts with similar content written in Standard American English. The data scientist brings this finding to their manager.

a) Using the analytical frameworks from Chapter 7, explain the most likely mechanism through which this bias arose. b) The manager suggests waiting until the next scheduled system review (in 8 months) to address the finding, arguing that action now would require significant engineering resources and could create liability by creating a record of known problems. Evaluate this argument from ethical, legal, and business perspectives. c) Describe what you believe the company should do in the next 30 days, the next 90 days, and the next year. Justify each set of actions.

Model Answer Guidelines:

Part a: The most likely mechanism is label bias. Content moderation AI systems are trained on large datasets of human-labeled content in which human reviewers have marked certain posts as violating or non-violating community guidelines. Research has documented that human content moderators — who often lack familiarity with AAVE and other non-dominant language varieties — systematically misclassify AAVE posts as more harmful or rule-violating than identical content in Standard American English. The AI was trained on these biased human labels and learned to replicate the labelers' misclassifications. This may be compounded by training data bias if the training corpus underrepresented AAVE, causing the model to be generally less accurate on AAVE text and to apply majority-dialect patterns inappropriately.

Part b: The manager's argument fails on all three grounds. Ethically, the company is currently suppressing the speech of AAVE speakers at 3.4 times the rate of Standard American English speakers — an ongoing harm to a group already subject to discrimination. Delaying remediation by 8 months extends that ongoing harm. The "we don't have a record of known problems" framing is precisely the ethics-washing pattern identified in Chapter 7 — treating discovery of a problem as a liability to be managed rather than an obligation to be discharged. Legally, deliberately delaying action on known discriminatory outcomes after internal discovery has been found to aggravate liability in civil rights contexts; the argument that documentation of the problem creates liability has it precisely backwards. From a business perspective, the reputational risk of deploying a system with known discriminatory impact — particularly against Black users — is substantially higher than the engineering cost of addressing it, especially given the current media and regulatory environment around AI bias.

Part c: In the next 30 days: suspend automatic removal for posts flagged solely by the AAVE-affected model; implement human review for all content in the affected category; notify leadership and legal; and commission a full audit of the system's demographic performance. In the next 90 days: complete the audit; begin retraining with more representative labeling data that includes AAVE expertise; develop a remediation plan with specific milestones; and evaluate whether affected users whose posts were incorrectly removed should be notified. In the next year: deploy the retrained system with ongoing monitoring; establish a bias review process for all content moderation AI systems; create a feedback mechanism for users to report suspected bias in content moderation decisions; and consider disclosure in the platform's transparency report.

End of Quiz

Grading Summary: - Part I (Multiple Choice): 16 points - Part II (True/False): 10 points - Part III (Short Answer): 40 points - Part IV (Applied Scenarios): 45 points - Total: 111 points

Instructors may scale to 100 points or use as a 100-point assessment by omitting one applied scenario question.