Chapter 32 Quiz: Ethics in Data Science: Bias, Privacy, Consent, and Responsible Practice

Instructions: This quiz tests your understanding of Chapter 32. Answer all questions before checking the solutions. For multiple choice, select the best answer. For short answer questions, aim for 2-4 clear sentences. Total points: 100.

Section 1: Multiple Choice (10 questions, 4 points each)

Question 1. In the ProPublica analysis of the COMPAS recidivism prediction tool, what was the primary finding regarding racial bias?

  • (A) The tool explicitly used race as a predictor variable
  • (B) The tool had significantly higher false positive rates for Black defendants than for white defendants
  • (C) The tool was less accurate overall than random chance
  • (D) The tool was only used in southern U.S. states
Answer **Correct: (B)** ProPublica found that COMPAS was roughly twice as likely to falsely label Black defendants as high-risk (false positive rate of 44.9% vs. 23.5% for white defendants). The tool did not explicitly include race as a feature (A is wrong), it performed better than random chance (C is wrong), and it was used in courts across the country (D is wrong). The key issue was that equal predictive parity coexisted with unequal error rates.

Question 2. What is "proxy discrimination" in the context of algorithmic decision-making?

  • (A) Using a substitute model when the primary model fails
  • (B) When features that correlate with protected attributes (like race or gender) produce discriminatory outcomes even though the protected attribute itself is not included
  • (C) When a proxy server collects discriminatory data
  • (D) Hiring someone to make decisions on your behalf
Answer **Correct: (B)** Proxy discrimination occurs when a model discriminates through features that are correlated with protected attributes. For example, using zip code as a feature can produce racial discrimination because residential segregation creates strong correlations between zip code and race. Removing the protected attribute does not prevent discrimination if proxies remain in the model.

Question 3. Amazon's AI hiring tool was scrapped because:

  • (A) It was too expensive to maintain
  • (B) It learned to penalize women because the historical training data reflected a male-dominated industry
  • (C) It violated GDPR
  • (D) It could not process enough resumes per hour
Answer **Correct: (B)** Amazon's hiring tool was trained on resumes from the previous 10 years, during which most successful hires were men. The model learned that male-associated characteristics predicted "success" and penalized indicators associated with women (like "women's" in club names or women's college affiliations). This illustrates that models trained on historically biased data will learn and perpetuate those biases.

Question 4. The mathematical impossibility result regarding algorithmic fairness states that:

  • (A) No algorithm can be completely accurate
  • (B) When base rates differ between groups, it is impossible to simultaneously satisfy all common definitions of fairness
  • (C) Fairness can only be achieved with perfectly balanced datasets
  • (D) Algorithms are always less fair than human decision-makers
Answer **Correct: (B)** Researchers proved that when groups have different base rates (e.g., different actual reoffending rates), you cannot simultaneously equalize false positive rates, false negative rates, AND predictive parity across groups. This means every algorithmic system must choose which type of fairness to prioritize — a value judgment, not a technical decision.

Question 5. Which of the following is NOT a principle of the GDPR?

  • (A) Right to erasure (the right to be forgotten)
  • (B) Data minimization (collect only what is necessary)
  • (C) Right to maximum data collection for better service
  • (D) Purpose limitation (data cannot be repurposed without consent)
Answer **Correct: (C)** GDPR requires the opposite: data *minimization*, meaning organizations should collect only the data necessary for the stated purpose. (A), (B), and (D) are all genuine GDPR principles. The regulation fundamentally opposes the "collect everything" approach and instead requires organizations to justify each piece of data they collect.

Question 6. Research has shown that "anonymized" data can often be re-identified because:

  • (A) Hackers can always decrypt anonymized data
  • (B) Combinations of quasi-identifiers (like zip code, date of birth, and gender) can uniquely identify individuals
  • (C) Anonymization software always contains bugs
  • (D) People share their anonymized data on social media
Answer **Correct: (B)** Research by Latanya Sweeney demonstrated that 87% of the U.S. population could be uniquely identified using only zip code, date of birth, and gender — three fields commonly included in "anonymized" datasets. Quasi-identifiers do not individually identify anyone, but their combination narrows the possibilities to a single person. This is why anonymization alone is insufficient for privacy protection.

Question 7. Differential privacy works by:

  • (A) Encrypting all data before analysis
  • (B) Adding carefully calibrated noise to results so that individual records cannot be distinguished
  • (C) Deleting individual records after analysis
  • (D) Storing data on separate servers for different groups
Answer **Correct: (B)** Differential privacy adds mathematical noise to query results or data outputs so that the result is approximately the same whether or not any single individual is included in the dataset. This provides a formal, provable privacy guarantee — unlike anonymization, which can be defeated empirically. The tradeoff is between privacy strength (epsilon parameter) and data accuracy.

Question 8. In the context of data ethics, a "feedback loop" occurs when:

  • (A) Users provide feedback on a product
  • (B) A model's outputs influence the data it is subsequently trained on, potentially amplifying its biases
  • (C) Data is shared between departments
  • (D) A model is retrained on new data
Answer **Correct: (B)** A feedback loop occurs when a deployed model's predictions affect the real world in ways that influence future training data. Predictive policing is a classic example: the model directs police to certain neighborhoods, generating more arrests there, which creates more "crime data" from those neighborhoods, which makes the model more confident in targeting them. The model becomes self-reinforcing rather than learning from independent evidence.

Question 9. The term "surveillance capitalism" refers to:

  • (A) Government surveillance of financial markets
  • (B) A business model where companies generate revenue by collecting and monetizing behavioral data to predict and influence human behavior
  • (C) Using security cameras in retail stores to prevent theft
  • (D) Monitoring employees' work output
Answer **Correct: (B)** Coined by Shoshana Zuboff, "surveillance capitalism" describes the economic model in which behavioral data — clicks, searches, locations, purchases, social connections — is extracted, analyzed, and sold as predictions about future behavior. In this model, users are not the customers; they are the raw material. The concept highlights how data collection practices are driven by economic incentives that may conflict with user welfare and privacy.

Question 10. Joy Buolamwini and Timnit Gebru's research on facial recognition revealed that:

  • (A) Facial recognition technology works equally well for all demographic groups
  • (B) Error rates were highest for dark-skinned women and lowest for light-skinned men, with differences up to 43x
  • (C) The technology only failed for people wearing glasses
  • (D) All commercial systems had error rates above 50%
Answer **Correct: (B)** Their 2018 study found error rates ranging from 0.0-0.8% for light-skinned men to 20.8-34.7% for dark-skinned women — a disparity of up to 43 times. The cause was training data dominated by light-skinned and male faces. The research demonstrated how technical systems can encode demographic biases through unrepresentative data, and it led major companies to improve their systems.

Section 2: True or False (4 questions, 4 points each)

Question 11. True or False: Removing protected attributes (race, gender, age) from a model's features is sufficient to prevent the model from discriminating.

Answer **False.** Removing protected attributes does not prevent discrimination because other features can serve as proxies. Zip code correlates with race. First name correlates with gender and ethnicity. University correlates with socioeconomic status. The model can learn discriminatory patterns through these proxy variables even without direct access to protected attributes. Preventing discrimination requires testing model outcomes across groups, not just removing inputs.

Question 12. True or False: If a model has the same overall accuracy for two groups, it is fair.

Answer **False.** Equal overall accuracy can mask very different types of errors. A model could have 80% accuracy for both Group A and Group B, but achieve this through different error distributions — high false positives for one group and high false negatives for the other. The COMPAS case demonstrated exactly this: roughly equal overall accuracy coexisted with dramatically different false positive and false negative rates across racial groups.

Question 13. True or False: The Cambridge Analytica scandal involved collecting data from people who never took the personality quiz and never consented to data collection.

Answer **True.** The personality quiz app collected data not only from the approximately 270,000 people who took the quiz but also from their Facebook friends — up to 87 million people total. These friends never consented to data collection. Facebook's policies at the time technically allowed this access, but the friends had no knowledge that their data was being harvested for political profiling.

Question 14. True or False: Algorithmic decision-making is inherently less ethical than human decision-making.

Answer **False.** Both algorithmic and human decision-making can be biased. Human decision-makers are subject to cognitive biases, stereotypes, fatigue, and inconsistency — judges give harsher sentences before lunch, doctors treat pain less aggressively in certain patients. Algorithms are biased too, but their biases can at least be measured, audited, and corrected. The question is not "algorithms vs. humans" but "how do we make either type of decision-making as fair and accurate as possible?"

Section 3: Short Answer (4 questions, 6 points each)

Question 15. Explain the concept of "informed consent" in data science and describe two ways it commonly fails in practice.

Answer **Informed consent** means that individuals understand what data is being collected about them, how it will be used, what risks are involved, and that they freely agree to participate. **Two common failures:** 1. **Terms-of-service consent is not truly informed.** When users click "I agree" on a 10,000-word legal document they have not read (and could not understand if they did), they are providing legal consent but not meaningful, informed consent. The information asymmetry between the company and the user makes true understanding impossible. 2. **Purpose creep.** Data collected for one stated purpose is used for another. A fitness app collects health data for "personalized recommendations" but sells it to insurance companies. Even if users consented to the original purpose, they did not consent to the secondary use. GDPR addresses this through the purpose limitation principle, but enforcement remains inconsistent.

Question 16. Explain the difference between demographic parity and equal opportunity as definitions of algorithmic fairness. Give an example of a situation where they would produce different outcomes.

Answer **Demographic parity** requires that the algorithm produces the same positive outcome rate across groups (e.g., the same proportion of men and women are hired). **Equal opportunity** requires that, among qualified individuals, the algorithm selects them at equal rates across groups (e.g., among genuinely qualified candidates, men and women are equally likely to be selected). **Example where they differ:** Consider a medical school admissions algorithm. If 40% of male applicants and 60% of female applicants meet the qualification threshold, then: - **Demographic parity** would require admitting equal proportions from each gender — potentially admitting less-qualified male applicants or rejecting more-qualified female applicants. - **Equal opportunity** would require that among qualified applicants, both genders have an equal chance of admission — potentially resulting in more women being admitted (since more women are qualified). The two definitions produce different outcomes whenever the base rates (qualification rates, reoffending rates, etc.) differ between groups.

Question 17. Describe how a model can create a harmful feedback loop. Use a specific example.

Answer A feedback loop occurs when a model's predictions influence the environment that generates future training data, causing the model to reinforce its own patterns. **Specific example — predictive policing:** A model trained on historical arrest data learns that certain neighborhoods have high crime rates. It directs more police to those neighborhoods. More police presence leads to more arrests (even for minor offenses that occur everywhere but are only enforced in heavily policed areas). The new arrest data confirms the model's predictions, making it even more confident in targeting those neighborhoods. Over time, the model amplifies the pattern regardless of whether the underlying crime rates actually differ. The neighborhoods are over-policed not because they are more dangerous, but because the model creates a self-fulfilling prophecy. Similar feedback loops can occur in content recommendation (promoting outrage because it generates engagement), lending (denying credit to certain communities, preventing them from building credit history), and hiring (screening out candidates from certain backgrounds, ensuring they never have the chance to prove themselves).

Question 18. What are three key principles of GDPR, and why are they relevant to data scientists even outside of Europe?

Answer **Three key GDPR principles:** 1. **Data minimization:** Organizations should collect only the data necessary for the stated purpose — not everything they can get. This challenges the "collect now, figure out uses later" approach common in data science. 2. **Right to erasure:** Individuals can request that their data be deleted. This has technical implications: models trained on deleted data may need to be retrained, and data pipelines must support deletion requests. 3. **Right to explanation:** Individuals subject to automated decisions have the right to "meaningful information about the logic involved." This pushes data scientists toward explainable models rather than black-box approaches. **Relevance outside Europe:** GDPR applies to any organization processing data of EU residents, regardless of where the organization is located. It has also influenced privacy legislation worldwide (California's CCPA, Brazil's LGPD, etc.). Even for domestic-only organizations, GDPR principles represent the direction of global privacy standards and are increasingly considered best practices.

Section 4: Applied Scenarios (2 questions, 8 points each)

Question 19. A university develops an algorithm to predict which incoming students are at risk of dropping out. The algorithm uses high school GPA, SAT scores, family income, and zip code. It recommends that at-risk students be assigned mandatory academic advising.

Apply the ethical framework from Section 32.6 to evaluate this system. Address all five questions: (1) Who benefits and who is harmed? (2) Is the data representative? (3) What are the failure modes? (4) Could it be misused? (5) Is it transparent?

Answer **1. Who benefits and who is harmed?** - Benefits: Students who are correctly identified as at-risk and receive helpful advising. The university (higher retention rates, better outcomes). - Potential harms: Students falsely identified as at-risk may feel stigmatized, treated as less capable, or resentful of mandatory requirements. The "at-risk" label itself can become a self-fulfilling prophecy if it changes how faculty perceive and interact with a student. **2. Is the data representative?** - Using family income and zip code as predictors means the model will disproportionately flag low-income and minority students as "at-risk." These students may face real barriers, but the model may be capturing socioeconomic disadvantage rather than individual academic risk. Students from wealthy families facing personal crises may be missed. **3. What are the failure modes?** - False positives: Students wrongly labeled as at-risk are subjected to mandatory advising they do not need, potentially experiencing it as patronizing or stigmatizing. - False negatives: Students who are actually at risk but do not match the model's profile (e.g., wealthy students with personal problems) receive no intervention. - Disparate impact: The model likely flags a much higher proportion of low-income and minority students, creating a perception that these groups are inherently less capable. **4. Could it be misused?** - The risk labels could be used for purposes beyond advising — e.g., admissions committees could use them to screen out "risky" applicants. Insurers could use them to adjust financial aid. Faculty could treat labeled students differently. **5. Is it transparent?** - Can students see their risk score? Can they understand why they were flagged? Can they challenge the classification? If the system operates in secret, students cannot advocate for themselves. **Overall:** The system has good intentions but significant risks. Key improvements: make advising optional (not mandatory), do not disclose risk labels to anyone other than the advising office, audit for demographic disparities, and use the model to offer support rather than impose requirements.

Question 20. You are a data scientist at a social media company. Your team has built a content recommendation algorithm that maximizes user engagement (time spent on the platform). An internal study reveals that the algorithm promotes emotionally charged and divisive content because users engage with it more. The algorithm is highly profitable.

The VP of Product asks you to continue optimizing for engagement. What do you do? Address: (a) the ethical issues, (b) how you would communicate your concerns to the VP, and (c) what alternative approaches you would propose.

Answer **(a) Ethical issues:** - **Harm to users:** The algorithm may be promoting content that causes anxiety, outrage, political polarization, and decreased wellbeing. Optimizing for engagement is not the same as optimizing for user value — users can be "engaged" by content that makes them angry or upset. - **Societal harm:** Amplifying divisive content contributes to political polarization, erosion of shared reality, and potential real-world violence (research has linked social media amplification to hate crimes and political unrest). - **Informed consent:** Users did not consent to being psychologically manipulated for profit. They may not even be aware that the content they see is algorithmically curated to exploit their emotional responses. - **Feedback loop:** The algorithm promotes divisive content → users engage with it → the algorithm learns that divisive content "works" → more divisive content is promoted. The system escalates toward extremity. **(b) How to communicate concerns:** Frame the issue in terms the VP will understand — business risk, not just ethics. "Our algorithm is effectively training users to consume increasingly extreme content. This creates reputational risk (congressional hearings, press investigations), regulatory risk (content moderation regulations), advertiser risk (brands don't want to appear next to extreme content), and user attrition risk (users who burn out leave the platform permanently). Internal research at other platforms has shown that engagement-maximizing algorithms produce short-term gains but long-term user loss." **(c) Alternative approaches:** - Optimize for "time well spent" rather than "time spent" — measure user satisfaction, not just engagement - Include content diversity metrics in the optimization function to prevent filter bubbles - Down-weight content that is flagged as divisive or misleading - Add friction to sharing (e.g., "Did you read this article?" prompts) - Measure long-term retention, not just session length — users who are healthier on the platform stay longer over their lifetime If the VP rejects all alternatives, this becomes a question of personal ethics: do you comply, escalate, or leave? There is no single right answer, but the question deserves serious consideration.

Scoring Guide

Section Points
Multiple Choice (10 x 4) 40
True/False (4 x 4) 16
Short Answer (4 x 6) 24
Applied Scenarios (2 x 8) 16
Total 96

Note: Remaining 4 points are reserved for exceptional depth in short answer or scenario responses. Passing score: 70/100.

Ethics does not have a final exam. The real test is what you do when nobody is watching — when you notice a bias in your model and could ignore it, when you discover that your data was collected without consent, when you are asked to build something you believe will cause harm. This quiz tested your understanding of the concepts. Your career will test your character.