Quiz: AI and Justice — Criminal Justice, Civil Rights, and Accountability

Test your understanding before moving on. Target: 70% or higher to proceed confidently.

Section 1: Multiple Choice (1 point each)

1. What is the fundamental data problem with predictive policing systems?

  • A) The data is too old to be useful for current predictions
  • B) Historical crime data reflects policing patterns, not actual crime patterns
  • C) Police departments deliberately manipulate the data to target minorities
  • D) The algorithms are not sophisticated enough to analyze the data properly
Answer **B)** Historical crime data reflects policing patterns, not actual crime patterns *Why B:* The core issue is that arrest and incident data measures where police have been active, not where crime actually occurs. Over-policed neighborhoods produce more data, creating a biased picture. *Why not A:* While data freshness can be an issue, the fundamental problem is bias in the data, not its age. *Why not C:* The problem is structural, not the result of deliberate manipulation by individual actors. *Why not D:* The algorithms can be quite sophisticated — the problem is in the data they are trained on, not the analytical methods. *Reference:* Section 17.1

2. In the context of predictive policing, a "runaway feedback loop" occurs when:

  • A) Officers ignore the algorithm's recommendations
  • B) The algorithm is updated too frequently
  • C) The algorithm's outputs become its own future inputs, reinforcing existing patterns
  • D) Multiple police departments share the same algorithm
Answer **C)** The algorithm's outputs become its own future inputs, reinforcing existing patterns *Why C:* The algorithm directs police to certain areas; increased policing produces more arrest data; that data feeds back into the algorithm, which then directs even more policing to those same areas. *Why not A:* Ignoring recommendations would actually break the loop, not create one. *Why not B:* Update frequency is not the mechanism that creates feedback loops. *Why not D:* Sharing algorithms across departments is a different concern. *Reference:* Section 17.1

3. The ProPublica investigation of COMPAS found that the algorithm:

  • A) Used race as a direct input variable
  • B) Had higher false positive rates for Black defendants who did not reoffend
  • C) Was less accurate overall than random chance
  • D) Had been deliberately programmed to discriminate
Answer **B)** Had higher false positive rates for Black defendants who did not reoffend *Why B:* ProPublica found that Black defendants who did not go on to reoffend were nearly twice as likely as white defendants to be incorrectly labeled as high risk. *Why not A:* COMPAS did not use race as a direct input — the disparities arose through correlated factors. *Why not C:* The algorithm had meaningful predictive accuracy; the issue was differential error rates, not overall failure. *Why not D:* There is no evidence of deliberate discriminatory programming; the bias is structural. *Reference:* Section 17.2

4. Northpointe (COMPAS developer) responded to ProPublica by pointing to:

  • A) The algorithm's use of race-neutral variables
  • B) Predictive parity — equal accuracy among those labeled high risk across racial groups
  • C) The fact that judges could ignore the scores
  • D) Independent validation by federal regulators
Answer **B)** Predictive parity — equal accuracy among those labeled high risk across racial groups *Why B:* Northpointe argued that among defendants scored as high risk, the percentage who actually reoffended was roughly the same for Black and white defendants. *Why not A:* While true, this was not the primary defense against ProPublica's specific findings. *Why not C:* While true in principle, this does not address the fairness question about the scores themselves. *Why not D:* There was no federal regulatory validation of COMPAS at the time. *Reference:* Section 17.2

5. The reason both ProPublica and Northpointe could be "right" about COMPAS is:

  • A) They were analyzing different datasets
  • B) One was using qualitative methods and the other quantitative
  • C) Different fairness metrics are mathematically impossible to satisfy simultaneously when base rates differ between groups
  • D) ProPublica was biased in its analysis
Answer **C)** Different fairness metrics are mathematically impossible to satisfy simultaneously when base rates differ between groups *Why C:* This is the fairness impossibility theorem — when the base rate of the predicted outcome (recidivism) differs between groups, equalizing false positive rates and maintaining predictive parity cannot both be achieved. *Why not A:* Both analyzed the same Broward County dataset. *Why not B:* Both used quantitative methods. *Why not D:* ProPublica's analysis was methodologically sound; the disagreement was about which metric matters, not about the data. *Reference:* Section 17.2

6. In the Loomis v. Wisconsin case, the Wisconsin Supreme Court ruled that:

  • A) Risk assessment tools violate due process and cannot be used in sentencing
  • B) Risk assessment tools are constitutional as long as they are not the sole basis for sentencing
  • C) Defendants have a right to inspect the source code of any algorithmic tool used in their case
  • D) The use of COMPAS constituted racial discrimination
Answer **B)** Risk assessment tools are constitutional as long as they are not the sole basis for sentencing *Why B:* The court allowed the use of COMPAS scores while cautioning that they should be one factor among many, not the determinative factor. *Why not A:* The court did not find a due process violation, though it acknowledged concerns. *Why not C:* The court did not establish a right to inspect source code, which remains an unresolved issue. *Why not D:* The case was decided on due process grounds, not equal protection or racial discrimination. *Reference:* Section 17.3

7. Proxy variables are a concern in algorithmic justice because:

  • A) They make algorithms more expensive to run
  • B) They are variables correlated with protected characteristics like race, reproducing disparities even when race is not an explicit input
  • C) They are difficult to collect in a timely manner
  • D) They always produce false positives
Answer **B)** They are variables correlated with protected characteristics like race, reproducing disparities even when race is not an explicit input *Why B:* Variables like zip code, employment status, and prior arrest history are correlated with race due to historical discrimination, so they can reproduce racial disparities even when the algorithm does not "see" race. *Why not A:* Computational cost is not the issue with proxy variables. *Why not C:* Data collection timing is not the core concern. *Why not D:* Proxy variables can affect both false positives and false negatives; the issue is systematic bias, not a single type of error. *Reference:* Section 17.3

8. The EU AI Act addresses AI in criminal justice by:

  • A) Banning all AI in law enforcement
  • B) Allowing unrestricted AI use as long as outcomes are disclosed
  • C) Classifying law enforcement AI as "high risk" with mandatory oversight, assessments, and transparency
  • D) Leaving regulation entirely to member states
Answer **C)** Classifying law enforcement AI as "high risk" with mandatory oversight, assessments, and transparency *Why C:* The EU AI Act requires conformity assessments, human oversight, transparency, and documentation for high-risk AI systems, which include law enforcement applications. *Why not A:* The Act does not ban all AI in law enforcement, though it prohibits certain specific uses like most real-time public facial recognition. *Why not B:* Disclosure alone is insufficient under the Act; substantive requirements apply. *Why not D:* The Act is EU-wide regulation, not a delegation to member states. *Reference:* Section 17.3

9. An Algorithmic Impact Assessment (AIA) should include all of the following EXCEPT:

  • A) An equity analysis of how the system affects different demographic groups
  • B) Documentation of the system's training data and known limitations
  • C) A guarantee that the system will produce perfectly fair outcomes
  • D) A plan for community input and ongoing monitoring
Answer **C)** A guarantee that the system will produce perfectly fair outcomes *Why C:* No system can guarantee perfectly fair outcomes — the fairness impossibility theorem demonstrates this. An AIA should identify potential disparities and mitigation strategies, not promise perfection. *Why not A:* Equity analysis is a core component of any responsible AIA. *Why not B:* Data documentation and limitation disclosure are essential. *Why not D:* Community input and monitoring are recommended best practices. *Reference:* Section 17.5

10. The "accountability gap" in AI-assisted justice refers to:

  • A) The time delay between an algorithm's prediction and the court's decision
  • B) The space between an AI system's harmful output and any actor who can be held meaningfully responsible
  • C) The difference in accuracy between the algorithm and human judges
  • D) The gap between public expectations and the algorithm's actual capabilities
Answer **B)** The space between an AI system's harmful output and any actor who can be held meaningfully responsible *Why B:* The accountability gap arises because multiple actors (developers, vendors, agencies, judges) each bear partial responsibility, enabling each to deflect blame to others. *Why not A:* The gap is about responsibility, not timing. *Why not C:* Accuracy comparison is a different issue from accountability. *Why not D:* While expectations gaps exist, the accountability gap specifically concerns who is responsible for harms. *Reference:* Section 17.4

Section 2: True/False with Justification (1 point each)

11. Risk assessment tools predict whether a person will actually commit a crime in the future.

Answer **False** *Explanation:* Risk assessment tools predict the probability of *rearrest*, not reoffending. Rearrest is influenced by policing intensity, surveillance, and prosecutorial decisions — not just individual behavior. A person in a heavily policed neighborhood is more likely to be rearrested for the same behavior than someone in a lightly policed area.

12. Removing race as an input variable from a risk assessment algorithm ensures that the algorithm's outputs will not correlate with race.

Answer **False** *Explanation:* Proxy variables — factors correlated with race, such as zip code, employment history, and prior arrest records — can reproduce racial disparities even when race is not an explicit input. Because these factors are shaped by historical discrimination, an algorithm using them can still produce racially disparate outcomes.

13. The United States has a comprehensive federal law specifically regulating the use of AI in criminal justice.

Answer **False** *Explanation:* As of the writing of this textbook, the United States does not have a comprehensive federal law specifically regulating AI in criminal justice. Some states and cities have passed targeted legislation (e.g., facial recognition disclosure laws, bias audit requirements), but there is no overarching federal framework comparable to the EU AI Act.

14. Human judges are consistently fairer than algorithmic risk assessment tools.

Answer **False (or at best: not established)** *Explanation:* Research on judicial decision-making reveals significant biases, including racial disparities in bail-setting and sentencing, and susceptibility to irrelevant factors like time of day. The comparison between human and algorithmic judgment is complex — neither is consistently fairer. The relevant question is not "algorithm vs. perfect justice" but "algorithm vs. the flawed human status quo."

15. Predictive policing systems discover where crime is actually occurring.

Answer **False** *Explanation:* Predictive policing systems trained on historical arrest data primarily discover where *policing* has been concentrated, not where crime is occurring. Because policing has historically been distributed unevenly along racial and socioeconomic lines, the systems risk amplifying existing enforcement disparities rather than revealing the true distribution of criminal activity.

Section 3: Short Answer (2 points each)

16. Describe the runaway feedback loop in predictive policing in four to five sentences. Start with the historical data and trace the cycle through at least two complete iterations.

Sample Answer Historical crime data reflects areas that were already heavily policed, showing higher arrest rates in those neighborhoods. A predictive policing algorithm trains on this data and identifies those same neighborhoods as "high risk." Police are directed to patrol those areas more intensively, resulting in more arrests. The new arrest data is fed back into the algorithm, which now has even stronger evidence that these areas are high crime. The algorithm recommends even more concentrated policing, and the cycle continues — amplifying the original pattern with each iteration. *Rubric — full credit requires:* - Clear identification of biased historical data as the starting point - Description of how the algorithm converts biased data into patrol recommendations - Explanation of how increased policing generates new biased data - Recognition that the cycle is self-reinforcing

17. Explain why the choice between different fairness metrics for a risk assessment tool is ultimately a political and ethical decision, not a technical one. Use the COMPAS case as an example.

Sample Answer The COMPAS case showed that equalizing false positive rates across racial groups (ProPublica's metric) and maintaining equal predictive accuracy among high-risk designations (Northpointe's metric) cannot both be achieved simultaneously when recidivism base rates differ between groups. Since both metrics represent legitimate and reasonable definitions of fairness, the choice between them cannot be resolved by mathematics or engineering alone. It requires deciding which type of error matters more — wrongly labeling innocent people as dangerous (false positives) or wrongly releasing dangerous people (false negatives) — and how to distribute those errors across racial groups. These are value judgments that should be made through democratic processes with input from affected communities, not delegated to algorithm designers. *Rubric — full credit requires:* - Reference to the specific conflict between fairness metrics in the COMPAS case - Explanation of why the conflict is mathematical, not a design flaw - Recognition that resolving the conflict requires value judgments - Acknowledgment that affected communities should participate in the decision

18. What is a "right to explanation" in the context of algorithmic justice, and why is it difficult to implement in practice?

Sample Answer A "right to explanation" is the principle that individuals affected by algorithmic decisions should receive a meaningful, human-understandable account of how the decision was reached — including what factors were considered and how they influenced the outcome. It is difficult to implement for several reasons: many algorithms are proprietary trade secrets; complex machine learning models may not be easily explainable even to their creators; "meaningful" explanation is subjective and context-dependent; and providing explanations that are both technically accurate and accessible to non-experts is a significant challenge. The EU's GDPR includes a version of this right, but the U.S. does not currently have a federal equivalent. *Rubric — full credit requires:* - Clear definition of the right to explanation - At least two specific challenges to implementation - Recognition of the tension between proprietary interests and transparency

Section 4: Applied Scenario (3–5 points)

19. A mid-size city is considering adopting a new AI-powered risk assessment tool for pretrial detention decisions. The vendor claims the tool is "89% accurate" and "race-neutral." You have been hired as an independent consultant to evaluate this proposal. Write a memo (200–300 words) to the city council that:

  • Identifies at least three questions they should ask the vendor
  • Explains why "89% accurate" and "race-neutral" are insufficient claims
  • Recommends specific conditions that should be met before deployment
Sample Answer **Memo to City Council:** The vendor's claims of "89% accuracy" and "race-neutral" require careful scrutiny. **Questions for the vendor:** 1. Accuracy of *what*? Is this predicting rearrest, failure to appear, or violent reoffending? Each has different implications. 2. How is accuracy measured? Overall accuracy can mask large disparities in error rates across racial groups. What are the false positive and false negative rates for Black, Latino, and white defendants separately? 3. What training data was used? Was the tool validated on a population similar to our city's demographics and criminal justice practices? **Why the claims are insufficient:** "89% accurate" is meaningless without context. An algorithm that predicts "no rearrest" for everyone in a population with a 15% rearrest rate would be "85% accurate" but entirely useless. "Race-neutral" means race is not an input — but proxy variables (zip code, employment status, prior arrests) can reproduce racial disparities without ever mentioning race. **Conditions for deployment:** - Independent bias audit before deployment and annually thereafter - Public disclosure of the algorithm's factors, validation data, and performance metrics disaggregated by race, gender, and age - Mandatory training for judges on algorithmic limitations - A community oversight board with authority to recommend discontinuation - A sunset clause requiring council reauthorization after two years - Clear documentation of who is responsible when the tool produces unjust outcomes *Rubric:* | Criterion | 0 pts | 1 pt | 2 pts | 3 pts | |-----------|-------|------|-------|-------| | Questions to vendor | No questions | 1 generic question | 2-3 relevant questions | 3+ targeted, specific questions | | Critique of claims | Accepts claims at face value | Identifies one concern | Identifies concerns with both claims | Connects to proxy variables and base rate issues | | Conditions | No conditions | Generic conditions | Specific, actionable conditions | Comprehensive, drawing on chapter concepts |

20. Revisit CityScope Predict. The Millhaven city council has voted 4-3 to deploy the system on a one-year trial basis. Councilmember Aisha Thompson, who voted against it, has asked you to draft a list of minimum safeguards. Design a comprehensive oversight plan (200–300 words) that addresses data quality, fairness monitoring, accountability, community involvement, and conditions for termination.

Sample Answer **Oversight Plan for CityScope Predict Trial:** **Data quality:** Require an independent audit of the historical crime data before the algorithm trains on it. Document known biases (e.g., neighborhoods with historically disproportionate policing). Commission supplementary data sources — victimization surveys, community reports — to provide a more complete picture than arrest records alone. **Fairness monitoring:** Publish monthly reports disaggregated by race and neighborhood, including: where patrols were directed, how many stops and arrests occurred in algorithm-directed areas vs. non-directed areas, and whether arrest rates in targeted areas increased simply because more officers were present. Define specific disparity thresholds that trigger automatic review. **Accountability:** Designate a single city official as the accountable party for the system's performance. Require officers to document the basis for any stop, search, or arrest in algorithm-directed areas. Create a complaint mechanism for residents who believe they were targeted unfairly. **Community involvement:** Establish a community oversight board with residents from the most-affected neighborhoods, including at least one formerly incarcerated person and one public defender. The board should receive quarterly briefings and have the authority to recommend suspension of the system. **Termination conditions:** The trial automatically ends if: (a) disparate impact exceeds predefined thresholds for two consecutive months, (b) the community oversight board votes to recommend termination, or (c) an independent audit finds that the system is amplifying rather than reducing inequitable policing patterns. *Rubric:* | Criterion | 0 pts | 1 pt | 2 pts | 3 pts | |-----------|-------|------|-------|-------| | Data quality | Not addressed | Generic mention | Specific measures | Includes supplementary data sources | | Fairness monitoring | Not addressed | Generic mention | Defines metrics | Includes disparity thresholds and disaggregated reporting | | Accountability | Not addressed | Generic mention | Designates responsibility | Includes documentation requirements and complaint mechanisms | | Community involvement | Not addressed | Generic mention | Includes oversight board | Board has genuine authority and diverse membership | | Termination conditions | Not addressed | Generic mention | Defines conditions | Specific, measurable criteria tied to equity outcomes |

Scoring & Next Steps

Score Assessment Recommended Action
< 50% Needs review Re-read sections 17.1–17.3, redo Part A exercises
50–70% Partial Review weak areas, redo Part B exercises
70–85% Solid Ready to proceed; revisit any missed topics
> 85% Strong Proceed; consider Deep Dive extensions