Case Study: COMPAS and the Question of Algorithmic Fairness

Background

In 2013, a man named Glenn Rodriguez appeared before a judge in a Wisconsin courtroom. He had been charged with a nonviolent offense. The court ordered a presentence investigation, which included running his profile through COMPAS — Correctional Offender Management Profiling for Alternative Sanctions — a risk assessment tool built by the company Northpointe (later renamed Equivant).

COMPAS assigned Rodriguez a risk score. The score was based on a proprietary algorithm that considered 137 questions — some drawn from his criminal record, others from a questionnaire covering factors like his employment history, social environment, and attitudes. The output was a number on a 1-to-10 scale representing his predicted likelihood of reoffending.

Rodriguez's score was high. The judge factored it into the sentencing decision. Rodriguez received a longer sentence than he might have otherwise.

Multiply this story by tens of thousands. COMPAS and tools like it have been used in courtrooms across the United States, influencing bail decisions, sentencing lengths, parole determinations, and supervision levels for millions of defendants. By the time journalists at ProPublica decided to investigate, algorithmic risk assessment had become a quiet but pervasive feature of the American justice system.

The ProPublica Investigation

In May 2016, ProPublica published "Machine Bias," a data-driven investigation of COMPAS that has become one of the most widely cited pieces of investigative journalism about AI.

The ProPublica team obtained risk scores for over 7,000 defendants arrested in Broward County, Florida, between 2013 and 2014. They then tracked those defendants for two years to see who was actually rearrested and who was not. This allowed them to compare the algorithm's predictions against real outcomes.

Their findings were stark:

  • Black defendants who did not go on to reoffend were nearly twice as likely as white defendants to be falsely labeled as high risk (44.9% false positive rate for Black defendants vs. 23.5% for white defendants).
  • White defendants who did go on to reoffend were nearly twice as likely as Black defendants to be falsely labeled as low risk (47.7% false negative rate for white defendants vs. 28.0% for Black defendants).
  • The overall accuracy of COMPAS was roughly 65% — only marginally better than untrained humans and far from the precision one might expect for a tool influencing liberty decisions.

ProPublica's central conclusion: COMPAS was biased against Black defendants. It systematically overpredicted risk for Black people and underpredicted risk for white people.

The Rebuttal

Northpointe responded quickly and forcefully. Their argument centered on a different fairness metric: predictive parity (also called calibration). Among defendants scored as high risk, the proportion who actually went on to reoffend was roughly the same for Black and white defendants — about 63%. In other words, when the algorithm said "high risk," it was equally accurate regardless of race.

This was not a rhetorical dodge. It was a mathematically valid claim. Predictive parity is a legitimate and widely used fairness metric. Northpointe's argument was that the algorithm treated Black and white defendants the same way in terms of the meaning of a high-risk designation.

The Mathematical Reality

The debate between ProPublica and Northpointe set off an extraordinary burst of academic research. Within months, teams of computer scientists, statisticians, and legal scholars published a series of papers that clarified the underlying mathematics.

The key result, demonstrated independently by several research groups (including Kleinberg, Mullainathan, and Raghavan at Cornell, and Chouldechova at Carnegie Mellon), was this:

When the base rate of the predicted outcome differs between groups, it is mathematically impossible to simultaneously satisfy three individually reasonable fairness criteria:

  1. Calibration (predictive parity): Among those labeled high risk, the same proportion actually reoffends in each group.
  2. Balance for the positive class (false negative rate parity): Among those who actually reoffend, the same proportion is correctly identified as high risk in each group.
  3. Balance for the negative class (false positive rate parity): Among those who do not reoffend, the same proportion is incorrectly labeled as high risk in each group.

In Broward County, the base rate of rearrest was higher for Black defendants than for white defendants — roughly 51% vs. 39%. Given this difference, the three criteria above are mutually exclusive. Satisfying any two necessarily violates the third.

This is not a software bug. It is not sloppy engineering. It is a mathematical fact — an impossibility theorem, as inescapable as the Pythagorean theorem. No amount of better data, smarter programming, or good intentions can overcome it.

What the Impossibility Means

The impossibility theorem does not mean fairness is hopeless. It means that the choice of which fairness criterion to prioritize is a human choice, not a technical one. The algorithm cannot resolve the question for us. We have to decide.

Consider the stakes of each choice:

  • Prioritize false positive rate parity (ProPublica's metric): Ensure that innocent people of all races face equal chances of being wrongly labeled as dangerous. This protects individuals from unjust punishment — but may reduce the tool's overall predictive accuracy.

  • Prioritize calibration (Northpointe's metric): Ensure that a "high risk" score means the same thing regardless of race. This makes the tool more consistent and interpretable — but allows the burden of false positives to fall disproportionately on one group.

  • Prioritize false negative rate parity: Ensure that actual reoffenders of all races are equally likely to be identified. This protects public safety equally — but may increase false positives for some groups.

Each choice reflects a different value. Each has defensible arguments. And each has real consequences for real people sitting in real courtrooms.

The Broader Context

The COMPAS debate is not just about one algorithm in one county. It illuminates several broader truths about AI in the justice system:

The problem is not unique to COMPAS. Any risk assessment tool trained on criminal justice data in the United States will encounter the same mathematical constraints, because the base rates of rearrest differ between racial groups — a difference driven at least in part by disparities in policing, prosecution, and systemic inequality.

"Race-blind" is not race-neutral. COMPAS does not use race as an input variable. But because the factors it does consider — criminal history, employment status, neighborhood characteristics — are correlated with race (due to the legacy of discrimination), the algorithm can reproduce racial disparities without ever "seeing" race.

Transparency does not automatically solve the problem. Even when ProPublica made its data and methodology public, and Northpointe provided its counteranalysis, the debate did not resolve. Both sides were correct by their own metrics. The core question — which metric should a society prioritize — is not answerable by data alone.

The baseline matters. COMPAS is roughly 65% accurate. Studies of human judges suggest they are no more accurate, and may be less consistent. The question is not "Is the algorithm perfect?" but "Is it better than the status quo?" — and "better by whose definition?"

Discussion Questions

  1. Values clarification: If you had to choose one fairness metric for a pretrial risk assessment tool, which would you choose — false positive rate parity, calibration, or false negative rate parity? Explain your reasoning, and describe who benefits and who bears the cost of your choice.

  2. Institutional design: Imagine a jurisdiction is committed to using a risk assessment tool despite the fairness impossibility. Design a set of procedural safeguards that would minimize the harm caused by whichever fairness criterion is sacrificed. Consider training, override protocols, review processes, and sunset clauses.

  3. Comparative perspective: The COMPAS debate played out largely in American terms — shaped by American racial history, American constitutional law, and American policing patterns. How might the analysis differ in a country with a different demographic composition and a different history of discrimination?

  4. Abolition vs. reform: Some scholars and advocates argue that the entire project of algorithmic risk assessment is irredeemable — that prediction in the criminal justice context inevitably reinforces systemic inequality, and the appropriate response is not to fix the tools but to abandon the prediction paradigm entirely. What are the strongest arguments for and against this position?

  5. Personal reflection: If you were a defendant facing a sentencing decision, would you want the judge to consult a risk assessment score? Under what conditions? How would your answer change if you knew the base rates for your demographic group were different from those of another group?

Mini-Project Options

Option A: Reproduce the analysis. ProPublica published its data and methodology. Using the publicly available Broward County dataset (available on GitHub), verify at least one of ProPublica's key findings — for example, the false positive rate disparity. You do not need to code this from scratch; walk-through tutorials are available. Document what you find and whether your results match ProPublica's.

Option B: Fairness metric comparison. Choose three fairness metrics (e.g., false positive rate parity, calibration, and demographic parity). For each, write a 200-word explanation of what it measures, what values it reflects, and in what contexts it would be most appropriate. Then explain the impossibility result in plain language.

Option C: Stakeholder analysis. Identify five stakeholder groups affected by COMPAS-type tools (e.g., defendants, judges, prosecutors, crime victims, community organizations). For each group, describe their likely position on algorithmic risk assessment and the values underlying that position. Where do the stakeholder positions align, and where do they conflict?

References and Further Reading

  • Angwin, J., Larson, J., Mattu, S., & Kirchner, L. (2016). Machine Bias. ProPublica. [Tier 1 — Primary investigative journalism]
  • Dieterich, W., Mendoza, C., & Brennan, T. (2016). COMPAS Risk Scales: Demonstrating Accuracy Equity and Predictive Parity. Northpointe Research Department. [Tier 1 — Vendor rebuttal with data]
  • Chouldechova, A. (2017). Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big Data, 5(2), 153–163. [Tier 1 — Peer-reviewed impossibility proof]
  • Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016). Inherent Trade-Offs in the Fair Determination of Risk Scores. arXiv preprint arXiv:1609.05807. [Tier 1 — Foundational mathematical analysis]
  • Dressel, J., & Farid, H. (2018). The accuracy, fairness, and limits of predicting recidivism. Science Advances, 4(1), eaao5580. [Tier 1 — Peer-reviewed comparison of COMPAS to untrained humans]