Case Study 20-2: COMPAS and Due Process — What Constitutional Law Says About Algorithmic Risk Assessment

Overview

In February 2016, Eric Loomis was sentenced to six years in prison for attempting to flee a police officer and other charges in La Crosse County, Wisconsin. The sentencing judge had before him a COMPAS risk assessment generated by Northpointe — the same tool discussed in Chapters 9 and 19 — that classified Loomis as "high risk" for recidivism and medium-high risk of violence. The judge specifically referenced the COMPAS assessment in his sentencing remarks.

Loomis challenged his sentence, arguing that the use of COMPAS violated his right to due process under the Fourteenth Amendment. He raised three specific arguments: that using COMPAS in sentencing violated his right to be sentenced based on accurate information, because COMPAS's accuracy was contested; that COMPAS's use of gender as an input variable violated equal protection; and that the algorithm's opacity — its scores were generated by a proprietary system that neither Loomis nor his attorneys could examine — violated his right to a meaningful opportunity to respond to information used against him.

The Wisconsin Supreme Court rejected all three arguments. Loomis v. Wisconsin (2016) became the first state supreme court decision addressing the constitutionality of AI risk assessment in criminal sentencing, and it set a precedent — one that has been both followed and criticized — for how courts address the due process implications of algorithmic decision-making.

Background: COMPAS and Its Use in Criminal Justice

COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) is a commercial risk assessment tool developed by Northpointe (later Equivant). It generates risk scores based on a questionnaire covering criminal history, social background, and attitudes. In many jurisdictions where it is used, COMPAS scores are incorporated into pre-sentencing reports — the documents that judges review before imposing sentence.

The use of risk assessment tools in criminal sentencing has a long history predating AI: actuarial risk assessment instruments have been used in parole and probation decisions since the 1920s. COMPAS represented a new generation of these tools: more systematic, automated, and commercially provided. By the mid-2010s, COMPAS was being used in courts across the United States, and risk assessment tools generally had become integral to "evidence-based sentencing" reforms championed by both progressive and conservative criminal justice reformers.

Why Risk Assessment Tools Are Used

Proponents of risk assessment tools in criminal justice argue that they reduce disparities in sentencing by providing a systematic, data-based alternative to purely subjective judicial assessment. If judges assess risk idiosyncratically — relying on intuitions that may reflect implicit racial bias — replacing subjective judgment with an actuarial tool should reduce racial disparities, the argument goes. Research on the alternative — sentencing without risk assessment tools — shows that unguided judicial discretion produces its own racial disparities.

Opponents argue that actuarial risk assessment encodes historical disparities (because predictions are based on historical data from a racially biased criminal justice system), provides false precision (scores carry the appearance of objective science), and shifts accountability from humans to algorithms in a domain where the stakes — liberty — are among the highest imaginable.

Loomis v. Wisconsin: The Constitutional Arguments

The Accuracy Argument

Loomis's first argument was that COMPAS's scores were not proven accurate enough to support the substantial weight the sentencing court gave them. Due process requires that sentencing be based on accurate information; a sentencing tool that cannot be validated — or that has known accuracy problems — should not be used, the argument goes.

The Wisconsin Supreme Court rejected this argument, finding that the sentencing court did not impose sentence based solely on the COMPAS assessment. The judge had considered multiple sources of information, and the COMPAS score was one factor among many. The court held that COMPAS had been validated and that its use as one factor in a broader sentencing analysis was consistent with due process.

This analysis is legally defensible but incomplete. The validation evidence for COMPAS was disputed — ProPublica's analysis (published in May 2016, shortly after the Wisconsin Supreme Court heard argument in Loomis) found significant racial disparities in COMPAS's predictive accuracy. The court's reliance on validation evidence without engaging with the contested accuracy research reflects a pattern that has persisted across AI-related constitutional challenges: courts defer to official validation processes without critically examining their quality.

The Gender Argument

Loomis's second argument was that COMPAS uses gender as an explicit input variable — generating different scores for men and women based on gender — in violation of equal protection. The use of gender as a factor in criminal sentencing is constitutionally and legally contested: the Supreme Court has held in other contexts that explicitly gendered distinctions in official action require justification under intermediate scrutiny.

The Wisconsin Supreme Court dismissed this argument, finding that because gender was a factor in the general population scoring methodology (not in the specific "risk of recidivism" score used in Loomis's case), the gender-specific component of COMPAS did not affect his sentence. This reasoning is technical and limited; it does not address the broader question of whether using gender as an explicit variable in criminal sentencing is constitutionally permissible.

The gender question is significant beyond Loomis's specific case. Many risk assessment instruments use gender as a predictor, and there is research supporting its predictive validity (men have higher recidivism rates than women in the historical data on which these tools were trained). But using gender as a sentencing factor raises equal protection concerns that are not resolved by predictive validity — a sentencing factor that disadvantages men because men (as a group) reoffend more often than women is exactly the kind of group-based classification that equal protection doctrine was designed to scrutinize.

The Opacity Argument

The most legally significant argument in Loomis was the opacity challenge: COMPAS's algorithm was a trade secret, and neither Loomis nor his attorneys could examine how the score was generated. Due process requires a meaningful opportunity to respond to information used against you at sentencing. How can a defendant meaningfully challenge a score when he cannot examine the algorithm that generated it?

The Wisconsin Supreme Court dismissed this argument by noting that Loomis had access to the COMPAS questionnaire responses — the inputs — and that he could challenge the accuracy of those inputs. The court found that this was a sufficient opportunity to challenge the COMPAS assessment, even without access to the algorithm.

This reasoning is widely criticized by legal scholars. Challenging the inputs of an opaque algorithm is not the same as challenging the algorithm itself. Even if Loomis's questionnaire responses were accurate, the algorithm's weighting of those responses — which determines the score — is inaccessible and unchallengeable. A defendant who was falsely scored as high-risk because the algorithm improperly weighted certain factors (a factor correlation that inflates risk for people with Loomis's specific background, perhaps) would have no way to identify or challenge that error.

The court's response — that COMPAS had been validated, and that defendants could challenge the general validity of the tool — does not address the individual error concern. Population-level validation establishes that the tool is accurate for the population on average; it does not establish that the specific score for the specific defendant is accurate. The defendant's due process interest is in the accuracy of their specific score, not the population average.

Post-Loomis Developments

Other Courts' Responses

Courts in other jurisdictions have reached different conclusions on the due process implications of algorithmic risk assessment.

In State v. Bullock (Montana, 2021), the Montana Supreme Court held that using a specific AI risk assessment tool (the LSI-R) in parole decisions violated due process because the defendant was not given access to information about how the tool worked. The court found that mere access to the questionnaire inputs was insufficient — defendants must be able to meaningfully challenge the algorithmic process.

Federal courts have generally been reluctant to engage with due process challenges to risk assessment tools, typically finding that judicial notice of multiple factors — including the risk score — satisfies due process, without engaging seriously with the opacity problem.

In Eckles v. State (Colorado, 2022), the Colorado Court of Appeals held that a sentencing court's reliance on a commercial risk assessment tool without any validation evidence specific to Colorado defendants raised insufficient due process concerns — while acknowledging the academic debate about risk assessment accuracy across demographic groups.

The pattern across these cases shows substantial judicial deference to algorithmic risk assessment, limited engagement with the technical accuracy and bias debates, and a tendency to resolve constitutional challenges through procedural adequacy findings rather than substantive analysis.

Legislative Responses

Several jurisdictions have enacted or proposed legislation restricting or governing risk assessment tools in criminal justice:

Illinois enacted legislation in 2020 requiring that risk assessment tools used in sentencing be validated for the specific population to which they are applied, be regularly updated, and be reviewed for racial bias.

California has had ongoing legislative debates about prohibiting or restricting the use of commercial risk assessment tools in criminal sentencing, driven by concerns about racial bias documented by the ProPublica analysis and subsequent research.

New Jersey uses a risk assessment tool (the Arnold Public Safety Assessment) in bail decisions, with specific transparency features: the algorithm is open-source, validation studies are publicly available, and the tool's factors are disclosed to defendants.

New Jersey's approach demonstrates that the opacity problem is not technically necessary: algorithmic risk assessment tools can be designed to be transparent, publicly available, and externally validatable. The opacity of COMPAS is a consequence of Northpointe's commercial decision to treat the algorithm as a trade secret — not a technical necessity.

The Structural Due Process Problem

What Due Process Requires

Due process, at its core, requires that government action depriving individuals of liberty or property be preceded by notice of the basis for the action and an opportunity to be heard. In criminal sentencing, this means the defendant must be informed of the information the court is considering, must have an opportunity to challenge that information, and must receive a reasoned explanation for the sentence imposed.

Algorithmic risk assessment challenges these requirements in several ways:

Notice is inadequate when the algorithm is a trade secret. A defendant who receives a COMPAS score without access to the algorithm that generated it has received a number, not an explanation.

Opportunity to challenge is inadequate when the algorithm cannot be examined. Challenging the inputs (which the defendant can access) is not equivalent to challenging the algorithm's weighting of those inputs (which the defendant cannot access).

Reasoned explanation is inadequate when the sentencing court relies on a score without engaging with its derivation. A judge who says "the risk assessment tool says high risk, and I give that significant weight" has not provided the kind of explanation that due process requires — because the score itself requires explanation.

The Accountability Vacuum

The opacity of commercial risk assessment tools creates an accountability vacuum analogous to the accountability gap discussed in Chapter 18. When a defendant is sentenced based on a COMPAS score, and the sentence is appealed, and the appellate court defers to the trial court's use of the score — no one has actually evaluated whether the score was accurate for this defendant.

The accountability vacuum is structural: it results from the combination of trade secrecy (protecting the algorithm from disclosure), judicial deference (accepting algorithmic outputs without examining their derivation), and limited appellate review (which rarely engages with the accuracy of individual algorithmic assessments). Addressing the vacuum requires structural reforms: open-source algorithms or mandatory disclosure in criminal proceedings, defendants' right to challenge algorithmic evidence with appropriate technical assistance, and meaningful appellate review of algorithmic accuracy claims.

Implications for AI Liability in Criminal Justice

The Loomis case and its successors reveal several important points about AI liability in the criminal justice context:

Constitutional claims face high barriers. Due process claims against risk assessment tools have generally failed, reflecting a combination of judicial deference to algorithmic tools that are formally validated and courts' limited technical capacity to evaluate algorithmic accuracy. Constitutional reform of AI in criminal justice requires either better legal doctrine (more searching due process review) or legislative change.

Civil rights claims are more promising. Claims under 42 U.S.C. § 1983 (civil rights claims against state actors) for racial discrimination — arguing that using a racially disparate risk assessment tool in sentencing violates equal protection — have not yet been fully litigated. The ProPublica findings could support such a claim, though the legal standards for equal protection challenges to facially neutral practices are demanding.

Commercial actors in government contexts raise special accountability challenges. Northpointe/Equivant is a private company providing services to the government. Government use of private AI tools creates accountability gaps: the government actor (the court) argues the algorithm is validated; the private actor (Northpointe) argues the algorithm is proprietary; and the defendant falls between them, unable to challenge the tool that influences the deprivation of their liberty.

Transparency requirements are the most tractable reform. Making risk assessment algorithms used in criminal justice publicly available — as New Jersey has done — addresses the opacity problem without requiring resolution of the deeper fair use and trade secret questions. Where transparency is legally feasible (government use of algorithms), it should be the baseline expectation.

Discussion Questions

1. The Wisconsin Supreme Court held that access to COMPAS questionnaire inputs was sufficient for due process purposes, even without access to the algorithm. Do you agree? What would "meaningful opportunity to challenge" an algorithmic risk assessment require?

2. COMPAS uses gender as an input variable, and it produces racially disparate false positive rates. Both features raise constitutional concerns under equal protection. How would you assess each under the relevant equal protection standards?

3. New Jersey's use of the Arnold Public Safety Assessment — an open-source, publicly validated tool — demonstrates that algorithmic transparency in criminal justice is feasible. What obstacles prevent other jurisdictions from adopting similar transparency requirements for the tools they use?

4. The Loomis decision has been cited by subsequent courts as precedent for using risk assessment tools in sentencing. How would you distinguish Loomis in a challenge to a risk assessment tool that: (a) has been found to have larger racial disparities than COMPAS; (b) is used for bail determination rather than sentencing; or (c) is used to determine mandatory minimum sentences rather than within-guidelines sentencing?

5. Should criminal defendants have a right to conduct discovery from AI vendors — to obtain access to proprietary algorithms and training data — when those algorithms are used in their criminal cases? What arguments support and oppose such a right, and how would trade secret concerns be balanced against defendants' due process interests?