Case Study 01: The Algorithm in the Dock

Automated Decision-Making in the Criminal Justice System

Introduction

In 2013, a Wisconsin man named Eric Loomis stood before a judge for sentencing following a conviction related to a drive-by shooting. He had not fired the weapon, but had been convicted of attempting to flee an officer and operating a vehicle without the owner's consent. The judge consulted, among other materials, a risk score generated by a software system called COMPAS. The score labeled Loomis as presenting a "high risk" of recidivism. The judge sentenced him to six years — the maximum available for the charges.

When Loomis challenged the sentencing on constitutional grounds, his case traveled to the Wisconsin Supreme Court. The court's 2016 decision in State v. Loomis has since become one of the most-discussed judicial precedents on algorithmic governance in the United States — not because it resolved the underlying questions about AI and due process, but because it failed to do so while making clear that the questions were real, significant, and unanswered.

This case study examines the proliferation of algorithmic risk assessment in US criminal justice, the constitutional challenge Loomis brought, what we know and don't know about how these tools work, and what transparency in this domain would actually require.

1. The Proliferation of Risk Assessment Tools in US Criminal Justice

Risk assessment in criminal justice is not a new concept. For decades, judges, parole boards, and pretrial release officers have used structured risk instruments — standardized questionnaires that produce numerical risk classifications — to inform decisions about detention, sentencing, and supervision. These tools replaced purely intuitive judicial judgment with a systematized, ostensibly evidence-based process.

The modern generation of algorithmic risk tools emerged in the 1990s and 2000s, driven by the development of more sophisticated statistical and machine learning methods and the availability of large criminal justice datasets. By 2019, risk assessment tools were in use in criminal justice systems in at least 46 US states, applied at various stages of the justice process: pretrial release decisions, sentencing, parole and probation determinations, and post-release supervision intensity.

COMPAS — Correctional Offender Management Profiling for Alternative Sanctions — is among the most widely deployed. Developed by the firm Northpointe (now Equivant), it has been used in Wisconsin, California, New York, Florida, Michigan, and many other states. Other significant tools include the LSI-R (Level of Service Inventory — Revised), the PSA (Public Safety Assessment), the Ohio Risk Assessment System, and Virginia's risk assessment instrument used in sentencing.

These tools share a common logic: they take as inputs a set of factors — criminal history, demographic information, questionnaire responses about lifestyle and social circumstances — and produce as output a numerical score or categorical risk classification. High scores are associated with predictions of higher recidivism risk; low scores suggest lower risk. Decision-makers are intended to use these scores as one input among many, though in practice the influence of a quantitative risk score on judicial behavior is difficult to measure and almost certainly varies widely.

The stated rationale for these tools is evidence-based justice: replacing the demonstrated inconsistencies of unstructured judicial intuition with systematic, validated instruments. Research has consistently shown that judicial risk assessment through pure intuition is unreliable, inconsistent, and often influenced by extra-legal factors including the defendant's demeanor, articulateness, and appearance. Structured instruments, the argument goes, introduce consistency and reduce the role of irrelevant factors.

That argument has merit. But it comes bundled with a set of governance problems that the criminal justice system has been slow to recognize and address.

2. Loomis v. Wisconsin: The Constitutional Challenge and Its Outcome

Eric Loomis's challenge to his sentence focused on three specific constitutional arguments.

The accuracy argument. Loomis argued that because he could not access the COMPAS algorithm, he could not verify that the score assigned to him accurately reflected his actual circumstances. The algorithm processed his responses to a questionnaire and his official criminal history records — but without knowing how those inputs were processed, Loomis could not identify whether errors in the inputs had affected the output, or whether the algorithm's processing of his particular circumstances was appropriate.

The individualized sentencing argument. Wisconsin constitutional and statutory law requires that sentences be individualized — tailored to the specific circumstances of the specific defendant. COMPAS scores are population-level predictions: the algorithm predicts the probability that a person with a given profile will recidivate, based on the historical recidivism rates of people with similar profiles. Applying this population-based prediction to an individual sentence is conceptually problematic: the individual defendant is not the population cohort, and a 60% population recidivism rate says nothing definitive about any individual member of that population.

The gender and race discrimination argument. The COMPAS questionnaire uses different scoring norms for men and women, meaning that equivalent responses yield different scores depending on the respondent's gender. Loomis argued this amounted to sex discrimination. He also raised race discrimination concerns, noting that variables correlated with race were used as inputs.

The Wisconsin Supreme Court rejected all three arguments. On the accuracy argument, the court noted that defendants could challenge the factual information in the presentence report that fed into the score. On individualized sentencing, the court held that the judge had considered many factors beyond the COMPAS score. On discrimination, the court noted that COMPAS does not use race as a direct input, and that using different actuarial tables for different groups does not per se constitute discrimination.

Most significantly, the court held that while defendants have no right to know the algorithm's specific proprietary methodology, using the score was permissible provided the judge was aware of its limitations and treated it as only one factor in a multi-factor analysis.

The US Supreme Court declined to hear the case in 2017, letting the Wisconsin decision stand.

The Loomis holding does not close the due process question — it avoids it. By framing the issue as whether the judge improperly relied on COMPAS, rather than whether COMPAS's opacity is itself unconstitutional, the court sidestepped the fundamental problem: that meaningful due process requires the ability to challenge the basis for adverse governmental decisions, and that opacity systematically prevents such challenges.

3. What Is Known About How These Tools Work

The opacity of proprietary risk assessment tools is not uniform — different tools disclose different levels of information.

COMPAS. The most publicly documented aspects of COMPAS include: (1) the general conceptual domains it covers (criminal involvement history, social environment, residential stability, family issues, school and work situation, peer associations, substance abuse); (2) aggregate validation statistics (published in Northpointe's practitioner guide and in peer-reviewed research); and (3) the general questionnaire items, which have been analyzed by researchers. What is not disclosed is the precise mathematical algorithm — the specific weights assigned to each item, the interactions among items, the transformation functions applied, and the precise decision rules for assigning risk categories.

This means that a researcher, a defense attorney, or a defendant can read the questionnaire but cannot determine how answers are converted into scores. The scoring function is opaque.

LSI-R. The Level of Service Inventory — Revised is somewhat more transparent; the scoring methodology has been published in academic literature, though the full instrument and scoring guide are proprietary products sold by Multi-Health Systems.

PSA. The Public Safety Assessment, developed by the Arnold Foundation (now Arnold Ventures), represents the most transparent end of the spectrum among widely used tools. The PSA's algorithm has been published — it uses nine specific criminal history variables with specific weights, and the formula is publicly available. Any defendant or researcher can understand exactly how a PSA score is calculated.

4. The Trade Secrecy Defense: Legitimate Business Interest vs. Due Process Right

Northpointe's trade secrecy claim is not frivolous as a matter of intellectual property law. COMPAS represents a substantial investment in research and development. The specific algorithm — the weighting scheme, the interaction terms, the scoring logic — reflects proprietary modeling choices that differentiate COMPAS from competitors. Disclosure would allow competitors to replicate the core product.

But the trade secrecy defense in the criminal justice context has a deeply problematic structure: it allows a private company to profit from a government process while insulating the critical tool in that process from the scrutiny that due process requires. The state uses COMPAS's scores to deprive people of liberty; the company that produces the scores is shielded from accountability by IP law.

This is not a unique problem. Expert witnesses routinely provide testimony in criminal trials about the results of proprietary methodologies — forensic software, drug testing kits, DNA analysis tools — and courts have had to grapple with how much disclosure is required to allow cross-examination. The analogy is instructive: in the Daubert framework for expert scientific testimony, courts assess reliability and scientific validity, which requires at least some disclosure of methodology. The question is why the same framework should not apply to algorithmic expert evidence.

Several commentators have argued that when the government uses a proprietary algorithm in a context where due process rights are at stake, the government's use of that algorithm effectively converts the proprietary information into a matter of public concern — and that trade secrecy protections should yield, at least partially, to due process requirements. Under this view, the government could license COMPAS only subject to a condition allowing defendants' experts to examine the algorithm under appropriate confidentiality protections.

This approach would not require full public disclosure. Expert examination under confidentiality — a common arrangement in patent litigation — would allow defense experts to assess the algorithm without destroying its commercial value. Several courts have used similar arrangements in other contexts. There is no obvious legal obstacle to this approach; it would simply require courts to insist on it.

5. The Arnold Foundation's PSA: A More Transparent Alternative

The Public Safety Assessment was developed by the Laura and John Arnold Foundation (now Arnold Ventures) beginning around 2013 and is now in use in several US jurisdictions for pretrial risk assessment. The PSA offers a case study in what greater transparency looks like in practice.

The PSA uses nine criminal history variables: age at current arrest, whether the current offense is violent, whether the person had a pending charge at the time of the current offense, prior misdemeanor convictions, prior felony convictions, prior violent convictions, prior failure to appear in the past two years, prior failure to appear older than two years, and prior incarcerations. The algorithm assigns specific numerical weights to each factor and combines them into a composite score. This entire methodology is publicly available in peer-reviewed publications and on the Arnold Ventures website.

The transparency of the PSA has several practical benefits. Defense attorneys can check whether the factual inputs are accurate — if a client has no prior felony convictions but the PSA score reflects two, the error can be identified and challenged. Researchers can assess whether the weights are appropriate and whether the instrument performs fairly across demographic groups. Jurisdictions considering adoption can evaluate the tool's methodology before deployment. And affected defendants understand, to a degree not possible with COMPAS, why they received the score they did.

The PSA is not without controversy. Critics have argued that because it relies primarily on criminal history, and because criminal history reflects historical patterns of racially biased policing and prosecution, PSA scores may incorporate and perpetuate racial inequities in the criminal justice system. These are valid concerns — and they are much easier to investigate with a transparent instrument than with an opaque one. The transparency of the PSA enables critics to engage substantively with its methodology in a way that opacity forecloses.

6. What Transparency Would Require: Open-Source vs. Documented Proprietary

Transparency in algorithmic criminal justice tools does not require that all tools be open-source, though open-source development has significant advantages. A range of transparency measures can be distinguished:

Full open-source disclosure: The algorithm, training data, and implementation code are publicly available. Any researcher or developer can inspect, replicate, and critique. The PSA does not quite reach this standard (the implementation code is not open-source), but its published methodology comes close.

Documented proprietary with expert access: The algorithm is proprietary, but qualified experts — defense attorneys' experts, court-appointed technical advisors — can access the full documentation under confidentiality protections. This is analogous to how confidential business information is handled in civil litigation and is a viable middle path.

Validated proprietary: The algorithm is proprietary, but the developer is required to provide standardized validation reports to regulatory or judicial authorities, documenting performance by demographic group, error rates, and predictive accuracy. External reviewers assess these reports. This is weaker than expert access but substantially better than the current baseline.

Undisclosed proprietary: The current status of most commercial criminal justice AI tools. No independent validation review, no expert access, no transparency beyond the vendor's own assertions. This is the baseline that Loomis implicitly endorsed.

Advocates in the criminal justice reform community have argued that at minimum, the "validated proprietary" standard should be required for any tool used in consequential criminal justice decisions, and that the "documented proprietary with expert access" standard should be the preferred approach.

7. European Comparison: GDPR's Article 22 Would Bar Most of These Uses

The European Union's General Data Protection Regulation provides a sharp contrast to the US approach. Article 22 provides that data subjects have the right not to be subject to a decision based solely on automated processing that significantly affects them, unless the data subject has consented, the decision is necessary for a contract, or it is authorized by member-state law. Pretrial detention and sentencing decisions obviously significantly affect defendants.

Criminal justice decisions in EU member states would almost certainly require a human decision-maker to genuinely review and endorse the risk assessment — a human who is accountable and can explain the basis for the decision. The use of an algorithmic score as a significant input without meaningful human engagement, of the kind that Loomis permitted, would likely be difficult to square with Article 22's requirements in many EU jurisdictions.

Moreover, the GDPR's general accountability and data protection principles — data minimization, purpose limitation, and the requirement to maintain records of processing — would impose documentation requirements that go well beyond what US criminal justice AI governance currently demands.

The EU AI Act, which entered into force in 2024, explicitly categorizes AI used in criminal justice risk assessment as "high-risk," requiring conformity assessments, technical documentation, logging, and human oversight before deployment.

This does not mean the European approach is problem-free. Member states have implemented Article 22 inconsistently, and the "authorized by law" exception is capacious. But the European framework at least begins from the premise that automated decisions with major consequences require transparency and human accountability — a premise that US law has not yet accepted.

8. The Reform Movement: New Jersey's Response and Louisiana's Disclosure Law

Criminal justice reform advocates have pursued several strategies to address the opacity of algorithmic risk assessment.

New Jersey implemented a sweeping bail reform in 2017, largely eliminating cash bail for most offenses and replacing it with risk-based release decisions using the PSA. The choice of the PSA — a transparent instrument with published methodology — was deliberate. New Jersey also created a monitoring infrastructure, requiring detailed reporting on pretrial release outcomes by demographic group, enabling ongoing evaluation of whether the system is producing fair outcomes.

Louisiana enacted legislation in 2017 requiring disclosure of the algorithm used in any risk assessment tool used in criminal justice decisions. The law requires that the State Division of Probation and Parole provide defendants with the specific factors considered in their risk assessment and the weight given to each factor. This represents a meaningful step toward transparency, though it falls short of full algorithmic disclosure.

New York State imposed a moratorium on certain uses of algorithmic risk assessment in sentencing, reflecting concerns about the tools' fairness and the lack of adequate transparency and validation requirements.

The Pretrial Justice Institute and other advocacy organizations have published standards for the appropriate use of risk assessment tools, including transparency requirements, demographic performance reporting, and the principle that risk assessment tools should inform but not replace human judgment.

These reform efforts are meaningful but remain incomplete. The fundamental problem — that proprietary vendors can supply algorithmic tools to government agencies that use them to make consequential decisions, without adequate transparency or accountability — has not been legislatively resolved at the federal level.

9. The Broader Lesson: Institutional Opacity in Government AI

The COMPAS case illustrates a broader governance problem that extends well beyond criminal justice. When governments rely on proprietary AI systems for consequential decisions, several structural problems emerge:

Accountability displacement. The government decision-maker can blame the algorithm; the algorithm vendor can say they only provided a tool; no one is fully accountable for the outcome.

Regulatory capture. Vendors who supply AI tools to government agencies develop relationships with those agencies that create barriers to switching tools or imposing accountability requirements. The vendor becomes an entrenched supplier rather than an accountable contractor.

The legitimacy deficit. Democratic legitimacy requires that government decisions be traceable to human judgment exercised according to publicly known rules. When an opaque algorithm substitutes for human judgment, the basis for democratic accountability is undermined.

The transparency ratchet. Once an opaque system is deployed and decision-makers become reliant on it, it becomes progressively harder to impose transparency requirements — the vendor resists, the agency is dependent, and the political cost of disruption is high.

The lesson is that transparency requirements must be imposed before procurement, not retrofitted after deployment. Government agencies should require as a condition of procurement that any AI system used in consequential decisions must meet specified transparency standards — and those standards should be set by legislatures or regulatory bodies rather than left to vendor discretion.

10. Discussion Questions

1. The Loomis decision holds that defendants do not have a constitutional right to know the precise algorithm behind their COMPAS score. Is this holding consistent with the due process values the Constitution is designed to protect? What would a more robust due process standard require, and how would courts enforce it against proprietary vendors?

2. The Arnold Foundation's PSA demonstrates that it is possible to build a widely used criminal risk assessment tool with a fully published algorithm. Why do you think most commercial tools are not similarly transparent? What market or regulatory conditions would need to change to make transparency the default?

3. Northpointe argues that COMPAS does not use race as an input variable, and therefore does not racially discriminate. ProPublica's analysis found that COMPAS produces racially disparate outcomes — higher false positive rates for Black defendants. How should courts and policymakers evaluate this conflict? Does the absence of race as a direct input rule out racial discrimination?

4. Several states have required disclosure of risk assessment algorithms as a condition of their use in criminal justice. What are the potential drawbacks of mandatory disclosure requirements? Could disclosure create new problems — such as defendants or their counsel gaming the assessment, or enabling wealthy defendants to benefit disproportionately from transparency?

5. The EU AI Act classifies criminal justice risk assessment as "high-risk" AI requiring extensive documentation, human oversight, and conformity assessment. The United States has no equivalent federal requirement. Should the US adopt a similar framework? What obstacles — political, legal, practical — would such a framework face?

This case study should be read alongside Chapter 13 (Section 13.4 on the COMPAS opacity problem and Section 13.7 on sector-specific transparency obligations) and in connection with Chapter 15 (Fairness Metrics and the Limits of Quantified Equity) and Chapter 18 (Accountability).