Case Study 30-2: ShotSpotter and the Evidence-Free Surveillance Tool

Overview

ShotSpotter (renamed SoundThinking in 2023) is an acoustic gunshot detection system deployed in approximately 120 American cities, primarily in urban neighborhoods with higher rates of gun violence. The system uses networks of sensors mounted on buildings and poles to detect, classify, and locate sounds that its AI identifies as gunshots, alerting police dispatchers in real time with the purported location of the shot.

ShotSpotter has been marketed to police departments as a public safety technology that provides immediate intelligence on gun crime — enabling faster police response, better documentation of gun violence that goes unreported, and more targeted enforcement of gun laws. Cities have paid millions of dollars annually for ShotSpotter contracts. Yet a sustained investigation by the MacArthur Justice Center, published in 2021, and subsequent independent analyses, found that the overwhelming majority of ShotSpotter alerts led to no evidence of a gun crime. Chicago's experience with ShotSpotter — culminating in the city's termination of its contract in 2024 — provides the most extensively documented case study of the gap between a surveillance technology's marketing claims and its documented operational reality.

How ShotSpotter Works

The Technology

ShotSpotter's system deploys acoustic sensors in a network covering a defined geographic area — typically a square mile per cluster of sensors. When the sensor network detects a sound, the audio is transmitted to ShotSpotter's automated processing system, which uses machine learning to classify the sound as a gunshot, firework, vehicle backfire, construction noise, or other category. If the system classifies the sound as a likely gunshot, it uses triangulation across multiple sensor readings to estimate the location and generates an alert to local police dispatch, typically within 60 seconds of the detected sound.

A critical element of ShotSpotter's process — one that was not widely known until investigative reporting — is human review. ShotSpotter employs acoustic experts who review AI-generated alerts and can modify the system's classification. If the AI classifies a sound as a gunshot, a ShotSpotter reviewer can confirm, modify, or cancel the alert before it reaches police dispatch. Crucially, these reviewers can also upgrade alerts: if the AI classifies a sound as a probable gunshot, a reviewer can reclassify it as a definite gunshot; if the AI classifies a sound as potentially something other than a gunshot, a reviewer can reclassify it as a gunshot.

This human review process raises questions about the nature of ShotSpotter's AI claims: if humans can override the AI's classification, the system's outputs are not purely algorithmic. And if humans are consistently overriding in a particular direction — upgrading uncertain classifications to definite gunshots — the system's claimed accuracy may reflect reviewer judgment as much as AI performance.

The MacArthur Justice Center Investigation

The Key Finding

In 2021, the MacArthur Justice Center — a civil rights law organization affiliated with Northwestern University School of Law — published an investigation of ShotSpotter's performance in Chicago. The investigation analyzed data from over 40,000 ShotSpotter alerts dispatched to Chicago police over a 21-month period.

The central finding: 89% of ShotSpotter alerts dispatched to Chicago police led to no evidence of a gun crime.

When Chicago police responded to a ShotSpotter alert, they found evidence of a gun crime in approximately 11% of cases. In 89% of cases — tens of thousands of police deployments — officers arrived at the scene and found nothing: no victim, no shooting suspect, no shell casings, no reports from residents, no corroborating evidence of a shooting.

This is an extraordinary false positive rate by any standard. If a police informant provided information that led to no evidence of the reported crime 89% of the time, that informant's reliability would be questioned and their tips would not routinely generate police deployment. Yet ShotSpotter alerts consistently triggered police response despite this documented rate.

The Racial Geography of ShotSpotter Deployment

The 40,000 alerts in the MacArthur investigation were not geographically uniform. ShotSpotter was deployed in specific Chicago neighborhoods — predominantly Black and Latino neighborhoods on the South and West sides. The investment of ShotSpotter sensor networks in these areas meant that residents of these neighborhoods were subject to much higher rates of police contact generated by ShotSpotter alerts than residents of other areas.

When police respond to ShotSpotter alerts and find no evidence of a gun crime, they sometimes find other bases for police-community interaction — stops, searches, inquiries — that can result in arrests for non-gun-related offenses or simply in police-community encounters that carry their own risks, particularly for residents of communities with strained police relationships. A system that generates 89% false alerts in heavily monitored minority neighborhoods while not being deployed in other neighborhoods creates a fundamentally asymmetric surveillance burden with documented racial geography.

The Quality of Alerts: Audio Evidence Problems

A separate dimension of the ShotSpotter investigation concerns the quality of audio evidence generated by the system and its use in criminal prosecutions.

APM Reports, in a 2021 investigation, documented cases in which ShotSpotter evidence was used in criminal prosecutions — cases where the prosecution's case rested partly on ShotSpotter's classification of a sound as a gunshot. Defense attorneys and independent acoustic experts who examined ShotSpotter audio evidence in specific cases raised serious concerns:

In a Chicago case involving Ernesto Lopez: ShotSpotter initially classified a sound as a firecracker; after the alert, a person was shot and killed. Reviewing the event retrospectively, ShotSpotter's staff analyst reclassified the alert from "firecracker" to "probable gunshot." The reclassified alert was used as evidence by prosecutors. Defense experts challenged the reclassification as retroactive alteration of evidence to fit an investigative theory.

In multiple cases: Defense attorneys found that ShotSpotter had revised its alerts after incidents occurred, changing classifications in ways that aligned with the prosecution's theory of the case. ShotSpotter has maintained that retrospective review and reclassification is a legitimate and disclosed part of its process; critics have characterized it as evidence manipulation.

The admissibility of ShotSpotter evidence in criminal prosecutions has been contested in multiple cases. Courts have generally admitted it, but with notable exceptions: an Illinois judge in 2022 excluded ShotSpotter evidence in a murder case, finding that the system's lack of independent validation and the opacity of its retrospective review process made the evidence unreliable under the applicable evidentiary standards.

Chicago's Experience: Escalation and Termination

The Contract History

Chicago first deployed ShotSpotter in 2012 in a limited area. The program expanded substantially over subsequent years, eventually covering much of the South and West sides. By 2021, Chicago was paying approximately $33 million over a three-year contract for ShotSpotter coverage. The program was championed by successive police superintendents and supported by community members in some neighborhoods who believed faster police response to gunshots would save lives.

The city's relationship with ShotSpotter was accompanied by persistent criticism from advocacy groups, defense attorneys, and some community members who argued that the system diverted police resources to ineffective alert response, subjected minority neighborhoods to disproportionate surveillance, and generated evidence problems in criminal prosecutions.

The Inspector General Investigation

In 2022, Chicago's Inspector General (OIG) released an audit of ShotSpotter that provided the most detailed official analysis of the system's performance in any city. The OIG found:

• CPD could not identify any cases in which ShotSpotter deployment had documented a measurable reduction in gun violence
• The system generated an average of 61 alerts per day, consuming significant police response resources
• ShotSpotter evidence had been used in prosecutions despite questions about reliability
• CPD lacked adequate policies for governing how ShotSpotter evidence should be used and disclosed

The OIG concluded that Chicago could not demonstrate that ShotSpotter was achieving its public safety objectives, and recommended that CPD either develop adequate metrics to evaluate the system or discontinue its use.

The Contract Termination

In February 2024, Chicago Mayor Brandon Johnson announced that the city would not renew its ShotSpotter contract, which expired in September 2024. The decision was significant: Chicago was the largest US city using ShotSpotter, and its contract termination represented a major loss for the vendor and a significant public statement about the technology's value proposition.

The decision was preceded by years of advocacy, the OIG report, and sustained attention from researchers and journalists. Mayor Johnson's administration cited the lack of evidence for ShotSpotter's effectiveness in reducing gun violence and concerns about its impact on over-policed communities.

SoundThinking (the rebranded ShotSpotter parent company) disputed the evidence used to support the decision and argued that the city had not adequately studied the program's benefits. The company continued marketing and operating in other cities following Chicago's exit.

The Vendor Accountability Problem

The Evidence Gap

ShotSpotter's marketing emphasizes rapid response, documentation of unreported gun violence, and improved police intelligence. What it does not prominently feature is independent, peer-reviewed evidence of effectiveness in reducing gun violence.

The evidence base for ShotSpotter's public safety benefit is thin and methodologically limited. The company has cited studies showing faster police response times following ShotSpotter alerts, and has commissioned reports suggesting positive outcomes in specific deployments. Independent evaluations, including a 2019 analysis by researchers at the City University of New York, found no statistically significant reduction in gun violence in areas where ShotSpotter was deployed. A 2020 Everytown for Gun Safety analysis found no significant difference in gun violence outcomes between cities that deployed ShotSpotter and those that did not.

The Chicago OIG's finding that CPD could not demonstrate measurable public safety benefit from ShotSpotter deployment, after years of operation, represents the most significant official documentation of this evidence gap.

The Procurement Problem

How cities come to deploy ShotSpotter without adequate evidence is a question about procurement practices that applies broadly to AI tools in public safety contexts.

ShotSpotter has been effective at marketing to city officials through claims of technological innovation, testimonials from police departments, and limited demonstrations of rapid response capability that are compelling in isolation. Cities that procure ShotSpotter typically do not require rigorous independent evidence of effectiveness as a condition of contract; they accept vendor-provided evidence or rely on peer city references. Once deployed, evaluation is difficult because comparison groups are hard to construct and attribution of changes in gun violence to a single intervention is methodologically challenging.

The procurement dynamic creates an accountability gap: the vendor has strong financial incentives to market effectively and limited legal obligation to provide rigorous effectiveness evidence; the city officials making procurement decisions often lack the technical capacity to evaluate evidence adequately; the communities affected by the deployment have limited formal voice in the procurement decision; and once the system is deployed, its expansion is easier than its termination because contractors can point to any positive outcomes while alternative explanations for negative outcomes.

This procurement dynamic — effective vendor marketing, inadequate evidence requirements, limited community voice, and deployment stickiness — is not unique to ShotSpotter. It characterizes the adoption of many AI tools by police departments and criminal justice agencies across the country.

The Manufacturer's Relationship With Evidence

ShotSpotter/SoundThinking's response to critical analysis of its technology has raised additional accountability questions. The company has sent legal threats to researchers and journalists who published critical analyses. It has challenged the methodology of studies that found poor performance while relying on self-generated or commissioned studies to support its claims. In the context of the APM Reports investigation on evidence manipulation, the company provided explanations for its retrospective review process that not all independent observers found satisfying.

This response pattern — aggressive defense of commercial reputation combined with limited support for independent evaluation — is common among AI vendors whose products face scrutiny but whose contractual relationships with government clients provide access to public funds. It illustrates the structural problem: the vendors who know most about their systems' performance have the least incentive to fund rigorous independent evaluation.

Analysis: ShotSpotter as a Template for AI Accountability in Public Safety

The Three Accountability Failures

The ShotSpotter case reveals three interrelated accountability failures that recur across AI in public safety contexts:

1. Evidence accountability failure: The deployment of a surveillance technology affecting millions of people across dozens of cities, with documented high false positive rates and no peer-reviewed independent evidence of effectiveness in reducing the violence it is deployed to address. The public safety benefit that justifies the surveillance burden has not been demonstrated; it has been asserted.

2. Procurement accountability failure: City governments spent millions of dollars annually on a technology that their own oversight bodies (Chicago OIG) could not find evidence of benefit for, because the procurement process did not require rigorous effectiveness evidence and the ongoing management of the deployment did not include meaningful outcome evaluation.

3. Criminal justice accountability failure: Evidence generated by a system with documented reliability problems was used in criminal prosecutions, creating a risk of wrongful conviction grounded in technology that its developer retrospectively modified. The lack of clear evidentiary standards for AI-generated evidence in criminal proceedings allowed evidence of questionable reliability to reach juries without adequate scrutiny.

Implications for AI Procurement in Public Safety

The ShotSpotter case provides a template for improved AI procurement in public safety contexts:

Cities and counties considering AI public safety technologies should require independent, peer-reviewed effectiveness evidence as a condition of procurement — not vendor-commissioned studies or testimonials from police departments, but rigorous evaluation by researchers without financial ties to the vendor.

Contracts should include performance accountability clauses: specific, measurable outcomes that the technology is expected to achieve, with contract termination provisions if outcomes are not met within defined periods.

Community input mechanisms should be built into procurement processes for surveillance technologies that will disproportionately affect specific communities, with genuine procedural standing rather than advisory status.

Evidence generated by AI systems should be subject to clear standards for use in criminal proceedings, including disclosure of error rates, validation evidence, and any retrospective modification of alerts before their use as evidence.

Independent oversight bodies — inspectors general, independent auditors, academic evaluators — should have routine access to AI system performance data sufficient to evaluate whether deployed technologies are achieving their stated purposes.

Discussion Questions

1. Chicago paid approximately $33 million over three years for a ShotSpotter system that its own Inspector General found no measurable evidence of benefit from. What procurement reforms would prevent similar expenditures?

2. ShotSpotter's practice of retrospective review — reclassifying alerts after an incident occurred — is presented by the company as a quality improvement mechanism. Critics characterize it as evidence manipulation. How should courts evaluate ShotSpotter evidence, and what disclosure requirements should apply when it is used in criminal proceedings?

3. ShotSpotter is deployed primarily in Black and Latino neighborhoods. Is this distribution of surveillance justified by higher rates of gun violence in those areas, or does it constitute constitutionally and ethically problematic racial targeting?

4. If ShotSpotter generates police response to locations where crime is occurring only 11% of the time, what are the consequences of the other 89% of police deployments? What does research on police-community contact in over-surveilled communities suggest about the costs of high false positive policing interventions?

5. The vendor challenged and criticized independent researchers who found poor performance. What institutional structures could protect independent research on AI public safety tools from vendor pressure while maintaining appropriate academic standards?