Case Study 38.1 — The NAS and PCAST Reports: The Reform That Was Demanded, and the Reform That Didn't Happen

A note on sourcing and tone. This case study treats two real, publicly available federal documents — the 2009 National Academy of Sciences report Strengthening Forensic Science in the United States: A Path Forward and the 2016 report of the President's Council of Advisors on Science and Technology, Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature-Comparison Methods — and the institutional response to them. These are Tier-1 sources; the facts below (the reports' existence, their core findings and recommendations, and the broad shape of the response, including the lapse of the National Commission on Forensic Science in 2017 and the Department of Justice's formal disagreement with PCAST) are matters of public record. Where the pace or completeness of a reform is summarized, we speak qualitatively and avoid invented percentages. The point of this case study is not a single trial but the largest reform story in the modern history of the field — and its sobering, instructive incompleteness.

Background: how two reports came to be written

By the mid-2000s, the pressure on forensic science had become impossible to ignore. The DNA exonerations (Chapter 6) had moved from a trickle to a flood, and when researchers studied the closed files, flawed or misapplied forensic science turned up as a contributing factor in roughly half of them. The field's own assurances — "trust this, it's science" — were colliding with documented proof that the science had helped convict innocent people. Something had to give.

In 2005, Congress directed the National Academy of Sciences to conduct a comprehensive, independent assessment of the state of forensic science in the United States. A committee of scientists, statisticians, forensic practitioners, and legal experts spent several years on the task. The result, published in 2009, was the most authoritative examination the field had ever received — and its conclusions were devastating in their understatement. The report found American forensic science "badly fragmented," lacking uniform standards and a central scientific authority, and — in its single most-quoted finding — concluded that, apart from nuclear DNA analysis, no forensic method had been rigorously shown to reliably connect evidence to a specific source. The disciplines, it found, had largely entered the courtroom on the authority of their practitioners rather than through scientific validation.

Seven years later, a second body sharpened the diagnosis. The President's Council of Advisors on Science and Technology — senior scientists advising the President — published a 2016 report focused specifically on the feature-comparison methods (fingerprints, firearms, bite marks, hair, and the rest). PCAST asked a single, scientist's question of each method: has it been shown, by appropriately designed empirical studies, actually to work, with a measured error rate? It introduced the now-central distinction between foundational validity (is the method itself valid?) and validity as applied (was it applied properly in this case?), and it rendered verdicts: some methods met the bar, some did not, and bite marks failed it outright with little prospect of rescue.

Together, these two reports are the intellectual foundation of forensic reform. They did not merely criticize. Each laid out, in detail, what the field must do.

What was demanded

The reports' recommendations, taken together, amount to a complete reform agenda. The chapter (§38.3) collects them; the essentials:

• Structural independence. Remove crime laboratories from the administrative control of law enforcement and prosecutors. (NAS's deepest structural recommendation.)
• A strong national authority. Create an independent federal entity to set standards, fund research, and oversee the field — NAS proposed a "National Institute of Forensic Science."
• Foundational-validity research. Fund and conduct the black-box studies that would establish, for each method, whether it works and how often it errs.
• Admit only what is validated. PCAST: feature-comparison methods lacking established foundational validity should not be admitted; those that are valid should be testified to only with their error rate.
• Honest, standardized reporting. End "individualization," "zero error rate," and "reasonable degree of scientific certainty" overstatement.
• Cognitive-bias safeguards. Study and adopt context management against the contextual bias that contaminates interpretation.
• Universal accreditation and certification, and the reform of coroner systems toward medical examiners.

This was not a fringe wish list. It was the considered judgment of the National Academy of Sciences and the President's own science advisors, specific and actionable.

What actually happened — the honest accounting

Here is where the case study earns its place in the book's reform chapter, because the gap between what was demanded and what was done is the whole lesson. The grading is qualitative and honest:

The reforms that ask the field to measure and report itself advanced — partially. Accreditation spread and became mandatory in some states. Some important validation research was funded and conducted, most notably black-box studies of latent-print examination that, for the first time, measured the discipline's false-positive rate (and demolished the "zero error rate" claim in the process). Several disciplines published consensus documents curtailing overstated language, and the science of cognitive bias became mainstream within the field's own scholarship. The FBI's admission of decades of flawed microscopic-hair-comparison testimony (Chapter 19) was a genuine institutional reckoning. OSAC (§38.4) was created to lead the standards movement. These are real gains, and honesty requires crediting them.

The reforms that ask the field to restructure power did not. And this is the heart of it:

• Independence: largely not done. A decade-plus after NAS named it the deepest structural fix, only a handful of jurisdictions have independent or partially independent labs. Most remain embedded in law enforcement. The single most important recommendation is the least adopted.
• A strong national authority: not created as proposed. No "National Institute of Forensic Science" came into being. Instead, weaker, advisory bodies emerged — and the most significant of them, the National Commission on Forensic Science (a Justice Department–convened advisory body that did serious work on standards and ethics), was allowed to lapse in 2017 when its charter was not renewed. The strong, independent authority NAS asked for was never built, and even the weak advisory version was let go.
• Active judicial gatekeeping: largely not done. Many courts declined to exclude long-accepted methods even after PCAST gave them the scientific basis to do so. And critically, the Department of Justice formally disagreed with PCAST, declining to adopt its recommendation that courts require error-rate testimony or exclude unvalidated methods — the nation's chief prosecuting authority setting itself against the scientific community's verdict on the evidence it uses to convict.

What the reports did — and didn't — establish

The reports established, beyond serious scientific dispute, what good forensic science requires. That intellectual victory is permanent and enormous: there is now an authoritative, public, citable standard against which any method, lab, or testimony can be measured, and a defense attorney armed with NAS and PCAST can force a real Daubert fight over testimony that for decades sailed in unchallenged.

What the reports did not do — could not do, by themselves — was compel their own implementation. A report is an argument, not a law. It can change what counts as informed opinion; it cannot, on its own, restructure a single laboratory or exclude a single piece of evidence. The reports handed the field, the courts, and the legislatures a finished blueprint. Whether to build from it was, and remains, a question of will — and the pattern of what got built and what didn't reveals exactly where the will was present and where it was absent.

The lesson

The lesson is the chapter's thesis, and the book's: the crisis in forensic science is no longer primarily a scientific problem, because the science of what to fix and how is largely settled. It is an ethical and structural problem — a problem of will. The reforms that advanced were the ones the field could adopt without surrendering its alignment with the prosecution: better standards, more research, more honest language, voluntary improvements to technique. The reforms that stalled were the ones that required restructuring power — taking labs out of police departments, building an authority that could compel change, and empowering judges to exclude beloved but unvalidated methods. The field proved far more willing to improve its technique than to change who its scientists work for.

That is exactly the pattern you would predict if the obstacle were not ignorance but interest — and recognizing it is the difference between despairing that "forensic science is broken" and understanding, precisely, which fixes are missing and why. It is also, in the end, a map for the reformer: the unfinished business is named, and the naming is the first step toward demanding it be finished.

Discussion questions

1. The NAS report (2009) and the PCAST report (2016) had different scopes. State each report's central question, and explain how PCAST's foundational-validity / validity-as-applied distinction sharpened the NAS report's broader finding (Chapter 6).

2. The chapter argues a clear pattern in which reforms were adopted and which were not. State the pattern, and explain why it supports the conclusion that the obstacle is "interest, not ignorance." What evidence in this case study most strongly supports that reading?

3. The National Commission on Forensic Science was allowed to lapse in 2017. Connect this to the NAS report's call for a strong national authority. What does the difference between "an advisory commission that can be dissolved" and "an institute with real power" tell you about how seriously the deepest recommendation was taken?

4. The Department of Justice formally disagreed with PCAST. Using the concept of institutional conflict of interest (§38.5), explain why this response was predictable — and why it is not adequately answered by pointing out that most analysts are honest.

5. A prosecutor argues: "These reports are just academic opinion; our methods have decades of successful casework behind them." Using Chapter 6's distinction between casework and validation (why casework cannot measure its own error rate), explain why this is not a sufficient answer to the reports.

6. The reports "handed the field a finished blueprint" but could not compel implementation. If you were designing the one additional mechanism most likely to convert the blueprint into reality, would you choose a structural fix (mandated independence), a legal fix (stricter judicial gatekeeping), or a market fix (defense-driven Daubert challenges)? Defend your choice, and name what it would and would not accomplish.