Appendix F: The Validated vs. Questioned Methods Scorecard

This is the single most important reference in the book. It is the validity spectrum (Chapter 1, sharpened in Chapter 6) rendered as a working scorecard — method by method, what the science can honestly support, what goes wrong, and where the two landmark reviews placed it. Every discipline chapter located its own method on this spectrum; this appendix gathers all of them in one place so you can answer, fast, the question that protects an innocent person: Where does this method sit, what claim is being made, and does the science support that claim?

Read three framing rules before any verdict below, because they matter more than memorizing positions.

1. Position is a ceiling, not a guarantee. A method's place on the spectrum is the best reliability it can offer when done perfectly. Top-of-spectrum DNA can be dragged to a worthless result by a contaminated tube or a biased setup. The spectrum tells you the method's potential; validity as applied tells you what happened this time.
2. The spectrum is question-specific. The same discipline can sit at opposite ends depending on what you ask it. Forensic odontology is near the discredited end for "these teeth made this bite mark" and near the solid end for "these dental records identify this body." Always ask about the specific claim, never the discipline's name.
3. The honest verb tracks the position. As foundation weakens, the strongest defensible statement weakens in lockstep: identifies (with a stated probability) → consistent with (at a measured error rate) → class-level association only → the science does not support this comparison at all. An examiner whose confidence rises as the science weakens has inverted the relationship — and that inversion is, almost exactly, the shape of the wrongful convictions in this book.

What "NAS 2009" and "PCAST 2016" mean in the columns below. The NAS 2009 report (Strengthening Forensic Science in the United States) surveyed the whole field and found that, apart from nuclear DNA, the disciplines lacked the rigorous foundation their courtroom authority implied. The PCAST 2016 report zoomed in on feature-comparison methods and asked, of each, whether black-box studies (many examiners, many comparisons of known ground truth, error rate measured) had shown it actually works — distinguishing foundational validity (is the method valid?) from validity as applied (was it done right here?). A method PCAST did not address (e.g., toxicology, digital, accounting) is marked "not a feature-comparison method; outside PCAST's 2016 scope."

On the error-rate figures. Where a number appears it is illustrative of the right order of magnitude, drawn from the qualitative findings the chapters discuss — not a precise rate. Exact rates vary by study and conditions. Treat them as "roughly this big," never as a citable statistic.

F.1 The spectrum at a glance

   STRONGER FOUNDATION  ◄─────────────────────────────────────────────►  WEAKER / DISCREDITED
   (validated, error rate known)                        (no validated basis, or affirmatively debunked)

   Single-source DNA  >  Instrumental    >  Latent        >  Simple DNA    >  Firearms/      >  Microscopic   >  Bite
   Toxicology (conf.)    chemistry          fingerprints     mixtures (PG)    toolmarks,        hair "match",     marks
   (GC-MS confirmed)     (drugs, debris)    (valid; non-     (valid in        complex DNA       handwriting       (discredited;
                                            zero error)      envelope)        mixtures, BPA,    ID, footwear/     exonerations)
                                                                              GSR, soil,        tire "to one"
                                                                              shoe/tire class

   ── what you can honestly say shifts as you move right ──►
   "identifies, with    "is X, to        "consistent with;  "X× more        "cannot exclude;   "cannot exclude;  "the science
    a stated RMP"        analytical       false-match rate    probable if     class-level        of limited        does not
                         certainty"        ~1 in N"           contributor"    association"        value"            support this"

Digital forensics, forensic accounting, forensic pathology, anthropology, entomology, odontology-as-ID, and questioned-document non-handwriting work do not slot neatly onto a single left-right line because they are not "compare-a-feature-to-a-source" methods; their scorecards below state their own honest reach.

F.2 The master scorecard

Each row: the method, its foundational validity, its principal error modes, its NAS 2009 / PCAST 2016 status, and what it can honestly establish (the verb). Detailed notes for the contested methods follow in F.3–F.6.

Biological evidence

Pattern and impression evidence

Chemical and toxicological evidence

Digital and other domains

F.3 The strong end, read honestly

The methods at the left of the spectrum earned their place by being built outside the courtroom — by scientists with no stake in any verdict — and validated, quantified, and stress-tested before they were asked to convict anyone. That sequence is the whole difference. Three honest cautions keep "strong" from becoming "infallible":

• Single-source DNA is the anchor, but its strength is for its actual claim — identifying the source of cells. Ask it "whose cells are these?" and it is the finest tool forensic science has. Ask it "is this person guilty?" and it is silent — and the most dangerous courtroom moments are when someone mistakes that silence for a yes. The transfer problem, relatives, and ordinary lab error are real edges on even this number (Appendix E).
• Instrumental chemistry and confirmed toxicology are strong because an instrument reads a molecule against a standard, not an examiner's eye against a memory. But "present" is not "impairing," a presumptive test is not a confirmation, and the most common failure is skipping confirmation — the roadside field-test problem that has jailed innocent people on a color change.
• Confirmed identification ≠ correct inference. GC-MS can prove gasoline is in the debris; that does not prove arson. The valid path to an incendiary finding runs through valid origin analysis and confirmed ignitable liquid — never through the debunked indicators that helped execute Cameron Todd Willingham.

F.4 The contested middle: where most courtroom danger lives

The middle of the spectrum is the most consequential, because these methods are admitted every day and their limits are the easiest to overstate. The pattern is consistent: a class-level statement is defensible; an individual-source statement is not.

• Latent fingerprints are the cautionary heart of the middle. PCAST found them foundationally valid — and in the same breath reported a measured, non-zero false-match rate wholly incompatible with the "zero error rate" examiners claimed for a century. The Brandon Mayfield case (Chapter 14) is the proof: the FBI was 100% confident and 100% wrong, driven partly by context and confirmation bias. The honest verb is "agreement supporting a common source, with a stated error rate," never "identification to the exclusion of all others."
• Firearms and toolmarks rest on a subjective "sufficient agreement" threshold that PCAST found inadequately validated by black-box studies. The defensible claim is class-level ("consistent with; cannot exclude"); the leap to "this gun and no other" outran the science.
• Footwear, tire, paint, glass, soil, and fibers are all honest class/associative evidence whose value rises with distinctiveness or rarity and collapses when a common feature is sold as significant. The two genuine exceptions worth knowing: a physical fracture fit (glass, or any broken object that uniquely re-assembles) can support a single-source association, and a rare fiber or distinctive wear pattern strengthens an association without ever reaching individualization.
• GSR and BPA sit lower in the middle for the same reason: the underlying observation can be sound (a primer-residue particle is real; a single droplet's physics is real) while the inference drawn from it (this person fired a gun; the attack happened here, this way, in this order) is contaminated by transfer, examiner subjectivity, or overinterpretation. The FBI discontinued routine GSR analysis in 2006; BPA was flagged by NAS as resting more on examiner experience than on science.

🧠 The bias multiplier. Everything in the middle is human judgment, and human judgment drifts toward the expected answer. When the examiner knows the suspect's profile, the detective's theory, or the "right" result, borderline calls bend that way — not from dishonesty but from ordinary confirmation bias (Chapter 31). This is why context management and sequential unmasking matter most precisely for the methods in this section, and why "experience" is not a substitute for a measured error rate.

F.5 The discredited end

These methods make individual-source claims their science cannot support. The tell, every time, is certainty that rises as the foundation falls.

• Bite-mark comparison is the field's most thoroughly debunked discipline. It needs two unproven premises — that human dentition is unique and that skin reliably records it — and fails both; examiners disagree even on whether a mark is a bite. Multiple exonerations (Ray Krone, Roy Brown, and others) were built on bite-mark "matches." PCAST found it not foundationally valid and saw little prospect that it could be. The honest verb is, in practice, nothing as identification.
• Microscopic hair comparison as a "match" method is discredited for individualization. The FBI's own review found examiners had overstated the strength of hair testimony in the large majority of reviewed cases — not by lying about what they saw under the scope, but by letting a jury hear "microscopically consistent" as "his hair." The valid path for hair-to-source is mtDNA, with its lineage limits stated.
• Handwriting individualization, the old fire "indicators," and profiling-as-prediction belong here too for their strongest claims: each was built on practitioner assertion rather than validation, and each has a documented failure (Willingham for the fire folklore; Richard Jewell for the profile). The instrumental or genuinely-careful parts of these fields (ink chemistry, indented writing, competency assessment) are valid; the individualizing or predictive claims are not.

⚠️ The diagnostic inversion. The DNA analyst — strongest method — speaks in careful probabilities and refuses "identifies" without a number. The bite-mark examiner — weakest — says "this defendant and no other, to a reasonable degree of dental certainty." When the confidence of the testimony runs opposite to the strength of the method, that inversion is itself a warning, and it is almost exactly the shape of the wrongful convictions this book studies.

F.6 How to use this scorecard

When you meet a forensic claim — in a chapter, a courtroom, a news story, or a case file — run it through five questions, in order:

1. What is the specific claim? Not "fingerprint evidence" but "the examiner says this print is the defendant's to the exclusion of all others." The discipline is not valid or invalid; a claim is supported or not.
2. Where does the method sit for that claim? Use F.2. Remember the spectrum is question-specific — odontology for body ID vs. for bite marks lands in two different columns.
3. Is the verb honest for that position? "Identifies (with a number)" only at the strong end; "consistent with / cannot exclude" in the middle; "class-level association" for class evidence; "the science does not support this" at the discredited end. A verb stronger than the position is overstatement.
4. What is the error rate — and was it measured? "We've never been wrong" is not an error rate. If the black-box studies were never done, that absence is itself a finding (and a Daubert problem — Appendix D).
5. Was the method applied validly here? Position is a ceiling. Check contamination, chain of custody, bias exposure, and whether the sample fell within the method's validated envelope. A top-of-spectrum method, misapplied, can produce a bottom-of-spectrum result.

The reflex this appendix exists to build. By the end of the book you should be unable to hear a forensic claim without the spectrum lighting up: Where does this sit? What claim is being made? Does the science support that claim, and was it done right this time? That reflex — not any single position in any table — is what protects an innocent person.

Cross-references: Chapter 1 (class vs. individual characteristics; the validity spectrum introduced); Chapter 6 (NAS 2009, PCAST 2016, foundational validity vs. validity as applied — the spectrum's backbone); Chapters 7–9 and Appendix E (DNA and its statistics, the strong anchor); Chapters 10, 14–24 (the discipline-by-discipline verdicts summarized here); Chapter 30 and Appendix D (how each verdict becomes honest — or dishonest — testimony); Chapter 31 (the cognitive bias that drags the middle of the spectrum downward); Chapter 38 (the reforms that would make these verdicts the courtroom norm).