Case Study 37.2 — Auditing a Drift

A learner's dossier, set beside the master table

What this is

Below is a completed drift audit — the Step 2 comparison from §37.5's dossier section, filled in for twelve claims. It is a [constructed teaching example]: no real person's dossier, assembled to show a pattern clearly enough to be read. Real audits are messier, and yours will be.

The exercise is not to grade the learner. It is to practice the move §37.10 asks for, which almost nobody makes unprompted: stop counting how many were right and start reading which direction the wrong ones ran.

Work through the table before reading the analysis.

The audit

Ratings marked "book" are reproduced from the master table (§37.5). Where the book gives a split, the learner's claim is compared against the arm that matches the version they rated — noted in the reason column.

Counts: 4 MATCH · 6 HIGHER · 2 LOWER.

Reading the numbers

A learner who stops at the first number concludes: four out of twelve, I did badly. That is the least informative thing available in this table, and it is also not obviously true — three of the disagreements are defensible, as we will see.

Apply §37.10 Step 3 instead. Six HIGHER against two LOWER, across twelve comparisons, is a ratio of three to one. Past the rough threshold of two to one over ten or more comparisons, that is a signal rather than noise. This learner has generous drift.

Now ask the diagnostic question: what do the HIGHER entries have in common?

Entries 2, 4, 8, and 10 are all Part III compounds — the growth-hormone-axis and repair compounds, sold direct to consumers, the densest ❌ block in the book. Entry 3 is a cosmetic ingredient. Every one of them is a compound a person can buy. That is the fingerprint of a stake: the drift concentrates precisely where the learner has something invested, financially or otherwise, in the answer.

Read the reason column and the mechanism becomes explicit:

• Entry 4: "it demonstrably raises GH and IGF-1." True, and it is a surrogate. Raising a hormone is not improving body composition, and improving body composition is not improving function. The book's ⚠️ rests on randomized human data showing increased fat-free mass; the learner's ✅ rests on a hormonal measurement one step further back.
• Entry 10: "both compounds definitely raise growth hormone." The same move, one rung lower. This is rule 3 of §37.2 in its purest form — never upgrade a rating with mechanism — and it is the reason the book's split on this row exists at all: the surrogate claim is supportable and the outcome claim is not.
• Entry 8: "it's been used in horses for years." This is a claim form, and it is rated in the master table: "this compound has been used in animals for years, therefore it is safe in humans" is ❌ (Chapter 31). The learner reached for an inference the book had already evaluated.
• Entry 2: "so many people report it worked." Volume of anecdote, which is not a tier-moving input under any rule in §37.2.
• Entry 7: "the early trials looked promising." Correct, and incomplete — the row is a ⚠️/❌ downgrade, and the later, larger studies are the reason. The learner stopped reading at the moment the evidence was most encouraging, which is precisely the phase 2 optimism trap of Chapter 9.

Now the two LOWER entries, which are the more interesting half.

• Entry 5: botulinum toxin rated down because it is heavily marketed. This is rule 4 — never downgrade a rating with distaste — and the evidence the learner did not consult is multiple adequately powered randomized placebo-controlled trials with blinded assessment, consistent and large effects, and decades of post-approval use.
• Entry 12: Semax rated down because of its country of origin. The book's ⚠️ is not an endorsement; it exists because a real supporting literature is difficult for most readers of this book to obtain and assess. The learner's conclusion is roughly in the right neighborhood and their reason is not — "can't trust it" is a prior about a source, not an assessment of evidence.

So this learner has both biases running at once. The dominant one is generous, on compounds they have a stake in. The minor one is severe, on compounds whose presentation they dislike. Neither is respectable and both have the same underlying shape: a judgment about a source standing in for an examination of evidence.

Which disagreements are actually fine

Step 4 of the dossier exercise asks for two piles, and this audit fills both.

Pile one — defensible. Entry 11 is a MATCH, but notice the reason: the learner reasoned from molecular weight and the location of the neuromuscular junction, which is a Chapter 1 argument arriving at a Chapter 30 conclusion without needing the trial data. That is exactly the skill the book is for. Entry 12 is a LOWER that lands close to a defensible position — a reader who said "⚠️ overstates what I can personally verify about this literature, and I would need translated primary sources to move" would have written a correct disagreement.

Pile two — carrying the bias. Entries 2, 3, 4, 8, and 10. Every reason in that pile is either a mechanism, a surrogate, an anecdote count, or an already-rated claim form. None names evidence about the claimed endpoint in the stated population.

The size of pile two relative to pile one is the real score, and it is five to two. The MATCH count of four never mattered.

The Step 5 sentence

Here is what this learner should write in their dossier, dated:

Six of my twelve ratings ran higher than the book's, five of those six were compounds I had bought or was considering, and in four cases my stated reason was a mechanism or a hormone level rather than an outcome in the population the claim named.

That sentence is worth more than the audit that produced it. It is also falsifiable: run the audit again in a year on a different set of claims and find out whether it still holds.

Questions

1. Recount the audit yourself. Do you agree with every DIRECTION label? Identify at least one where you would argue for a different label, and say why. (Hint: look hard at entries 3, 7, and 10, where a split rating is involved.)

2. The analysis claims the MATCH count "never mattered." Argue against that. Under what circumstances would a low match count be the most important number in the audit?

3. Four of the six HIGHER entries give a reason that is a surrogate or a mechanism. Rewrite each of those four reasons as it would have to read to move the rating legitimately — that is, name the evidence that would justify the learner's tier.

4. Entry 8's reason is a claim form the book already rates ❌. Find two more reasons in the table that are also rated claim forms — either in §37.6 or in the master table — and give their chapter numbers.

5. This learner has generous drift on compounds they bought and severe drift on compounds they find distasteful. Construct the mirror-image learner: twelve entries producing a clear severe drift overall. What kinds of compounds would their LOWER entries cluster on, and what would their reason column look like?

6. † Entry 12 is scored LOWER, but the analysis concedes the learner "lands close to a defensible position" with an indefensible reason. Is a right answer for a wrong reason an error in a drift audit? Argue both ways, then say how you would score it in your own audit — and what that choice reveals about what you think the audit is measuring.