Case Study 2 — What the Trial Registry Made Visible

Publication bias, outcome switching, and the reform that let anyone check

Type: Real, public, ongoing · Tier 1 institutional facts, Tier 2 magnitudes · Relevance: §5.5, §5.10

Background: a literature you cannot audit

Suppose ten trials are run on a drug. Four show benefit; six show nothing.

If all ten are published, a reader — or a meta-analyst — sees a mixed and unimpressive picture and concludes correctly that the drug's effect is uncertain and probably small.

If only the four positive ones are published, the same reader sees four consistent positive trials and concludes the drug works. Nobody lied. No individual paper is fraudulent. Every published result is real. And the literature as a whole is a systematic misrepresentation of the evidence.

This is publication bias, and for most of the twentieth century there was no way to detect it, because there was no record of which trials had been run. A trial that produced a disappointing result and was never written up simply did not exist as far as any reader could tell. You cannot notice an absence you have no way of counting.

A related problem operated within individual papers. A trial measures many things. If the pre-specified primary endpoint disappoints but a secondary measure looks good, a paper can be written around the secondary measure — presenting as the finding something that was never the question. This is outcome switching, and it was likewise undetectable, because the original protocol was not public.

The reform

The response was structural rather than exhortative: make trials register before they begin.

ClinicalTrials.gov, run by the U.S. National Library of Medicine, became the largest such registry. Registration requires stating, in advance and publicly, what the trial will measure, in whom, and what the primary endpoint is. The World Health Organization maintains an international registry platform linking national registries.

Two developments gave registration force:

Journal policy. The International Committee of Medical Journal Editors (ICMJE) adopted a policy that member journals would not publish trials that had not been prospectively registered. This changed registration from a virtue to a precondition for publication.

Regulation. Legal requirements in the United States and the European Union require registration and results reporting for certain categories of trial.

The effect was not to eliminate either problem. Compliance is incomplete, enforcement is uneven, and results are frequently reported late or not at all. The effect was to make both problems visible and countable — which is the precondition for doing anything about them.

🔬 Read the Study — the registry as an instrument

text FIGURE 5.CS2 — "Counting what isn't there" [real institutional reform] THE STUDY Not a study — an infrastructure change. Prospective trial registration (ClinicalTrials.gov and the WHO registry platform), enforced by ICMJE journal policy and by law in some jurisdictions. THE QUESTION How can a reader detect trials that were run and not published, and endpoints that were changed after the fact? WHAT IT SHOWS Registration creates a public record of intent, against which the published record can be compared. Studies auditing registries against publications have repeatedly found substantial non-publication and meaningful rates of discrepancy between registered and reported endpoints. WHAT IT DOESN'T It does not compel publication, guarantee compliance, cover trials that were never registered, or apply meaningfully to ANIMAL studies — which remain largely unregistered. It also cannot make an unpublished trial's data available; it only reveals that the trial existed. THE VERDICT One of the most consequential methodological reforms in modern medicine, and incompletely implemented. THE LESSON The most important information about a literature is often what is MISSING from it — and missingness is invisible until somebody builds the instrument that counts it.

Why this matters for peptides specifically

Three consequences, and the third is the one that decides Part III.

You can check a trial's registration yourself, free, in about two minutes. Search ClinicalTrials.gov for a compound. Read the registered primary endpoint. Compare it to what the paper reported. This is a genuinely available check that almost nobody performs, and when a discrepancy exists it is usually decisive.

You can find trials that never reported. A registry entry that has been sitting at "completed" for years with no posted results and no publication is informative. It does not prove the result was negative — trials go unpublished for many reasons including funding collapse and staff turnover — but non-publication is not randomly distributed with respect to outcome, and a compound whose completed trials never publish is telling you something.

And the animal literature has no equivalent. This is the crucial asymmetry.

Human trials increasingly leave a trace whether or not they publish. Animal experiments do not. There is no comparable registry, no publication requirement, and no way to count what was run and shelved.

So when a compound is supported by "over a hundred animal studies," you are reading a filtered sample of unknown size. The filter selects for positive results, and its strength is unknown and unknowable. That is not an argument that the animal data is fabricated — it is almost certainly not. It is an argument that the apparent consistency of an animal literature is not evidence of consistency, because you cannot see the denominator.

This applies directly to the compounds in Part III, whose supporting evidence is overwhelmingly animal work.

⚠️ Hype Check — "over a hundred studies"

"There are over a hundred published studies on this peptide."

What's true: there very likely are. This is usually a checkable and accurate statement, and people who make it are not generally lying.

What it does not tell you:

  • How many studies were run. The published count is a numerator with no denominator.
  • What rung they sit on. A hundred animal and cell studies is a hundred studies at the bottom of the ladder. One adequately powered human RCT outranks all of them for the human question.
  • Whether they are independent. Many may come from the same group, using the same model, with the same assumptions. Ten papers from one laboratory is closer to one finding than to ten.
  • What they measured. Studies of gut protection, angiogenesis, and neuroprotection in rodents do not accumulate into evidence about human tendon healing. They are different claims.
  • Whether any of them tested the claim being made.

The reframe that works: a study count is a measure of research activity, not of evidential weight. Those are different quantities, and only one of them tells you whether something works. Ask instead: how many completed randomized controlled human trials? For a great many popular peptides the answer is zero, and that single number is more informative than the hundred.

What this case teaches

Absence is evidence, and it is the hardest evidence to see. Everything in this book about "what does NOT exist" — the dossier's Field 5 line, the ❌ ratings, the standing question about missing trials — descends from the insight that made registries necessary.

Methodological reform is possible and it is partial. Registration genuinely improved the human trial literature. It did not fix it, and it barely touched the preclinical literature. Progress in research integrity is real, incremental, and does not arrive evenly.

And you can do this check yourself. Most of the tools in Chapter 5 require you to evaluate what is in front of you. This one lets you find out what is not in front of you, free, in a browser. It is the highest-yield two minutes available to a non-specialist, and Chapter 39 recommends bringing the result of it to a clinical conversation.

Discussion questions

  1. Explain publication bias to someone who has never heard of it, using the ten-trial example. Then explain why no individual paper needs to be fraudulent for the literature to be misleading.

  2. Outcome switching involves reporting a secondary measure as though it were the question. Why is this more serious than it sounds — what does pre-specification actually protect against?

  3. Trial registration was made effective by journal policy and law rather than by asking researchers to behave better. What does that suggest about how to fix a systemic problem in a field?

  4. Animal studies remain largely unregistered. Design a system that would address this. What are the obstacles — practical, financial, and cultural?

  5. A completed trial has sat on a registry for four years with no posted results and no publication. What can you legitimately conclude? What can you not? How would your answer change if there were six such trials?

  6. Apply it. Pick one compound from your dossier and search ClinicalTrials.gov for it. Record what you find: how many trials, what phases, what status, what endpoints, whether results are posted. Then write one paragraph on whether the registry picture matches the impression you had from other sources.