Case Study 40.2: The Book Audits Itself

The situation

vqelab.claims encodes the eight questions of §40.4. A checklist that only ever indicts other people's work is not a checklist — it is a rhetorical device. So this case study runs it on this book's own headline results, and reports what fails.

Three fail. One of them is the result Chapter 36 is named for.

Audit 1: Chapter 36's central measurement

The claim, as the chapter states it: VQE reaches the (2e,2o) active space's exact answer to $2.04\times10^{-9}$ Ha, while the active space itself is wrong by $2.01\times10^{-2}$ Ha — a ratio of 9,870,104.

>>> from vqelab.claims import Claim, audit_claim
>>> result = audit_claim(Claim(
...     statement="VQE reached 2.04e-09 Ha inside a space wrong by 2.01e-02 Ha.",
...     baseline="CCSD(T)", n_samples=None,
...     denominator="Hartree, against exact diagonalization",
...     statistic="absolute error", accounts_for_total_error=True,
...     produces=("an energy",), measurement_can_detect=True,
...     cost_shots=18_456_984))
>>> result["verdict"], list(result["failures"])
('UNSUPPORTED', ['how_many_samples'])

Seven of eight. It fails on sample size, and it should: the ratio was measured on one molecule at one geometry — LiH at 3.0140 Bohr, STO-3G.

Is the finding wrong? Almost certainly not. Active-space truncation error exceeding optimizer error is not a delicate effect; it follows from the optimizer converging on an exactly-solvable 4-qubit problem while the truncation discards real correlation energy. The direction is robust.

But "almost certainly robust" is a different claim from "measured", and the specific number — 9,870,104 — is a property of that one geometry. Chapter 36 Exercise 36.17 asks the reader to vary the bond length from 1.5 to 6.0 Bohr precisely because the chapter did not, and the honest expectation is that the ratio moves by orders of magnitude across that range.

What the chapter should have said: "on this molecule and geometry, seven orders of magnitude" rather than letting a single number carry a general claim.

Audit 2: Chapter 31's dynamical decoupling result

Chapter 31 measured dynamical decoupling making things significantly worse: XX at $-0.0053 \pm 0.0012$.

>>> from vqelab.claims import significant
>>> significant(-0.0053, 0.0012)
True

This one passes, and it is worth seeing why it is the strongest measurement in the book.

The claim is negative — it says a standard technique did not help — so there was no incentive to find it. It carries its uncertainty. The effect is 4.4 standard errors. And the direction is surprising, which means it got checked rather than waved through.

★ The measurements that survive scrutiny best are the ones nobody wanted. Chapter 34's kernel concentration prediction failed and the chapter reported both halves; Chapter 37's $p=4$ regression evaporated at eight seeds. In both cases the unwanted result was the reliable one, because a surprising number gets audited and a confirming number does not.

Audit 3: Chapter 39's queue arithmetic

§39.3's utilization figures — 2.31 × 10⁻⁵ at a five-minute queue, 43,340× wall clock — are the basis of the chapter's central recommendation.

>>> audit_claim(Claim(
...     statement="At a 5-minute queue, utilization is 2.31e-05.",
...     baseline="local simulation", n_samples=None,
...     denominator="fraction of job lifetime", statistic="ratio",
...     accounts_for_total_error=False,
...     produces=("a utilization figure",),
...     measurement_can_detect=False, cost_shots=4096))["failures"].keys()
dict_keys(['how_many_samples', 'total_or_method_error',
           'can_the_measurement_see_it'])

Three failures, and the third is the serious one. measurement_can_detect=False because the queue time was never measured — this environment has no provider credentials, and the chapter says so in a callout at the top.

The circuit durations are real calibration data. The queue values are representative. The ratio combines a measurement with an assumption, and a reader who skims past the callout will take the whole line as measured.

What the chapter did right: declared the limitation prominently, labelled the queue values where they appear, and kept the two kinds of number visually separate. What it could not do: make the conclusion as solid as a measured one. §39.3's claim is structurally certain — 7 milliseconds against any plausible queue gives a tiny ratio — and numerically assumed.

What fails, and what that means

Two of the book's four headline results fail its own checklist, on questions the book itself introduced.

That is the correct outcome, and a checklist that never indicted its author would be worthless. The value is not in the pass/fail — it is that the failures are specific. "Chapter 36 needs more geometries" and "Chapter 39's queue is assumed, not measured" are actionable; "this book is approximately trustworthy" is not.

🔬 Honest Assessment: audit_claim deliberately returns no score.

Chapter 36 passing seven of eight and the vendor claim in Example 2 passing one of eight are not points on a scale. Chapter 36 has a specific, nameable, fixable gap. The vendor claim cannot be evaluated at all, because without a real baseline there is nothing to evaluate against.

A single number averaging those would be the last instance of the error this book spent forty chapters on: the easy number is not the number that answers the question.

Questions

1. Run the audit on Chapter 37's ten-instance comparison. Which checks pass, and is the verdict right?
2. Chapter 36's finding is called "almost certainly robust." Design the experiment that would move it to "measured," and estimate what it costs.
3. Chapter 39 declared its queue limitation in a callout. Is that sufficient? Propose a stronger convention for separating measured from assumed numbers in technical writing.
4. [measure] Pick any three claims from Parts I–IV and audit them. Report the failures without softening them.
5. The hard one. This case study audits four claims the author selected. That is itself a sampling problem — the same one Chapters 27, 28, 33, 34, 37, 38, and 39 committed. Design a selection procedure that would not be vulnerable to it, then say honestly whether applying it would change the conclusion.