Chapter 27: Testing Quantum Programs

Chapter 26 ended with an anecdote: a debugging tool that reported no divergence for a circuit whose process fidelity against its reference was 0.9498, because its default input could not see the defect. The measured version of that anecdote was 41% of structured test inputs blind, 0 of 100 random ones.

This chapter turns that into policy.

Testing quantum programs is harder than testing classical ones for a reason that has nothing to do with quantum mechanics being strange: you usually have no oracle. For most circuits worth writing there is no independent source of the correct answer — if there were, you would not need the quantum computer. Chapter 23's Shor is the rare exception, and it is an exception precisely because multiplication is easy.

So the question is not "does it produce the right answer?" but "what can I assert about a program whose right answer I do not know?" This chapter has four answers, ranked by how much they cost and how much they catch — and it measures both.

The headline result is §27.3's scoreboard: three of the four oracle-free properties tested here pass a circuit that is definitely broken.

27.1 What is a unit test for a circuit?

A classical unit test asserts a function's output on a chosen input. The quantum translation has four distinct forms, and they are not interchangeable:

   1. UNITARY EQUALITY      Operator(mine) == Operator(reference)
                            Exact. Input-independent. Needs a reference and 4^n memory.

   2. STATE ASSERTION       Statevector.from_instruction(qc) matches an expected state
                            Exact. Depends entirely on the input you chose.

   3. PROPERTY              a relation that must hold, with no reference at all
                            Exact. Only as strong as the property.

   4. DISTRIBUTION          measured counts match an expected distribution
                            STATISTICAL. The only one that works on hardware.

Only the fourth survives contact with a real device, and it is the weakest and most expensive. That tension organizes the whole chapter.

27.2 The tier list, and what each tier costs

Testing advice is worthless without cost numbers, so here they are — measured, per assertion, for a 3-qubit QFT:

   assertion                              cost      assertions per CI-minute
   unitary equality (exact)             0.34 ms                     174,006
   statevector, 1 fixed input           0.21 ms                     279,433
   statevector, 20 random inputs        5.75 ms                      10,437
   transpile to FakeManilaV2 + verify  12.76 ms                       4,701
   simulated counts, 1,000 shots       85.43 ms                         702
   simulated counts, 100,000 shots    631.25 ms                          95

A shot-based test costs roughly 250–3,600× an exact one. That ratio is the single most important number for designing a suite: it means you can afford tens of thousands of exact assertions and a few hundred sampled ones, so every check that can be made exact should be.

📌 These are wall-clock timings on one machine, and they move by tens of percent between runs — re-running the example gave 0.33 ms, 0.17 ms, 5.39 ms, 15.43 ms, 82.17 ms, 603.89 ms for the same six rows. Read the ratios, not the absolute values, and measure your own before setting a CI budget.

But exact tests do not scale in the same direction:

    qubits   Operator equality    statevector x20
         3              0.4 ms             5.2 ms
         5              1.1 ms            10.9 ms
         7              5.9 ms            17.9 ms
         9            149.6 ms            30.6 ms
        11          3,488.0 ms            53.4 ms
        12         16,526.8 ms            74.8 ms

Operator equality is $\mathcal{O}(4^n)$ and dies first — by 12 qubits a single assertion takes 16 seconds. Statevector tests are $\mathcal{O}(2^n)$ and are still running twenty random inputs in 75 ms at the same size. The crossover is around 9 qubits.

(This table is timed with fewer repetitions than the one above, so its absolute numbers run a little high — the 3-qubit row reads 0.4–0.9 ms across runs against 0.33 ms above. The shape is the point.)

⚠️ The strongest test is not the best test at every size. Unitary equality is input-independent and cannot be blind, which makes it the right default — but past about 9 qubits it stops being affordable and you must fall back to random statevectors, which trade certainty for reach.

Pick the tier from the size you have to test, then make the tier as strong as it can be.

27.2.1 Reading the scaling table honestly

The two columns do not merely have different exponents. They are in different regimes, and the step-to-step ratios say so:

   step               measured   4^n predicts   n^2 4^n predicts
    3 ->  5 qubits       2.75x            16x             44.44x
    5 ->  7 qubits       5.36x            16x             31.36x
    7 ->  9 qubits      25.36x            16x             26.45x
    9 -> 11 qubits      23.32x            16x             23.90x
   11 -> 12 qubits       4.74x             4x              4.76x

Below about 7 qubits the timing is not measuring an exponent at all. A 3-to-5-qubit step should cost 16× and costs 2.75×; the matrix is small enough that Python call overhead, circuit construction, and object allocation dominate, and the linear algebra is an afterthought. Any conclusion drawn from the top two rows is a conclusion about interpreter overhead.

Above 7 qubits the growth is consistently faster than $4^n$ — 25.4× and 23.3× where the exponent predicts 16×. That is not noise, and it has a cause. Operator(qc) does not allocate a $4^n$ matrix; it applies every gate to one. An $n$-qubit QFT contains $n(n+1)/2$ Hadamards and controlled-phase gates plus $\lfloor n/2 \rfloor$ swaps, so the work is $\Theta(g(n) \cdot 4^n)$ with $g(n)$ growing quadratically. Substituting the exact gate count gives predicted ratios of 25.29×, 23.18×, and 4.73× against measured 25.36×, 23.32×, and 4.74× — agreement within 0.6%.

So the honest complexity of the unitary-equality tier is $\mathcal{O}(n^2 4^n)$ for this circuit family, and $\mathcal{O}(4^n)$ is the right shorthand because the $4^n$ is what kills you. The refinement only matters when you extrapolate. A plain $4^n$ extrapolation from the 12-qubit row predicts about 4.4 minutes per assertion at 14 qubits; the $n^2 4^n$ form predicts about 6.0. Both are predictions, not measurements — but if you are budgeting CI time, the second is the one to budget against.

The statevector column tells the opposite story. Its ratios are 2.10×, 1.64×, 1.71×, 1.75×, 1.40× against $2^n$'s predicted 4× per two-qubit step. The exponential has not arrived by 12 qubits. Twenty random inputs at 12 qubits take 74.8 ms, or 3.74 ms each — still dominated by twenty separate random_statevector constructions and twenty separate .evolve calls. That is why this tier keeps working long after the other one stops.

One correction to the crossover figure while we are in the arithmetic. The measured Operator-to-statevector cost ratio is 0.33 at 7 qubits and 4.89 at 9. Interpolating log-linearly puts the crossing at $n \approx 7.8$. Nine qubits is the first measured row where the exact-unitary test is the more expensive of the two; the actual crossing is nearer eight.

💰 Cost and Queue: price the suite before you design it.

Two hundred distribution tests is a realistic suite size. From §27.2's per-assertion costs:

text 200 tests @ 1,000 shots 17.09 s per commit 200 tests @ 100,000 shots 126.25 s per commit

On a simulator that is a nuisance. On hardware it is a bill. Two chapters measured the effective shot rate of a cloud QPU independently and agree: Chapter 37's 13,000,000 shots in 22 minutes is 590,909 shots/min, and Chapter 33's 27.8 QPU hours per million 1,000-shot predictions is 599,520 shots/min — a 1.5% discrepancy between two unrelated experiments.

Take 595,000 shots/min and Chapter 39's measured \$50 per QPU-minute. Two hundred tests at 1,000 shots is 200,000 shots, \$16.9 per commit**. At 10,000 shots it is **\$169 per commit, before queueing. Chapter 39 measured utilization at 2.31e-05 behind a five-minute queue — 43,340× wall clock — so the wall-clock cost of that suite is not the QPU cost at all.

This is the arithmetic behind §27.9's rule that hardware runs are not CI. It is not a matter of taste. A distribution suite on hardware costs roughly a developer-day of money per developer-day of commits, and returns a signal dominated by recalibration.

⚙️ Under the Transpiler: what makes a distribution test slow is not the shots.

code/example-03-two-error-rates.py carries a comment recording its own first version's failure: transpiling inside the measurement loop made 2,000 repetitions take over ten minutes instead of seconds. Caching one transpiled circuit per distinct circuit fixed it.

That is worth generalizing. Of the 85.43 ms in the 1,000-shot row, a large fraction is transpile(), not sampling — the 100,000-shot row is only 7.4× the 1,000-shot row despite having 100× the shots, which is exactly what a large fixed per-call cost looks like. Sampling itself is cheap; compilation is not.

A distribution test therefore has two costs with different scalings: a fixed compile cost per distinct circuit, and a marginal cost per shot. Hoisting the transpile out of the loop is not a micro-optimization — it is the difference between a nightly job and an every-commit one, and it makes the shot count the parameter you are actually free to raise in §27.6.

⚠️ Common Pitfall: == is not the comparison you want for a unitary.

A global phase is physically undetectable — no measurement of any kind can distinguish $U$ from $e^{i\theta} U$. Qiskit's operators do not agree on that. Taking a Bell-preparing circuit and adding global_phase += pi/4 to an otherwise identical copy:

text Operator(a) == Operator(b) False np.allclose(a.data, b.data) False Operator(a).equiv(Operator(b)) True process_fidelity(a, b) 1.000000 max |P_a - P_b| over all inputs 2.22e-16

The first two assertions fail on two circuits that are the same circuit. Optimization levels, gate decompositions, and basis translations all introduce global phases routinely, so this is not a corner case — it is the most common false failure in an exact test suite.

Assert with .equiv() or process_fidelity(...) == pytest.approx(1.0). Reserve == for the rare case where the phase is genuinely load-bearing, which is when the circuit is controlled — a global phase on a controlled subroutine becomes a relative phase, and then it matters.

27.3 Testing without an oracle

Now the hard case. You have a QFT implementation and no reference. What can you assert?

The natural answers are metamorphic properties — relations that must hold between outputs, requiring no independent correct answer. Here are four, applied to a QFT with a deliberately wrong rotation angle ($\pi/2 \to \pi/3$ on one controlled-phase gate):

Property 1 — the adjoint round trip. $U U^\dagger = I$ is the most-cited quantum test property.

   good:  U U^dag == I ?  True
   bad:   U U^dag == I ?  True

It passes the broken circuit, and it always will. qc.inverse() inverts your circuit, so the inverse contains the same wrong angle and cancels it exactly. A self-inverse property cannot detect an error that its own inverse reproduces.

Property 2 — unitarity. $U^\dagger U = I$ on the built matrix.

   good:  ||U^dag U - I|| = 1.47e-15
   bad:   ||U^dag U - I|| = 1.72e-15

Also passes. Every circuit built from gates is unitary by construction; this checks the framework, not your circuit.

Property 3 — a known answer. $\text{QFT}|0\dots0\rangle$ must be the uniform superposition, which is true and checkable without any reference.

   good:  max|p - 1/8| = 5.55e-17   PASS
   bad:   max|p - 1/8| = 5.55e-17   PASS

Passes too — and by now the reason should be familiar. Every control is in $|0\rangle$, so the wrong angle never fires. This is Chapter 26 §26.4's blind spot appearing as a test rather than as a debugging session.

Property 4 — the shift theorem. $\text{QFT}|x+1\rangle$ equals $\text{QFT}|x\rangle$ with each amplitude $k$ multiplied by $e^{2\pi i k/N}$. This is a genuine structural property of the transform, it needs no reference implementation, and it relates different inputs to each other.

   good:  worst deviation = 0.0000   PASS
   bad:   worst deviation = 0.1830   FAIL  <-- CAUGHT IT

The scoreboard:

   property                        needs a reference?   catches the bug?
   U U^dag == I                                   no                 NO
   unitarity                                      no                 NO
   QFT|000> uniform                               no                 NO
   shift theorem                                  no                YES
   random inputs vs QFTGate                      YES                YES

⚛️ A property is only useful if the bug can violate it.

"It is self-inverse" and "it is unitary" constrain almost nothing — an enormous space of wrong circuits satisfies both. The shift theorem constrains the relationship between different inputs, which is exactly where a wrong rotation angle lives.

The test to prefer is the one that would break if the code were wrong in the way you are worried about. Writing down the failure mode first, then the property, is the only reliable order.

Three of four oracle-free properties pass. That is not an argument against metamorphic testing — the one that works, works, and it needs no reference and runs in microseconds. It is an argument against assuming a property suite is a test suite because it is green.

27.3.1 Two more that work, and why

The shift theorem is not the only discriminating property available here, and it is worth seeing what its successful siblings have in common. The discrete Fourier transform satisfies two exact self-composition identities that need no reference implementation at all:

$$F^2 = R, \qquad F^4 = I$$

where $R$ is the reversal permutation $|x\rangle \mapsto |(-x) \bmod N\rangle$. The first follows in one line: $(F^2)_{jx} = \sum_k F_{jk}F_{kx} = \frac{1}{N}\sum_k \omega^{k(j+x)} = \delta_{j+x \equiv 0}$, because the geometric sum vanishes unless $j + x \equiv 0 \pmod N$. The second is the first applied twice, since $R^2 = I$.

Applied to the same good and bad 3-qubit QFTs:

   good:  max|U^2 - R| = 0.0000   PASS
   bad:   max|U^2 - R| = 0.5045   FAIL  <-- CAUGHT IT

   good:  max|U^4 - I| = 0.0000   PASS
   bad:   max|U^4 - I| = 0.7624   FAIL  <-- CAUGHT IT

Both catch the bug, and neither needs a reference. That takes the scoreboard to four of six oracle-free properties useful rather than one of four — which changes the tone of §27.3's headline without changing its finding. The finding was never "metamorphic testing is weak." It was "three specific, popular, cheap properties are worthless, and greenness does not tell you which kind you have."

The interesting question is what separates $U^4 = I$ from $UU^\dagger = I$. Both are identities of the form "some composition of the circuit with itself is the identity." Both are oracle-free. One is blind by construction and one caught the bug on the first try.

The difference is inversion. $UU^\dagger = I$ holds for every unitary matrix — it is the definition of unitarity, restated — so it constrains nothing whatever within the space of circuits you might have written. Worse, qc.inverse() is computed from your circuit, so any parameter error is reproduced with the opposite sign and cancels exactly. $U^4 = I$ uses only forward applications. No inverse is constructed, nothing cancels, and the identity is a genuine constraint: it forces every eigenvalue of $U$ onto a fourth root of unity, which the overwhelming majority of unitaries do not satisfy.

⚠️ Common Pitfall: any property whose statement contains qc.inverse() is suspect.

The pattern generalizes past the QFT. assert U @ U.inverse() == I is the first test everyone writes for a new circuit, and it is a test of the framework's inverse() method. It cannot fail for a parameter error, a wrong angle, a swapped control and target on a symmetric gate, or a missing global phase, because every one of those defects is faithfully mirrored into the inverse.

It can catch a genuinely non-unitary construction — a circuit with mid-circuit measurement or reset in it, or a hand-built matrix that is not unitary. If that is the failure mode you are worried about, keep it. If it is not, the property is decoration.

Prefer identities that use the circuit forwards. $U^k = P$ for some known permutation $P$, $U|x\rangle$ against $U|y\rangle$, $U$ against $U$ with a symmetry applied to the input — all of these route around the cancellation.

📐 Math Aside: how much does each property actually constrain?

"A property is only useful if the bug can violate it" can be made quantitative. Count the dimensions a property leaves free.

A 3-qubit circuit is an element of $U(8)$: 64 real parameters, or 63 once global phase is quotiented out. Now take the four properties in turn.

Unitarity removes zero. Every circuit assembled from gates is in $U(8)$ already, so the property is a tautology on the space being searched.

$UU^\dagger = I$ removes zero, for the same reason plus the cancellation argument above.

$\text{QFT}|000\rangle$ uniform constrains the magnitudes of one column: eight equations $|U_{k0}|^2 = 1/8$, of which seven are independent given normalization. 63 − 7 = 56 free parameters remain. The property eliminates 11% of the space.

The shift theorem relates every adjacent pair of columns: $\text{col}_{x+1} = D\,\text{col}_x$ with $D = \mathrm{diag}(\omega^k)$, $\omega = e^{2\pi i/8}$. Seven such relations determine columns 1 through 7 entirely from column 0. Only the eight complex entries of column 0 remain, and unitarity forces them to have equal magnitude $1/\sqrt{8}$.

Now combine the last two. Uniform column 0 plus the shift theorem forces $U_{kx} = \frac{1}{\sqrt 8}e^{i\phi_k}\omega^{kx}$ — that is, $U = \mathrm{diag}(e^{i\phi_k}) \cdot F$. Seven free real parameters out of 63, and they are all left-diagonal phases. Constructing such a $U$ with random phases and checking:

text shift-theorem deviation 1.24e-16 |col 0|^2 uniform deviation 2.78e-17 ||U^dag U - I|| 1.04e-14 max |U - F| 0.6532 max | |U|^2 - |F|^2 | (output probabilities) 5.55e-17

It is not the QFT — the matrices differ by 0.65 — and no computational-basis measurement of any input can tell it apart from the QFT, because a left diagonal phase multiplies amplitudes by unit modulus and probabilities are unchanged.

★ So the two properties that individually catch nothing and everything are, together, a complete characterization of the QFT's measurable behaviour. Property 3 is not worthless. It is worthless alone, and load-bearing in combination — which is a different and more useful statement than the scoreboard alone conveys.

The caveat is the usual one about global phases becoming relative phases. A left diagonal phase is invisible if you measure straight after the QFT, and entirely visible if the QFT is a subroutine with more non-diagonal gates downstream, which in Chapter 22's phase estimation it is. Whether a residual freedom matters depends on what runs after it.

27.3.2 A recipe for finding discriminating properties

The scoreboard is a warning, not a method. Here is the method, which is the one the chapter's project module encodes in check_property's discriminating field.

1. Write down the failure mode first. Not "what is true of my circuit" but "what will I get wrong?" For a hand-built QFT the answer is obvious in hindsight and invisible in advance: a rotation angle, because there are $n(n+1)/2$ of them and they all look alike. For an oracle it is an ancilla left dirty. For a transpiled circuit it is a qubit permutation. For a variational ansatz it is a parameter bound to the wrong wire.

2. Ask what relation that specific error breaks. A wrong controlled-phase angle changes how the transform responds to a change of input, so pick a property relating different inputs. A dirty ancilla breaks separability of the ancilla register from the data, so assert the reduced state on the ancillas is $|0\rangle\langle 0|$. A permutation breaks agreement with Operator.from_circuit, which §27.8 covers.

3. Verify the property is discriminating by deliberately breaking the circuit. This is the step everybody skips, and it costs one line: perturb a parameter, re-run the property, assert it now fails. check_property(name, predicate, circuit, perturbed=broken) does exactly this, and returns discriminating=False for three of §27.3's four.

A property suite with no perturbation test is a suite of claims about what your circuit is, with no evidence that any of them could have come out otherwise. Mutation testing is the classical name for step 3, and it transfers to quantum code essentially unchanged — with the pleasant difference that "mutate the circuit" here means "add $\varepsilon$ to one angle," which is a one-line, continuously tunable mutation operator that classical mutation testing would envy.

🔀 In Another Framework: the same four tiers, different names.

Cirq 1.7.0 exposes the exact tiers directly. cirq.unitary(circuit) builds the matrix; cirq.testing.assert_allclose_up_to_global_phase(u, v, atol=1e-8) is the assertion that avoids the == trap above; cirq.testing.random_superposition(dim) and cirq.testing.random_unitary(dim) generate the random inputs §27.4 insists on. cirq.testing.assert_implements_consistent_protocols has no Qiskit analogue — it is a property-check on a gate class, verifying that its decomposition, its unitary, and its __pow__ all agree.

PennyLane 0.45.1 gives qml.matrix(qnode)() for the unitary, qml.math.fidelity for state comparisons, and qml.assert_equal / qml.equal for structural comparison of operators and tapes — note that structural equality is a stronger and different assertion than unitary equality, and will reject two circuits that implement the same operator by different decompositions.

The design worth stealing is Q#'s. Its standard library has long distinguished AssertOperationsEqualInPlace, which compares on computational basis states, from AssertOperationsEqualReferenced, which compares by preparing an entangled ancilla register and checking the resulting Choi state. That distinction is exactly §27.4's measurement: the first is the assertion Chapter 26 measured as 4/8 blind, and the second is input-independent by construction and therefore cannot be blind at all. Qiskit's process_fidelity(Operator(a), Operator(b)) is the same idea reached from the other direction.

27.4 Property-based testing: use random inputs

When you do have a reference — a library gate, a previous version, a slow-but-obvious implementation — the question becomes which inputs to compare on. Chapter 26 answered it with a measurement:

   computational basis states blind:   4/8    ['000','010','100','110']
   states over {0, 1, +} blind:       11/27   (41%)
   random states blind:               0/100

So the pattern is the classical property-based testing pattern, transplanted:

@pytest.mark.parametrize("seed", range(20))
def test_matches_reference_on_random_inputs(seed):
    v = random_statevector(2 ** N, seed=seed)
    f = state_fidelity(v.evolve(mine), v.evolve(reference))
    assert f == pytest.approx(1.0, abs=1e-9)

Three details matter, and all three are load-bearing:

Seed the inputs, parametrized. range(20) with a seed per case gives twenty independent tests that are individually reproducible. A single test looping over twenty random states reports one failure and hides which input caused it.

Use more than one. Zero of one hundred random states were blind, but that is a statistical statement about one bug, not a guarantee. Twenty is cheap — 5.75 ms.

Never use $|0\dots0\rangle$ as your only input. It is the most structured state available and the first one anybody types.

27.5 Testing a distribution

Everything so far is exact and simulator-only. The moment you sample — which is the only thing hardware does — testing becomes statistics.

Here is the shot noise on a perfectly correct GHZ(3) circuit, measured as total variation distance from the exact distribution, over 40 runs at each shot count:

       shots    mean TVD       std   max over 40 runs   1/sqrt(N)
         100     0.04375   0.03199            0.12000     0.10000
       1,000     0.01313   0.01003            0.03700     0.03162
      10,000     0.00423   0.00283            0.01160     0.01000
     100,000     0.00121   0.00101            0.00401     0.00316

Every one of those numbers is the error of a correct circuit. A tolerance tighter than the max column will fail a correct circuit some fraction of the time. The scaling is the familiar $1/\sqrt{N}$ from Chapter 24 §24.3 — the same wall, arriving in a different discipline.

So a distribution test has two parameters, and choosing them is a real decision:

   shots   tolerance    false failures      rate    95% upper bound
   1,000        0.05           3/2,000    0.150%             0.320%
   1,000        0.10           0/2,000    0.000%             0.150%
  10,000        0.05           0/2,000    0.000%             0.150%
  10,000        0.02           0/2,000    0.000%             0.150%

In a suite of 200 such tests running on every commit, a 0.15% per-test rate means roughly one red build in every four or five commits, forever, on correct code. That is not catastrophic, and it is more than enough to make someone file a ticket — which is where Case Study 2 begins.

⚠️ Measuring a test's error rate is itself a sampling problem, with the same $1/\sqrt N$ wall.

The first draft of this section reported 1.0% (2/200) for the top row. That was one small sample of a rate near 0.15%: at 200 runs the standard error on such a rate is about 0.7% — larger than the rate — and observing 0, 1, 2, or 3 failures are all ordinary outcomes.

Re-running with 2,000 and then 3,000 runs gave 0.150% and 0.100%. Use the rule of three for a zero count: 0 failures in $N$ runs bounds the rate at roughly $3/N$, not at zero.

Two chapters ago this was Chapter 24 §24.3's correction — reporting a mean over 25 repetitions instead of one seeded draw. It applies to your test suite's own statistics too.

There is a second lesson buried in that correction. The 2/200 measurement used a GHZ circuit built with h(0); the 3/2,000 measurement used ry(pi/2, 0). Same state, same exact distribution, different transpiled circuit — so the simulator consumes randomness differently and the same seed gives different draws. Seeding pins a run, not a result. §27.7 returns to this.

📐 Math Aside: where the shot-noise floor comes from — the mean TVD is $0.39894\,S/\sqrt N$, and the floor is three times it.

The table above is measured. Its shape is derivable, and the derivation is short enough to be worth having, because it tells you the one thing the table cannot: how the floor changes for a distribution that is not GHZ(3)'s.

Sampling $N$ shots from a distribution $p$ gives counts $X_i \sim \text{Binomial}(N, p_i)$ and empirical frequencies $\hat p_i = X_i/N$ with standard deviation $\sigma_i = \sqrt{p_i(1-p_i)/N}$. For a roughly normal deviate, $\mathbb{E}|\hat p_i - p_i| = \sigma_i\sqrt{2/\pi}$. Total variation distance is half the summed absolute deviation, so

$$\mathbb{E}[\mathrm{TVD}] \;=\; \tfrac12 \sqrt{\tfrac{2}{\pi}} \sum_i \sqrt{\tfrac{p_i(1-p_i)}{N}} > \;=\; \frac{1}{\sqrt{2\pi N}} \sum_i \sqrt{p_i(1-p_i)} \;=\; \frac{0.39894\, S}{\sqrt N}$$

writing $S = \sum_i \sqrt{p_i(1-p_i)}$ for the shape factor of the distribution. The $1/\sqrt N$ is universal; the constant in front is not.

A noiseless GHZ(3) puts probability $1/2$ on two outcomes and zero on six, so $S = 2\sqrt{0.25} = 1$ exactly, and $\mathbb{E}[\mathrm{TVD}] = 0.39894/\sqrt N$. Against the chapter's measurements:

text shots predicted measured pred. sd meas. sd SE of mean diff in SE 100 0.03989 0.04375 0.03014 0.03199 0.00477 0.81 1,000 0.01262 0.01313 0.00953 0.01003 0.00151 0.34 10,000 0.00399 0.00423 0.00301 0.00283 0.00048 0.50 100,000 0.00126 0.00121 0.00095 0.00101 0.00015 -0.34

The predicted standard deviation is the half-normal's, $\sigma\sqrt{1-2/\pi} = 0.30139/\sqrt N$. Every measured mean is within 0.9 standard errors of the derived value, and every measured standard deviation within 7% — and the residual is itself a 40-run sampling artefact, which is this book's own recurring lesson pointed at its own table. A 20,000-repetition multinomial check of the formula reproduces it to within 0.3% for GHZ(3), for a uniform distribution over 8 outcomes, and for a uniform distribution over 64.

Now the useful part. For a distribution spread uniformly over $M$ outcomes, $S = M\sqrt{\tfrac1M(1-\tfrac1M)} = \sqrt{M-1}$:

text M S E[TVD] 2 1.0000 0.3989/sqrt(N) GHZ, Bell, any two-outcome result 8 2.6458 1.0555/sqrt(N) a full 3-qubit uniform output 64 7.9373 3.1665/sqrt(N) 1024 31.9844 12.7599/sqrt(N)

★ The floor grows as $\sqrt{M/N}$, so a wider output distribution costs shots at the same rate a tighter tolerance does. That consequence is developed in §27.6.2, and it is the reason the shot-noise floor is not a single number you can memorize.

27.5.1 The floor is a property of the distribution, not just the shot count

The project module in code/vqelab/testing.py encodes the floor as a constant:

def shot_noise_floor(shots, safety=3.0):
    return safety / np.sqrt(shots)

Its docstring justifies the safety factor from the table above — "the MAX TVD over 40 runs was about 3× the mean at every shot count measured," which the measurements bear out at 2.74, 2.82, 2.74, and 3.31. That is sound. But 3.0 is a factor applied to a mean the function never computes, and the mean depends on $S$.

⚠️ Common Pitfall: this chapter contains two different threes, and they are not the same rule.

text shot-noise floor ~ 3/sqrt(N) a TOLERANCE on TVD; 3 x the mean TVD 0.0949 at N = 1,000 shots rule of three ~ 3/N an upper BOUND ON A RATE from zero events 0.0015 at N = 2,000 runs

They share a constant and nothing else. The first is a safety factor covering the tail of a sampling distribution, and $N$ counts shots. The second is the standard zero-event confidence bound — §27.5's "0 failures in $N$ runs bounds the rate at roughly $3/N$" — and $N$ counts runs of the whole test. One is $\propto N^{-1/2}$; the other is $\propto N^{-1}$.

Mixing them produces a number that is wrong by orders of magnitude and looks plausible. At the scales in this chapter they differ by 63×. The two appear about two hundred lines apart in §27.5, which is close enough to be worth the warning.

Run the derivation forwards. The max-over-40 criterion is roughly $3\,\mathbb{E}[\mathrm{TVD}] = 1.197\,S/\sqrt N$. For GHZ(3), $S=1$, so the honest floor is $1.197/\sqrt N$ — and the measured max column, multiplied by $\sqrt N$, reads 1.200, 1.170, 1.160, 1.268. The derivation lands on the measurement to three significant figures.

For a uniform 3-qubit output, $S = \sqrt 7$ and the floor is $3.167/\sqrt N$.

★ So shot_noise_floor's constant is correct — for a distribution spread evenly across a full 3-qubit register, and for nothing else. Setting $1.197\,S = 3$ gives $S = 2.507$, or $M \approx 7.3$ outcomes. Below that the function is conservative; above it, dangerously optimistic. At 10 qubits with a spread output the true floor is $38.28/\sqrt N$ and the function returns $3/\sqrt N$ — low by a factor of 12.8.

🐛 Debug This: the project module refuses the configuration the chapter recommends.

§27.6 concludes that 10,000 shots at tolerance 0.02 is the right test, and Case Study 2 presents it as the fix that works. Both are backed by a measurement: 0 false failures in 2,000 runs.

Hand that configuration to the chapter's own helper and it is rejected:

```text

shot_noise_floor(10_000) 0.03 DistributionTest(target, shots=10_000, tolerance=0.02).below_noise_floor True assert_distribution(qc, target, shots=10_000, tolerance=0.02) ValueError: tolerance 0.0200 is below the shot-noise floor 0.0300 at 10,000 shots -- this test would fail on CORRECT code. Raise the shot count to at least 22,500, or loosen the tolerance and accept the loss of detection. ```

"This test would fail on CORRECT code" is a prediction, and the chapter measured it false. The configuration failed on correct code zero times in two thousand runs.

The guard is not wrong so much as distribution-blind. GHZ(3)'s real floor is $1.197/\sqrt N = 0.0120$ at 10,000 shots, and 0.02 clears it comfortably. The 3.0 constant assumes an output spread over ~7 outcomes; GHZ has 2.

The fix is to compute $S$ from the target, which the function already receives in every calling context:

python def shot_noise_floor_from_target(target, shots, safety=3.0): p = np.asarray(target, dtype=float) S = np.sqrt(p * (1.0 - p)).sum() return safety * S / np.sqrt(2.0 * np.pi * shots)

This returns 0.01197 for GHZ(3) at 10,000 shots and 0.03167 for a uniform 3-qubit output, matching both the measured max column and the constant the original was tuned on.

Note what kind of error this was. Nobody guessed. A real measurement — max ≈ 3× mean — was generalized into a constant by dropping the term it was proportional to. That is the same move as Chapter 24 §24.3's single seeded draw and §27.5's 2/200: a correct observation on one case, promoted to a rule without checking what it depended on.

⚛️ The Physics Underneath: the floor is not an engineering limitation.

A classical unit test reads the function's return value. It gets the whole answer, exactly, once.

A quantum circuit's answer is an amplitude vector, and the Born rule does not let you read it. A projective measurement returns one bitstring, sampled with probability $|\langle x|\psi\rangle|^2$, and destroys the superposition doing it. Every subsequent shot requires rebuilding the state from scratch.

So the distribution is never observed — it is estimated, from independent draws, and the estimator has variance. The $1/\sqrt N$ in the floor is the central limit theorem applied to that estimator, and no better simulator, faster gate, or cleverer compiler removes it. It is the same wall behind Chapter 24 §24.3's shot budget, Chapter 36's 1.91e20 shots for a 50-orbital crossover, and Chapter 37's 13,000,000 shots for $\epsilon = 0.01$ — three chapters, three disciplines, one exponent.

There is one genuine escape and it is not free: measure something other than the full distribution. Chapter 35's classical shadows are ~2.5× less accurate per observable but win 1.5–1.8× at equal total budget because they amortize across many observables. Every remedy is denominated in the currency of the disease — you trade a factor on the constant, never on the $\sqrt N$.

27.6 The flaky test and the blind test are the same knob

The obvious fix for a flaky test is to loosen the tolerance until it stops flaking. Here is what that buys, measured against a real bug — a rotation error of size $\varepsilon$ in the GHZ preparation:

   bug size eps   true TVD   detected @1k/tol.10   detected @10k/tol.02
           0.00     0.0000                    0%                     0%
           0.02     0.0100                    0%                     1%
           0.05     0.0250                    0%                    86%
           0.10     0.0499                    0%                   100%
           0.20     0.0993                   52%                   100%
           0.40     0.1947                  100%                   100%

Read the two configurations side by side.

1,000 shots with tolerance 0.10 has a 0% false-failure rate. It never flakes. It is also completely blind to a bug of size $\varepsilon = 0.10$ — a real defect with a true TVD of 0.0499, caught 0% of the time. It does not detect anything until the bug is four times larger, and even then only half the time.

10,000 shots with tolerance 0.02 also has a 0% false-failure rate, and catches $\varepsilon=0.05$ 86% of the time and $\varepsilon = 0.10$ always.

🔬 Honest Assessment: a test has TWO error rates, and tuning one blindly destroys the other.

Loosening the tolerance until the flakiness stops does not fix the test. It converts a test that occasionally failed on correct code into a test that never fails on anything — and a test that cannot fail is not evidence, which is the fourth time this book has arrived at that sentence.

The correct fix for flakiness is more shots, not more tolerance. Shots cost CPU time. Tolerance costs detection, silently.

And this is why §27.5's cost table matters. Going from 1,000 to 10,000 shots costs about 7× the runtime — 702 assertions per CI-minute down to roughly 100 — and buys the difference between a test that sees nothing and a test that sees a 5% error 86% of the time. That is the trade, stated honestly, and it is usually worth paying.

📌 Quote a distribution test's tolerance, its shot count, AND the smallest bug it can detect. The first two without the third describe the cost of a test without describing what it does.

📐 Math Aside: the two error rates are Type I and Type II error, and both are computable.

A distribution test is a hypothesis test that has not noticed it is one. Naming the parts makes both rates fall out in closed form, and the closed forms reproduce every number in this section's two tables.

$H_0$: the circuit is correct. $H_1$: it is not. Test statistic: $\mathrm{TVD}(\hat p, p_0)$. Reject $H_0$ when the statistic exceeds $\tau$.

text P(reject | H_0 true) = alpha = Type I = FALSE FAILURE = flakiness P(accept | H_1 true) = beta = Type II = MISSED BUG = blindness 1 - beta = power = detection rate

Flakiness and blindness are not two independent knobs. They are $\alpha$ and $\beta$, and $\tau$ slides you along the curve between them. That is the whole of §27.6, in the vocabulary statistics settled on a century ago.

Type I in closed form. A correct GHZ(3) produces only 000 and 111, so $\hat p_{000} + \hat p_{111} = 1$ exactly and $\mathrm{TVD} = |X/N - \tfrac12|$ with $X \sim \text{Binomial}(N, \tfrac12)$. Then $\sigma(X/N) = 1/(2\sqrt N)$ and

$$\alpha \;=\; P\!\left(\left|\tfrac{X}{N} - \tfrac12\right| \ge \tau\right) \;=\; 2\,\Phi\!\left(-2\tau\sqrt N\right)$$

text shots tol 2*tau*sqrt(N) predicted alpha measured (Sec 27.5) 1,000 0.05 3.1623 0.1565% 3/2,000 = 0.150% 1,000 0.10 6.3246 0.0000% 0/2,000 = 0.000% 10,000 0.05 10.0000 0.0000% 0/2,000 = 0.000% 10,000 0.02 4.0000 0.0063% 0/2,000 = 0.000%

★★ The formula predicts 3.131 false failures in 2,000 runs for the top row. The chapter measured three. The bottom row predicts 0.127 expected failures in 2,000 runs — so observing zero is not evidence the rate is zero, which is exactly what the rule of three is for.

Type II in closed form. A rotation error $\varepsilon$ in the state preparation gives $p_{000} = \cos^2\!\big(\tfrac{\pi/2+\varepsilon}{2}\big)$, and its true TVD from the target is $\delta = |p_{000} - \tfrac12|$. The statistic is now $|X/N - \tfrac12|$ with $X \sim \text{Binomial}(N, p_{000})$, so with $\sigma = \sqrt{p_{000}(1-p_{000})/N}$,

$$\text{power} \;=\; \Phi\!\left(\frac{\delta - \tau}{\sigma}\right) + \left[1 - \Phi\!\left(\frac{\delta + \tau}{\sigma}\right)\right] \;\approx\; \Phi\!\left(2\sqrt N\,(\delta - \tau)\right)$$

text eps p000 delta pred @1k/.10 meas pred @10k/.02 meas 0.00 0.50000 0.0000 0.0% 0% 0.0% 0% 0.02 0.49000 0.0100 0.0% 0% 2.3% 1% 0.05 0.47501 0.0250 0.0% 0% 84.1% 86% 0.10 0.45008 0.0499 0.1% 0% 100.0% 100% 0.20 0.40067 0.0993 48.3% 52% 100.0% 100% 0.40 0.30529 0.1947 100.0% 100% 100.0% 100%

Six rows, two configurations, twelve predictions, and the largest disagreement is 4 percentage points on a rate measured over 100 runs — where the binomial standard error is itself about 4 points. The $\delta$ column is the chapter's own measured "true TVD" column, reproduced from $\cos^2$.

★★ And the design rule falls straight out: power is 50% when $\delta = \tau$. Solving numerically confirms it exactly — at 1,000 shots and $\tau = 0.10$ the 50% crossing is at $\delta = 0.1000$; at 10,000 shots and $\tau = 0.02$ it is at $\delta = 0.0200$.

The smallest bug a distribution test catches half the time is the one whose true TVD equals the tolerance. No sweep required. The tolerance is the detection threshold, which is why quoting one without the other was never defensible.

27.6.1 Choosing shots and tolerance instead of inheriting them

Two inequalities now determine the whole design, and neither involves taste.

Constraint 1 — do not flake. Pick a false-failure budget $\alpha^*$. Inverting $\alpha = 2\Phi(-2\tau\sqrt N)$ gives

$$\tau \;\ge\; \frac{z_{\alpha^*/2}}{2\sqrt N} \;=\; \frac{1.645}{\sqrt N} \quad \text{for } \alpha^* = 0.1\%$$

Constraint 2 — detect the bug you care about. For power $1-\beta$ against a bug of true TVD $\delta$:

$$\delta \;\ge\; \tau + \frac{z_{1-\beta}}{2\sqrt N}$$

Taking $\tau$ at its minimum and substituting gives the smallest bug a non-flaky test can see:

$$\delta_{\min} \;=\; \frac{z_{\alpha^*/2} + z_{1-\beta}}{2\sqrt N} \;=\; \frac{2.066}{\sqrt N} \quad (\alpha^* = 0.1\%,\ 80\%\text{ power})$$

       shots   tau_min (a=0.1%)   delta_min 80%   delta_min 90%
       1,000             0.0520          0.0653          0.0723
       4,000             0.0260          0.0327          0.0361
      10,000             0.0165          0.0207          0.0229
      40,000             0.0082          0.0103          0.0114
     100,000             0.0052          0.0065          0.0072
   1,000,000             0.0016          0.0021          0.0023

★ To halve the smallest bug your suite can see, quadruple the shots. The table's 1,000-to-4,000 and 10,000-to-40,000 steps each halve $\delta_{\min}$ exactly. It is the same $1/\sqrt N$ that sets the noise floor, because it is the noise floor — detection threshold and flakiness floor are the same quantity with different constants in front.

Check the rule against the chapter's own configuration. At 10,000 shots with $\tau = 0.02$, constraint 2 puts the 80%-power threshold at $\delta = 0.0242$. The measured bug at $\varepsilon = 0.05$ has $\delta = 0.0250$ — just above it — and was caught 86% of the time. The rule works.

Now run it against the other configuration, and something uncomfortable appears. At 1,000 shots the minimum non-flaky tolerance is 0.052, and Case Study 2's team chose 0.10 — nearly twice as loose as they had to be. Here is what the tighter tolerance would have bought at the same shot count:

       eps     delta   power @tau=0.10   power @tau=0.052   chapter measured @0.10
      0.05    0.0250              0.0%               4.3%                       0%
      0.10    0.0499              0.1%              44.7%                       0%
      0.20    0.0993             48.3%              99.9%                      52%

At a false-failure rate of 0.100% — two expected red builds per 2,000 runs, essentially the rate they started with and called intolerable — the $\varepsilon = 0.20$ bug goes from 52% detection to 99.9%, and the $\varepsilon = 0.10$ bug from never detected to detected almost half the time. These are predictions from the derivation above, not measurements; but the derivation reproduced every measured cell in the two tables it was checked against.

📊 What the Numbers Say: "we tightened it and it got flaky" is not the same claim as "it flakes too often to be useful."

The team's actual position was a 0.15% failure rate, which is one red build per four or five commits. The alternative they never evaluated was 0.10% — one per six or seven — with detection of a real 20% error rising from a coin flip to a near-certainty.

They were not trading flakiness against detection. They were 2× away from the efficient frontier and moved in the wrong direction along it. Loosening $\tau$ from 0.05 to 0.10 bought them 0.15 percentage points of $\alpha$ and cost them essentially all of their power.

The reason this is easy to do is that $\alpha$ is measured constantly, by the build system, for free, whether you want it or not — while $\beta$ is measured never, by nobody, unless someone writes a perturbation sweep. The easy number is almost always the flattering one, because it stops the search.

27.6.2 The exponential hiding inside a distribution test

§27.5.1's shape factor has a consequence that does not show up on a 3-qubit GHZ state and dominates everything at scale.

For an output distribution spread across $M = 2^n$ outcomes, $S = \sqrt{M-1}$, so the noise floor is $0.399\sqrt{M-1}/\sqrt N \approx 0.399\sqrt{2^n/N}$. Holding a fixed TVD tolerance while adding qubits requires shots to grow like $2^n$. Solving for a mean TVD of 0.01:

   qubits            M       shots       QPU-min        wall     cost @ $50/min
        3            8      11,141          0.02        ~1 s               $1
        5           32      49,338          0.08        ~5 s               $4
       10        1,024   1,628,155          2.76       2.8 min           $138
       15       32,768  52,150,300         88.3        1.5 h           $4,413
       20    1,048,576   1.67e9          2,824        47.1 h         $141,211

The shot counts are derived. The times use the hardware shot rate two chapters measured independently — Chapter 37's 13,000,000 shots in 22 minutes and Chapter 33's 27.8 QPU hours per million 1,000-shot predictions, which agree to 1.5% at about 595,000 shots/min — and Chapter 39's measured \$50 per QPU-minute.

★★ A single distribution assertion on a 20-qubit circuit with a spread output is a 47-hour, \$141,000 test. Not a suite. One assertion.

This is where "the only tier that works on hardware" collides with "the only tier that scales." The exact tiers die at 9–12 qubits for memory reasons; the sampled tier survives to any width but its cost dies at roughly the same place, for statistical reasons, and nobody notices because the test still runs — it just stops meaning anything. A 1,000-shot TVD assertion on a 20-qubit spread distribution has a noise floor of 12.9, on a statistic that is bounded above by 1. It cannot fail. It is §27.6's blind test, arrived at by a completely different route.

🔬 Honest Assessment: what this does and does not rule out.

It does not say distribution testing is useless past 10 qubits. It says TVD over the full output distribution is, and TVD was chosen here because it is the natural distance and the easy one to compute.

The escape is to test a statistic whose variance does not grow with $M$. A marginal on three qubits has $M = 8$ however wide the register is. An expectation value $\langle Z_0 Z_1\rangle$ has standard error $\le 1/\sqrt N$ regardless of $n$. A parity, a Hamming-weight histogram, or the probability of one designated bitstring are all $\mathcal{O}(1/\sqrt N)$.

Chapter 30 makes the same choice under a different name — a benchmark is a statistic chosen so that its variance is affordable — and pays the same price: a statistic cheap enough to estimate is a statistic that has thrown away most of the distribution, and the bug may live in what was thrown away. This is "a measurement that cannot detect the thing being asked about" reaching Part V for the sixth time.

Assert on a low-variance statistic and say which one. "The measured distribution matches" is not a claim anyone can afford past ten qubits, and quoting a TVD tolerance at that width is a claim to have done something you did not do.

📉 Noise Report: on hardware, the exact distribution is the wrong target.

Everything in §27.5 and §27.6 compares against Statevector's exact distribution, which is correct on a simulator and wrong on a device. Real hardware adds a bias — decoherence, readout error, gate infidelity — on top of the shot noise, and bias does not shrink with $N$.

That splits the observed TVD in two: $\mathrm{TVD}_{\text{obs}} \approx \delta_{\text{noise}} + \mathcal{O}(S/\sqrt N)$. Past the shot count where the second term drops below the first, adding shots buys nothing — you are estimating the device's bias to ever greater precision. On Chapter 29's hardware-aware run at fidelity 0.9116 the bias term is of order $10^{-1}$, which a few hundred shots already resolve.

So a hardware distribution test's tolerance has to cover the device, and the device moves. Chapter 30 measured one chip's quoted two-qubit error ranging from 0.00750 to 0.07205 — a factor of 9.6 across a single backend — and Chapter 39 found cz error from 1.79e-03 to 1.00 on dead links. A tolerance sized for Monday's calibration is either flaky or blind by Thursday.

The workable pattern is a differential one: compare today's distribution against yesterday's run of the same circuit, not against the exact answer. That tests the thing that can actually regress — your code — and it puts the device's drift on both sides of the comparison, where it mostly cancels. It is also, precisely, §27.9's reason that hardware runs are validation and not testing.

27.7 Seeding makes tests reproducible, not accurate

Seeding is mandatory and it is not a solution.

   UNSEEDED, five runs of an identical circuit:
     {'00':  92, '11': 108}
     {'00': 102, '11':  98}
     {'00':  98, '11': 102}
     {'00':  85, '11': 115}
     {'00': 100, '11': 100}

   SEEDED (seed_simulator=42), five runs:
     {'00': 104, '11':  96}
     {'00': 104, '11':  96}
     {'00': 104, '11':  96}
     {'00': 104, '11':  96}
     {'00': 104, '11':  96}

The seeded run is identical every time — and it is 104/96, not 100/100. Seeding fixes which draw you get from the distribution; it does not make the draw representative.

⚠️ Common Pitfall: a seeded test is one arbitrary sample, reproducibly.

This is precisely Chapter 24 §24.3's methodology correction. A single seeded shot-noise measurement was quoted, the sampling API changed, the numbers moved, and the fix was to report a mean over 25 repetitions rather than one reproducible draw.

Seed so the suite is deterministic. Use enough shots so the number is right. These are two different problems and seeding only solves the first.

There is a related trap. A seeded test that passes tells you nothing about whether it passes for other seeds — so a seeded distribution test should either use enough shots that any seed would pass, or be parametrized over several seeds. Picking the one seed that passes is not testing; it is curve fitting.

27.7.1 The other seed, and why small circuits hide it

seed_simulator is not the only random variable in a quantum test. seed_transpiler is the second, and it is the one nobody parametrizes — including §27.8's example, which pins seed_transpiler=7 and tests one draw.

How much does that draw matter? Two later chapters measured it, and their answers are almost comically far apart:

   Ch.28  optimization levels 2 vs 3    differ in 14 of 40 circuit-seed pairs
   Ch.39  14-qubit layout, 24 seeds     fidelity 0.5755-0.7911 (2.03x in error)
                                        two-qubit gates 49-112
   Ch.39  the SAME test on 4 qubits     EXACTLY ZERO variation

★ At four qubits the transpiler seed is not a random variable at all. At fourteen it is worth a factor of two in error and more than a factor of two in gate count. The layout search has nothing to search over on a small register — every choice is equivalent — so it returns the same circuit every time and the variance is genuinely, exactly zero.

This has a direct consequence for a test suite, and it is not the obvious one. A suite built entirely on 3-to-5-qubit circuits will report a transpiler configuration as deterministic when it is a coin flip at the size you actually deploy. The tests are not wrong. They are being run in the one regime where the thing they would have caught does not exist.

So: pin seed_transpiler for reproducibility, exactly as you pin seed_simulator — and then parametrize over a handful of seeds for at least one circuit at realistic width, asserting on the spread rather than the value. Chapter 29's 0.9116 versus 0.7720 is a comparison between two transpiler configurations; without seed spread you cannot tell whether such a gap is a result or a draw.

🗝️ Version Note: the suite is what notices, so pin the environment.

This chapter's measurements were taken on Qiskit 2.5.1, qiskit-aer 0.17.2, qiskit-ibm-runtime 0.48.0, PennyLane 0.45.1, Cirq 1.7.0. That list belongs in the test suite's lockfile, not in a paragraph, for a reason this book has hit repeatedly:

text Qiskit 2.0 qiskit.pulse REMOVED, with add_calibration, .calibrations, backend.defaults, instruction_schedule_map, drive_channel Qiskit 2.1 mcx(mode="v-chain") removed (Ch. 19) PennyLane shots= on a device deprecated (Ch. 24)

Every one of those was discovered by code that stopped working, which is what a test suite is for. §27.8's argument that transpilation tests "pin behaviour across upgrades" is the same point: the tests you keep are the ones that convert a silent semantic change into a red build.

The corollary is the trap. A green suite on an unpinned environment is a suite whose meaning changes without a commit. If requirements.txt says qiskit>=2.0, then a passing build last Tuesday and a passing build today are not evidence about the same software, and the §27.5 observation that h(0) and ry(pi/2, 0) consume simulator randomness differently — same state, same exact distribution, different draws from the same seed — shows how little has to change for your reproducible numbers to stop reproducing.

27.8 Testing the circuit that actually runs

Chapter 26 §26.7 established that transpilation permutes your qubits and that verifying it requires Operator.from_circuit. As a test, that becomes:

def test_transpilation_preserves_the_unitary():
    t = transpile(qc, FakeManilaV2(), optimization_level=3, seed_transpiler=7)
    padded = QuantumCircuit(t.num_qubits)
    padded.compose(qc, qubits=range(qc.num_qubits), inplace=True)
    assert process_fidelity(Operator.from_circuit(t), Operator(padded)) == pytest.approx(1.0)

At 12.76 ms it is affordable — 4,701 per CI-minute — and it catches a category of failure nothing else in the suite touches. Two things make it worth writing even though the transpiler is well tested:

It catches your own transpiler configuration errors, which are far more common than transpiler bugs: a bad initial_layout, an optimization level that removes something you needed (Chapter 25's id noise slots), a basis gate set that does not include what you assumed.

And it pins behaviour across upgrades. Chapter 24 found shots= on a PennyLane device deprecated; Chapter 19 found mcx(mode="v-chain") removed in Qiskit 2.1. A test suite that only exercises logical circuits will not notice when transpilation changes underneath it.

The limit from Chapter 26 applies unchanged: on a 127-qubit backend the operator cannot be built at all, and the test must be written against a small backend with similar coupling structure — or replaced by a distribution comparison, at 250× the cost.

27.8.1 Testing a circuit you cannot simulate

Every tier in §27.2 needs either a $2^n$ statevector, a $4^n$ operator, or (from §27.6.2) a shot budget growing like $2^n$. Past roughly 30 qubits all three are gone, and the honest answer to "how do I test this?" is that you do not test it. You test five other things, and the composition is an argument rather than an assertion.

1. Test the parts exactly; get the whole by construction. A large circuit is built from subroutines that are individually small. Assert unitary equality on each at 3–8 qubits — where it costs 0.34 ms — and assert separately that the composition wires them together correctly, which is a structural check on the circuit object and needs no simulation at all.

2. Test the family at small $n$, then assert the construction is $n$-independent. Chapter 19's oracle went from 26,978 T gates without ancillas to 55 with — the interesting property is a property of the builder, and a builder verified exactly at $n = 3, 4, 5, 6$ against a reference, plus an assertion that its gate count follows the predicted formula at $n = 40$, is stronger evidence than one heroic simulation at $n = 20$.

3. Assert on structure, not amplitudes. Gate counts, depth, two-qubit-gate count, ancilla cleanliness, basis-set conformance, coupling-map conformance: all exact, all linear in the circuit size, all scale forever. Chapter 28's approximation_degree=0.9 producing zero two-qubit gates is exactly the kind of catastrophic configuration error a gate-count assertion catches instantly and no distribution test ever will, because a circuit that has been optimized into nothing still returns a distribution.

4. Embed instances whose answer you happen to know. The oracle problem is not uniform across inputs. Chapter 22 found phase estimation exact for dyadic phases — pick $\phi = 3/8$ and the output is deterministic, so a statistical algorithm acquires an exact test. Chapter 21's Grover at $N=16$ hits 0.9613 at 3 iterations, and Chapter 23's factoring of 15 gives four outcomes near 25% each. These are known-answer tests inside algorithms that are supposedly oracle-free, and they exist because you got to choose the instance.

5. Differential-test against a second implementation. Two independent constructions agreeing is weaker than a proof and much stronger than one construction passing its own properties. Chapter 18's interoperability path makes this cheap: build it in Qiskit, build it in Cirq or PennyLane, compare unitaries at the largest $n$ both can reach.

What none of these gives you is a test of the deployed circuit at deployed width. That is Chapter 26 §26.8's distinction, and it does not dissolve: at full scale you are validating, and validation is an argument that combines small exact tests, structural invariants, and known-answer instances into a case. Say that it is a case, not a pass.

🧪 Run It: measure the two error rates on something you broke yourself.

Everything in this chapter is reproducible from code/, and the exercises worth doing are the ones that change a number.

1. Move the bug. In example-02-testing-without-an-oracle.py, my_qft injects a wrong angle at j == 1 and k == 0. Move it to j == 2, k == 0 — the smallest-angle rotation — and re-run. Which properties still catch it, and how does the worst shift-theorem deviation change? A smaller angle error is a smaller signal; find where the shift theorem's tol=1e-9 stops being the binding constraint.

2. Add the two forward-composition properties. Write $U^2 = R$ and $U^4 = I$ as predicates and pass them to check_property(..., perturbed=bad). Confirm they come back discriminating=True where prop_adjoint_round_trip comes back False.

3. Find your own 50% crossing. Sweep $\varepsilon$ in example-03-two-error-rates.py between 0.10 and 0.30 at 1,000 shots and $\tau = 0.10$, and locate where detection crosses 50%. §27.6 predicts it lands where the true TVD equals 0.10, i.e. $\varepsilon \approx 0.20$. If your answer disagrees, the measurement wins — report it.

4. Fix the floor. Replace shot_noise_floor with §27.5.1's distribution-aware version and re-run assert_distribution at 10,000 shots and tolerance 0.02. It should stop raising.

5. Test the tighter tolerance. Measure the false-failure rate at 1,000 shots and $\tau = 0.052$ over 2,000 runs. §27.6.1 predicts 0.100%, or about two failures. Two thousand runs cannot distinguish 0.10% from 0.15%, so note that in your write-up rather than reporting a difference.

27.9 What goes in CI

Assembling the cost numbers into a policy:

   ON EVERY COMMIT (seconds)
     * unitary equality against references         174,006/min
     * property assertions (shift theorem, etc.)     exact, microseconds
     * random-input comparisons, ~20 seeds          10,437/min
     * transpilation verification, small backend     4,701/min
     * ancilla-cleanliness assertions (Ch. 26)       exact

   NIGHTLY (minutes)
     * distribution tests at 10,000+ shots              ~100/min
     * larger instances, 8-12 qubits
     * noise-model runs against expected degradation

   MANUALLY, ON HARDWARE (not in CI)
     * anything requiring a real device
     * validation, not testing (Ch. 26 Sec 26.8)

The organizing principle is Chapter 26's, stated as a budget: exact tests are 250–3,000× cheaper than sampled ones, so push every check as far up the list as it will go. A property you can assert exactly should never be asserted statistically.

And a warning about the last group. Hardware runs belong in CI only if you are prepared for a failing build to mean "the device was recalibrated," which it usually will. Chapter 12 measured a 288× spread in gate error across one chip; Chapter 30 will show how much of that moves week to week. A test whose failures you routinely ignore has negative value, because it trains the team to ignore failures.

27.9.1 When a test is worse than no test

"Negative value" is a strong claim and it deserves an accounting. A test's worth is

$$V \;=\; P(\text{catches a real bug}) \times C_{\text{bug}} \;-\; P(\text{false alarm}) \times C_{\text{investigation}} \;-\; C_{\text{maintenance}}$$

Every term but the first is a cost, so a test whose detection probability is zero has strictly negative value, and no amount of greenness changes the sign. This chapter has produced four species of it, and they are worth naming together because each one looks like diligence.

The blind test. 1,000 shots at tolerance 0.10: zero flakes, zero detection of a real $\varepsilon = 0.10$ defect. Its cost is not merely its runtime — it is the coverage the team believes it has and does not, which is the expensive kind.

The tautological test. $UU^\dagger = I$ and unitarity: exact, fast, elegant, and satisfied by every circuit that could ever be assembled. Three of §27.3's four oracle-free properties passed a circuit with process fidelity 0.9498 against its reference. A green light with no wiring behind it is worse than no light, because someone is steering by it.

The ignored test. The hardware run whose red build means "recalibrated." Its damage is transferable: it degrades every other test in the suite by teaching the team that red is a state to be retried rather than read.

The single-draw test. A seeded run reported as a measurement. This chapter's own first draft did it — 1.0% from 2/200, a rate that is actually 0.150% — and Chapter 24 §24.3 did it before that. The test is not broken; the inference from it is, and the failure mode is invisible because the number looks like every other number.

The common thread is that all four are silent. Flakiness announces itself; every other defect in this list is indistinguishable from success. That asymmetry is why the discipline this chapter argues for — quote the detection threshold, perturb the circuit, measure $\beta$ — cannot be replaced by attentiveness. There is nothing to be attentive to.

27.9.2 Where this chapter sits

Five chapters have now arrived at the same structure from different directions, and it is worth stating once as a single claim.

Chapter 5 established that counts are samples and a distribution is estimated rather than read. Chapter 26 measured a debugging tool blind on 41% of structured inputs and 0 of 100 random ones. This chapter turned that into two error rates and found the standard fix for one destroys the other. Chapter 28's optimization levels differ on 14 of 40 circuit-seed pairs — a difference invisible to any suite that fixes one seed. Chapter 30 will choose a benchmark statistic and pay §27.6.2's price for it.

In every case the failure is the same: a measurement was made, it came back clean, and nobody asked whether it could have come back otherwise. The tolerance below the noise floor, the property no bug can violate, the seed that pins one draw, the statistic that has averaged away the defect — these are four spellings of one word.

The test for it is always available and always cheap. Break the code on purpose and confirm the measurement notices. Everything else in this chapter is an elaboration of that one line.

What we measured

• Assertion costs, per test, 3-qubit QFT: unitary equality 0.34 ms, statevector on one input 0.21 ms, twenty random inputs 5.75 ms, transpile-and-verify 12.76 ms, 1,000 shots 85.43 ms, 100,000 shots 631.25 ms. Sampled tests cost 250–3,000× exact ones.
• Unitary equality is $\mathcal{O}(4^n)$ and takes 16.5 s at 12 qubits; twenty random statevectors take 74.8 ms at the same size. Crossover around 9 qubits.
• ★ Three of four oracle-free properties pass a definitely-broken QFT: $UU^\dagger = I$, unitarity, and $\text{QFT}|000\rangle$ uniform. Only the shift theorem caught it, with a worst deviation of 0.1830.
• Shot noise on a correct GHZ(3): mean TVD 0.04375 at 100 shots, 0.01313 at 1,000, 0.00423 at 10,000, 0.00121 at 100,000 — tracking $1/\sqrt N$.
• A tolerance of 0.05 at 1,000 shots gives a false-failure rate of 0.150% (3/2,000, 95% upper bound 0.320%) — about one red build in four or five commits for a 200-test suite. An earlier 2/200 estimate read 1.0%; measuring a rate that small needs thousands of runs, and a zero count bounds it at $3/N$, not at zero.
• ★★ 1,000 shots with tolerance 0.10 never flakes and is 0% sensitive to a real $\varepsilon=0.10$ bug. 10,000 shots with tolerance 0.02 also never flakes and catches $\varepsilon = 0.05$ 86% of the time. The fix for flakiness is shots, not tolerance.
• A seeded run gives 104/96 every time — reproducible, and still not 100/100.
• ★ Two more oracle-free properties DO catch the broken QFT: $U^2 = R$ (deviation 0.5045) and $U^4 = I$ (deviation 0.7624), where $R$ is the reversal permutation. Both use the circuit forwards, so no inverse reproduces the bug. The scoreboard is four of six useful, not one of four.
• The shot-noise floor is derivable: $\mathbb{E}[\mathrm{TVD}] = 0.39894\,S/\sqrt{N}$ with $S = \sum_i\sqrt{p_i(1-p_i)}$. For GHZ(3), $S = 1$; every measured mean above is within 0.9 standard errors of the derived value. For a uniform output over $M$ outcomes $S = \sqrt{M-1}$, so holding a TVD tolerance while adding qubits costs shots like $2^n$.
• ★ The project module's shot_noise_floor refuses the configuration this chapter recommends. assert_distribution(shots=10_000, tolerance=0.02) raises ValueError — "this test would fail on CORRECT code" — for a configuration measured at 0/2,000 false failures. The 3.0 constant is right for an output spread over ~7 outcomes and 2.6× conservative for GHZ(3)'s two.
• ★★ Both error rates have closed forms that reproduce the measured tables: $\alpha = 2\Phi(-2\tau\sqrt N)$ predicts 0.1565% where 0.150% (3/2,000) was measured, and the power formula predicts 84.1% where 86% was measured. Detection is 50% exactly when the bug's true TVD equals the tolerance.
• The smallest bug a non-flaky test can see is $\delta_{\min} = 2.066/\sqrt N$ at 80% power and a 0.1% flake budget. Halving it costs 4× the shots. At 1,000 shots, Case Study 2's team could have used $\tau = 0.052$ instead of 0.10 for the same flake rate they already tolerated.

The theme: a test that cannot fail is not evidence — now with the two error rates measured, and the discovery that the usual fix for a flaky test is the fastest way to produce one.