Chapter 27 — Key Takeaways (Testing Quantum Programs)
Chapter 26's anecdote, turned into policy — with both error rates measured.
The problem
For most circuits worth writing there is no oracle. If an independent source of the correct answer existed, you would not need the quantum computer. Chapter 23's Shor is the rare exception, and only because multiplication is easy.
So: what can you assert about a program whose right answer you do not know?
Four kinds of test
1. UNITARY EQUALITY Operator(mine) == Operator(reference)
Exact, input-independent. Needs a reference and 4^n memory.
2. STATE ASSERTION Statevector matches an expected state
Exact. Depends entirely on the input you chose.
3. PROPERTY a relation that must hold, no reference at all
Exact. Only as strong as the property.
4. DISTRIBUTION measured counts match an expected distribution
STATISTICAL. The only one that works on hardware.
★ What each tier costs
Per assertion, 3-qubit QFT:
unitary equality 0.34 ms 174,006 per CI-minute
statevector, 1 fixed input 0.21 ms 279,433
statevector, 20 random 5.75 ms 10,437
transpile + verify 12.76 ms 4,701
counts, 1,000 shots 85.43 ms 702
counts, 100,000 shots 631.25 ms 95
Sampled tests cost ~250–3,600× exact ones. Tens of thousands of exact assertions per CI-minute, or a few hundred sampled ones. Every check that can be made exact should be.
But they scale differently:
qubits Operator equality statevector x20
7 5.9 ms 17.9 ms
9 149.6 ms 30.6 ms <-- crossover
12 16,526.8 ms 74.8 ms
⚠️ The strongest test is not the best test at every size. Unitary equality is $\mathcal{O}(4^n)$ and cannot be blind — the right default until ~9 qubits, then unaffordable. Pick the tier from the size you must test, then make that tier as strong as it can be.
(Timings move tens of percent run to run. Read the ratios.)
★★ Testing without an oracle: three of four properties are worthless
Applied to a QFT with one wrong rotation angle (process fidelity 0.949760):
property needs a reference? catches the bug?
U U^dag == I no NO
unitarity no NO
QFT|000> uniform no NO
shift theorem no YES deviation 0.1830
random inputs vs QFTGate YES YES infidelity 0.0665
$UU^\dagger = I$ is blind by construction — qc.inverse() inverts your circuit, so the inverse
carries the same wrong angle and cancels it exactly. It is the most-cited property in quantum testing
and it can never catch a wrong gate parameter.
Unitarity tests the framework, not your code — every assembled circuit is unitary.
$\text{QFT}|000\rangle$ uniform is Chapter 26 §26.4's blind spot arriving as a test.
⚛️ A property is only useful if the bug can violate it.
The three failures each constrain one point or one automatic algebraic feature. The shift theorem constrains the relationship between different inputs — exactly where a wrong controlled-phase angle lives.
Write down the failure mode first, then the property that would break under it. The other order produces properties that are true, cheap, elegant, and worthless.
Property-based testing
computational basis blind: 4/8 structured (0,1,+): 11/27 (41%)
random states blind: 0/100
@pytest.mark.parametrize("seed", range(20))
def test_matches_reference_on_random_inputs(seed):
v = random_statevector(2 ** N, seed=seed)
assert state_fidelity(v.evolve(mine), v.evolve(ref)) == pytest.approx(1.0, abs=1e-9)
Seed the inputs, parametrized (a failure names its input). Use more than one. Never let $|0\dots0\rangle$ be your only input.
★★ Testing a distribution: the shot-noise floor
TVD of a correct GHZ(3) against exact:
shots mean TVD std max over 40 runs 1/sqrt(N)
100 0.04375 0.03199 0.12000 0.10000
1,000 0.01313 0.01003 0.03700 0.03162
10,000 0.00423 0.00283 0.01160 0.01000
100,000 0.00121 0.00101 0.00401 0.00316
Every number is the error of a correct circuit. Chapter 24 §24.3's $1/\sqrt N$ wall, in a new discipline. A usable floor is $\approx 3/\sqrt{N}$.
★★ The flaky test and the blind test are the same knob
bug size eps true TVD detected @1k/tol.10 detected @10k/tol.02
0.05 0.0250 0% 86%
0.10 0.0499 0% 100%
0.20 0.0993 52% 100%
Both configurations have a 0% false-failure rate. Neither flakes.
1,000 shots / tolerance 0.10 is 0% sensitive to a real $\varepsilon = 0.10$ bug. It stopped flaking because it stopped being able to fail.
10,000 shots / tolerance 0.02 catches $\varepsilon=0.05$ 86% of the time and $\varepsilon=0.10$ always.
🔬 A test has TWO error rates, and tuning one blindly destroys the other.
Flakiness is loud; blindness is silent. A flaky test files its own ticket. A blind test says nothing, and its silence is indistinguishable from correctness.
Fix flakiness with SHOTS, not tolerance. Shots cost CPU seconds you can see. Tolerance costs detection and does not send an invoice.
📌 Quote a distribution test's tolerance, its shot count, AND the smallest bug it can detect. The first two describe what a test costs. Only the third describes what it does.
Seeding: reproducible ≠ accurate
UNSEEDED: {'00': 92,'11':108} {'00':102,'11':98} {'00':98,'11':102} ...
SEEDED: {'00':104,'11': 96} every single time
104/96, not 100/100. Seeding fixes which draw you get; it does not make the draw representative — exactly Chapter 24 §24.3's correction to reporting one seeded value instead of a mean over 25.
Seed so the suite is deterministic. Use enough shots so the number is right. Two problems; seeding solves the first.
Testing what actually runs
assert process_fidelity(Operator.from_circuit(t), Operator(padded)) == pytest.approx(1.0)
12.76 ms, 4,701 per CI-minute. Worth it not because the transpiler is buggy but because it catches
your own configuration errors (bad layout, an optimization level that removes what you needed —
Ch. 25's id slots) and pins behaviour across upgrades (Ch. 24's deprecated shots=, Ch. 19's
removed mcx(mode="v-chain")).
What goes in CI
EVERY COMMIT (seconds) unitary equality · discriminating properties ·
~20 random-input comparisons · transpilation
verification · ancilla-cleanliness assertions
NIGHTLY (minutes) distribution tests at 10,000+ shots · 8-12 qubits ·
noise-model runs
MANUAL, NOT CI anything needing a real device -- that is validation
A test whose failures you routinely ignore has negative value, because it trains the team to ignore failures.
Common pitfalls
- Assuming a property suite is a test suite because it is green.
- Asserting necessary conditions ($UU^\dagger=I$, unitarity) and calling it coverage.
- Letting $|0\dots0\rangle$ be the only test input.
- Treating an available reference as unavailable because you are reimplementing it.
- Choosing a tolerance without computing the shot-noise floor.
- Fixing flakiness by loosening tolerance.
- Reporting a tolerance without the shot count and detection threshold.
- Believing a seeded test is an accurate one.
Project piece added this chapter
vqelab/testing.py — tvd, exact_distribution, sampled_distribution, shot_noise_floor;
DistributionTest that cannot be built without a shot count and reports false_failure_rate and
detection_rate, printing "THIS TEST NEVER FLAKES BECAUSE IT IS BLIND" when both are near zero;
assert_distribution raising ValueError (not AssertionError) for a sub-floor tolerance, with
the required shot count in the message; check_property recording discriminating; and
verify_against_reference refusing fewer than MIN_SEEDS inputs. 33 tests pass, including
test_a_test_that_never_flakes_because_it_is_BLIND,
test_the_famous_adjoint_property_is_NOT_DISCRIMINATING, and
test_three_of_four_oracle_free_properties_pass_a_broken_circuit.