Quiz: Testing Quantum Programs

Answers with explanations at the end.

1. Why is testing quantum programs harder than testing classical ones, in one sentence that does not mention superposition?

2. Name the four kinds of quantum test and say which one works on hardware.

3. A sampled test costs how much more than an exact one? What does that ratio imply for suite design?

4. Unitary equality is the strongest test available. Why is it not always the right one?

5. $UU^\dagger = I$ is the most-cited property in quantum testing. What class of bug can it never detect, and why?

6. What is the unitarity assertion $U^\dagger U = I$ actually testing?

7. $\text{QFT}|0\dots0\rangle$ is the uniform superposition. Why does asserting it fail to catch a wrong controlled-phase angle?

8. State the shift theorem and explain why it catches the bug the other three properties miss.

9. Give the general rule for whether a property is worth asserting.

10. Of four oracle-free properties tested in §27.3, how many pass a circuit with 5% infidelity?

11. Write the parametrized property-based test pattern, and give the three details that make it work.

12. Why should a property-based test never loop over its random inputs inside one test function?

13. A correct GHZ(3) circuit at 1,000 shots has what mean TVD from its exact distribution? What sets the scaling?

14. Define the shot-noise floor and give a usable formula.

15. A distribution test at 1,000 shots with tolerance 0.10 has a 0% false-failure rate. Is it a good test? Justify with a number.

16. State the two error rates of a statistical test and explain why one is much more dangerous than the other.

17. What is the correct fix for a flaky distribution test, and what is the incorrect one?

18. Three numbers fully specify a distribution test. Name them, and say which is usually missing.

19. A seeded test gives {'00': 104, '11': 96} every run. What has seeding fixed, and what has it not?

20. Why should assert_distribution raise ValueError rather than AssertionError when the tolerance is below the noise floor?

Answers

1. Because you usually have no oracle — for most circuits worth writing there is no independent source of the correct answer, since if there were you would not need the quantum computer.

2. Unitary equality (exact, input-independent, needs a reference and $4^n$ memory); state assertion (exact, depends on the input chosen); property (exact, no reference, only as strong as the property); distribution (statistical). Only the distribution test works on hardware, and it is the weakest and most expensive.

3. Roughly 250–3,600×. It means a suite can afford tens of thousands of exact assertions and only a few hundred sampled ones, so every check that can be made exact should be, and sampled tests should be reserved for what genuinely requires sampling.

4. Because it is $\mathcal{O}(4^n)$ — 16.5 seconds per assertion at 12 qubits, against 74.8 ms for twenty random statevectors. The crossover is around 9 qubits. Pick the tier from the size you must test, then make that tier as strong as it can be.

5. Any error in a gate parameter. qc.inverse() inverts your circuit, so a wrong angle appears in both the circuit and its inverse and cancels exactly. The property is not weak; it is blind by construction to every error the inverse reproduces.

6. The framework. Every circuit assembled from library gates is unitary by construction, so this verifies that the gate definitions are unitary and that matrix multiplication works. Your code is not involved.

7. Because from $|0\dots0\rangle$ every control is in $|0\rangle$, so every controlled-phase gate acts as the identity and the wrong angle never fires. This is Chapter 26 §26.4's blind spot appearing as a test — and Chapter 26 measured that 4 of 8 basis states and 41% of structured states are blind to this bug.

8. $\text{QFT}|x+1\rangle$ equals $\text{QFT}|x\rangle$ with amplitude $k$ multiplied by $e^{2\pi i k/N}$. It catches the bug because it constrains the relationship between different inputs, and a wrong controlled-phase angle is precisely a wrong relationship between inputs that differ in the controlling bit. Worst deviation: 0.1830.

9. A property is only useful if the bug can violate it. Necessary conditions satisfied by an enormous space of wrong circuits make poor tests. Write down the failure mode first, then the property that would break under it — the reverse order produces properties that are true, cheap, elegant, and worthless.

10. Three of four. $UU^\dagger = I$, unitarity, and $\text{QFT}|000\rangle$ uniform all pass. Only the shift theorem fails.

11.

@pytest.mark.parametrize("seed", range(20))
def test_matches_reference_on_random_inputs(seed):
    v = random_statevector(2 ** N, seed=seed)
    assert state_fidelity(v.evolve(mine), v.evolve(ref)) == pytest.approx(1.0, abs=1e-9)

Seed the inputs, parametrized (so a failure names its input); use more than one (0 of 100 blind is a statistical claim, not a guarantee); never let $|0\dots0\rangle$ be your only input.

12. Because a loop reports one failure and hides which input caused it. Twenty parametrized cases are twenty independently reproducible tests, and the failing one names its own seed.

13. 0.01313 (std 0.01003, max over 40 runs 0.03700). The scaling is $1/\sqrt N$ — the same wall as Chapter 24 §24.3's shot budget, arriving in a different discipline.

14. The TVD level below which a correct circuit fails some fraction of the time. A usable formula is $\approx 3/\sqrt N$ — the safety factor of 3 covers the tail, since the max TVD over 40 runs was roughly 3× the mean at every shot count measured.

15. No. It is 0% sensitive to a real bug of size $\varepsilon = 0.10$ — a defect with a true TVD of 0.0499, half the tolerance and undetectable by construction. It does not begin to notice anything until the bug is twice that size, and even then catches it only 52% of the time. It never flakes because it can never fail.

16. Flakiness (fails on correct code) and blindness (passes on broken code). Blindness is far more dangerous because flakiness is loud and blindness is silent — a flaky test files its own ticket, while a blind test says nothing and its silence is indistinguishable from correctness.

17. Correct: more shots. 10,000 shots at tolerance 0.02 has a zero false-failure rate and catches $\varepsilon = 0.05$ 86% of the time. Incorrect: loosening the tolerance, which converts a test that occasionally failed on correct code into one that never fails on anything. Shots cost CPU seconds you can see; tolerance costs detection and does not send an invoice.

18. The tolerance, the shot count, and the smallest bug it can detect. The third is almost always missing, and it is the one that makes the test a test — the first two describe what a test costs without describing what it does.

19. Seeding has fixed which draw you get from the sampling distribution, making the suite deterministic. It has not made the draw representative — 104/96 is not 100/100. This is exactly Chapter 24 §24.3's correction: report a mean over repetitions rather than one reproducible draw. Seed so the suite is deterministic; use enough shots so the number is right.

20. Because a broken test is a different kind of failure from a broken circuit. An AssertionError says "your code is wrong"; a tolerance below the noise floor means the test would fail on correct code, so nothing has been learned about the circuit. The ValueError should carry the shot count that would make the test valid.