Further Reading: Testing Quantum Programs

Tagged Tier 1 (confident it exists and recommended) and Tier 2 (real and worth seeking, but verify the current version or URL).

As with Chapter 26, this is a young literature. The most valuable reading here is classical: the software-testing community solved most of these problems decades ago, and the quantum-specific part is smaller than it first appears.

The classical reading that transfers directly

• Anything on property-based testing — the Hypothesis documentation (Python) or Claessen and Hughes, "QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs" (2000). §27.4's entire conclusion is QuickCheck's thesis: random inputs beat chosen inputs, because chosen inputs carry the same blind spots as the code. Chapter 26 measured it at 41% versus 0%. Tier 1.
• Literature on mutation testing. Deliberately perturb the code and check that the test suite notices. §27.3's discriminating flag is mutation testing with one mutant, and Case Study 1 is a suite that would have been caught by it on day one. Tier 1.
• Chen, Cheung, and Yiu, "Metamorphic testing: a new approach for generating next test cases" (1998). The original metamorphic testing paper, from the classical world. The technique §27.3 depends on, invented for exactly the situation quantum computing is permanently in: no oracle. Tier 1.
• Any treatment of statistical power in hypothesis testing. §27.6's two error rates are Type I and Type II errors, and the relationship between tolerance, sample size, and power is standard statistics. Read this before choosing a tolerance. Tier 1.
• The "rule of three" for zero-event confidence bounds. Observing 0 failures in $N$ runs bounds the rate at about $3/N$, not at zero. One page, and it prevents the mistake §27.5 made and corrected. Tier 1.

The quantum-specific work

• Huang and Martonosi, "Statistical assertions for validating patterns and finding bugs in quantum programs" (2019), ISCA. Chapter 26's recommendation, equally relevant here: assertions checked statistically, with the shot cost taken seriously. Tier 1.
• Huang and Martonosi, "QDB: From Quantum Algorithms Towards Correct Quantum Programs" (2018). A catalogue of real bugs in real quantum programs. Useful for choosing which failure modes your properties should target — which §27.3 argues is the only reliable way to design a property. Tier 1.
• Work on metamorphic testing specifically for quantum programs. A small but growing literature that catalogues quantum-specific metamorphic relations — the shift theorem is one instance of a general family. Tier 2 — check for recent surveys.
• Muskit, QuCAT, and similar quantum mutation- and combinatorial-testing tools. Automated mutant generation for quantum circuits, which is §27.3's discriminating check done systematically. Tier 2 — verify current maintenance status before adopting.
• Quantum equivalence checking with decision diagrams (the MQT tooling from TU Munich). Verifies circuit equivalence far beyond the $4^n$ limit that stops Operator at 12 qubits. The scalable version of §27.2's top tier, and the most practically useful item in this section. Tier 2.

On the statistics of distributions

• Standard treatments of total variation distance, and of goodness-of-fit tests (chi-squared, Kolmogorov–Smirnov). §27.5 uses TVD because it is interpretable and assumption-free; a chi-squared test can be more powerful at the same shot count, at the cost of assumptions. Exercise 27.26. Tier 1.
• Literature on distribution testing / identity testing in theoretical computer science. The question "how many samples to distinguish a distribution from a target?" has a formal answer, and it is $\Theta(\sqrt{2^n}/\varepsilon^2)$ for $n$-bit distributions. Note the $\sqrt{2^n}$: testing gets harder with qubit count, not just with precision. Tier 2.
• Cross-entropy benchmarking (XEB) as used in quantum supremacy experiments. A distribution test at extreme scale, with all of §27.5's issues and the added problem that the target distribution cannot be computed. Chapter 30's subject. Tier 1.

Tooling

• pytest parametrization, fixtures, and pytest.approx. §27.4's pattern is one decorator. If you write Python tests and have not read the parametrize documentation properly, do that first. Tier 1.
• Qiskit's own test suite. A large, mature example of testing quantum software, and it is public. Read how they test the transpiler in particular — it is the hardest thing in the codebase to test and the techniques are transferable. Tier 1.
• qiskit.quantum_info.random_statevector and random_unitary, and the Haar measure they sample from. Exercise 27.18 asks whether Haar-random is the best choice for bug-finding; it is worth knowing what you are drawing from. Tier 1.

Backward references

• Chapter 24 — the shot budget, and the $1/\sqrt N$ wall that sets every distribution test's sensitivity.
• Chapter 26 — the blind-input measurement this chapter's policy is built on, and the equality tests §27.1 inherits.

Forward references

• Chapter 28 — optimization, where a test suite stops being optional: every optimization is a claim that two circuits are equivalent.
• Chapter 30 — benchmarking, which is distribution testing pointed at the device instead of the code.

Where to go next. If one thing: the Hypothesis documentation, or the original QuickCheck paper. §27.4 is a rediscovery of their central claim, and the classical community has twenty-five years of accumulated technique for generating inputs that find bugs.

If two: add Chen et al. on metamorphic testing, the source of §27.3's method — and read it with Case Study 1 in mind, because the paper is clear about something this chapter learned by measurement: a metamorphic relation is only as good as its ability to be violated.

Then Chapter 28, where every transformation you apply is a claim of equivalence, and this chapter's suite is what makes the claim checkable.