Exercises: Testing Quantum Programs

These need qiskit, qiskit-aer, pytest, and qiskit-ibm-runtime for the fake backends. Solutions to starred exercises are in Answers to Selected Exercises.

A standing rule for this chapter: every test you write here must be checked against a deliberately broken circuit as well as a correct one. A test that has never been shown to fail is not yet a test.

The tier list

27.1 ★ Time the four kinds of assertion on a 3-qubit circuit and reproduce §27.2's table. How many of each fit in one CI-minute on your machine?

27.2 ★ Compute the ratio between the cheapest and most expensive assertion. What does it imply about how many sampled tests a suite can afford?

27.3 ★★ Time Operator equality and twenty random statevectors at 3, 5, 7, 9, 11, and 12 qubits. Where is the crossover on your machine, and why does it occur where it does?

27.4 ★★ At what qubit count does Operator equality exceed one second per assertion? Ten seconds? What is the largest circuit you could reasonably unit-test this way in CI?

27.5 ★★ Chapter 26's verify_transpilation costs about 13 ms. Measure it, and decide whether it belongs on every commit or nightly for a suite of 50 circuits.

Testing without an oracle

27.6 ★ Build a QFT with one deliberately wrong rotation angle. Confirm Operator inequality and compute the process fidelity against the correct version.

27.7 ★ Implement the adjoint round trip $UU^\dagger = I$ and confirm it passes both circuits. Explain in one sentence why it always will.

27.8 ★ Implement the unitarity check and confirm the same. What is this assertion actually testing?

27.9 ★★ Implement the shift theorem property and confirm it catches the bug. Report the worst deviation.

27.10 ★★ Write check_property(name, predicate, circuit, perturbed) returning a discriminating flag. Apply it to all four properties and reproduce the scoreboard.

27.11 ★★ Find a second property that catches the bug, different in kind from the shift theorem. Is it cheaper or more expensive?

27.12 ★★★ Design a property that would catch a swapped pair of qubits but that the shift theorem misses. What does this tell you about property coverage?

27.13 ★★★ Mutation-test your whole property suite: generate 20 perturbed circuits (wrong angles, missing gates, swapped qubits, extra gates) and report what fraction each property catches. Which property has the broadest coverage?

Property-based testing

27.14 ★ Write the parametrized random-input test from §27.4 and confirm it passes for a correct circuit and fails for the broken one.

27.15 ★ Show that a single test looping over twenty random states reports less useful information on failure than twenty parametrized tests. Why?

27.16 ★★ Reproduce Chapter 26's blind-fraction counts (4/8, 11/27, 0/100) for the QFT bug.

27.17 ★★ Write verify_against_reference(mine, reference, n_seeds) that refuses fewer than 8 seeds, and justify the number you choose in the error message.

27.18 ★★★ Random statevectors are drawn from the Haar measure. Is that the best distribution for finding bugs? Compare the detection rate of Haar-random inputs against random computational basis states and random product states on the same bug.

Distribution tests

27.19 ★ Measure the shot-noise TVD of a correct GHZ(3) at 100, 1,000, 10,000, and 100,000 shots over 40 runs. Confirm the $1/\sqrt N$ scaling.

27.20 ★ Implement shot_noise_floor(shots) and justify the safety factor from your measurement of the max-versus-mean ratio.

27.21 ★★ Measure the false-failure rate at 1,000 shots and tolerance 0.05. Use at least 2,000 runs and report a confidence interval. Why is 200 runs not enough?

27.22 ★★ Your measurement in 27.21 gives zero failures at some configuration. What upper bound can you place on the rate? Look up the rule of three and apply it.

27.23 ★★ Sweep the bug size $\varepsilon$ and reproduce §27.6's detection table for both configurations.

27.24 ★★ For a test at 1,000 shots and tolerance 0.10, find the smallest $\varepsilon$ detected more than 50% of the time. Do the same at 10,000 shots and tolerance 0.02.

27.25 ★★ Write assert_distribution so that it raises ValueError (not AssertionError) when the tolerance is below the noise floor, and include the required shot count in the message. Why is the distinction between the two exception types worth making?

27.26 ★★★ Replace TVD with a chi-squared goodness-of-fit test. Does it detect smaller bugs at the same shot count? What does it assume that TVD does not?

27.27 ★★★ Derive the shot count needed to detect a bug of size $\delta$ (in TVD) with 90% power at a 1% false-failure rate. Check your formula against §27.6's measurements.

Seeding

27.28 ★ Run the same circuit five times unseeded and five times seeded. Confirm the seeded runs are identical, and that they are not the exact distribution.

27.29 ★★ Build a GHZ state two ways — h(0) and ry(pi/2, 0) — and show that the same seed_simulator gives different counts. Explain why, and say what it implies for a seeded test.

27.30 ★★ Find a seed for which a correct circuit fails a tolerance-0.05 test at 1,000 shots. Then explain why "pick a seed that passes" is not a valid response.

27.31 ★★★ Design a seeding policy for a test suite: which seeds, how many, fixed or rotating, and what happens when one fails. Justify each choice against both error rates.

Project

27.32 ★★ (Project Checkpoint) Build vqelab/testing.py with tvd, exact_distribution, sampled_distribution, shot_noise_floor, DistributionTest, assert_distribution, check_property, and verify_against_reference. Write tests asserting:

  1. TVD endpoints and shape rejection.
  2. exact_distribution matches the GHZ distribution; sampled converges to it.
  3. shot_noise_floor scales as $1/\sqrt N$ and rejects non-positive shots.
  4. DistributionTest has no default shot count or tolerance (inspect.signature).
  5. A tolerance below the noise floor is flagged.
  6. assert_distribution raises ValueError — not AssertionError — below the floor.
  7. It passes correct code and raises AssertionError for a real bug.
  8. A test at 1,000 shots / tolerance 0.10 has BOTH a zero false-failure rate and a zero detection rate, and summary() says NEVER FLAKES BECAUSE IT IS BLIND.
  9. The same circuit at 10,000 shots / tolerance 0.02 has a zero false-failure rate AND catches $\varepsilon = 0.10$ every time.
  10. $\varepsilon = 0.10$ really does shift the exact distribution by TVD ≈ 0.0499.
  11. $UU^\dagger = I$ is NOT discriminating.
  12. Unitarity is not discriminating.
  13. A known answer on $|000\rangle$ is not discriminating.
  14. The shift theorem IS discriminating.
  15. Exactly three of the four oracle-free properties pass the broken circuit.
  16. Random inputs catch what $|000\rangle$ misses.
  17. verify_against_reference refuses fewer than MIN_SEEDS inputs.
  18. Exact assertions are more than 20× cheaper than sampled ones.

Tests 8, 9, 11 and 15 are the ones this chapter exists to encode.

27.33 ★★★ Extend the module with detection_threshold(correct, family, target_power=0.5) that sweeps a family of perturbed circuits and returns the smallest detectable bug. Make DistributionTest.summary() report it automatically.

Going further

27.34 ★★ Take Chapter 21's Grover implementation and write a full suite for it: which tier does each assertion belong to, and which properties are discriminating?

27.35 ★★★ Design the CI configuration from §27.9 for a real repository — which tests run on every commit, which nightly, and what the total wall-clock budget is. Then justify why nothing runs on hardware.

27.36 ★★★ Chapter 25's error-correction code has a natural metamorphic property: the logical error rate must decrease with distance below threshold and increase above it. Write it as a test. What shot count does it need, and what is the smallest deviation it can detect?