Exercises: Benchmarking Quantum Hardware

These need qiskit, qiskit-aer, and qiskit-ibm-runtime. Solutions to starred exercises are in Answers to Selected Exercises.

A standing rule for this chapter: never quote a device number without saying which statistic it is and whether dead elements are included. Case Study 1 is what happens otherwise.

The distribution

30.1 ★ Pull the full ecr, sx, and measure error records from FakeSherbrooke and report min, p25, median, mean, p75, p95, and max for each.

30.2 ★ Count the dead elements (error = 1.0) in each. What fraction of the two-qubit connectivity is unusable?

30.3 ★ Compute the spread (max/min over live elements) for all three. Which channel has the widest?

30.4 ★★ Show that the mean exceeds the median for all three, and explain what that says about the shape of the distribution.

30.5 ★★ Plot histograms of all three on a log scale. Which looks closest to unimodal?

30.6 ★★ Readout error has a median of 0.0198 and a max of 0.5000. What does a readout error of exactly 0.5 mean physically, and what does that qubit contribute to a computation?

Choosing a statistic

30.7 ★ Reproduce §30.3's table: six defensible statistics and their implied 100-gate survivals.

30.8 ★ What is the ratio between the smallest and largest? What is the ratio of the implied survivals?

30.9 ★★ Write quoted_fidelity(dist, statistic, include_dead) with no defaults for either choice. Justify the absence of defaults in the docstring.

30.10 ★★ Your circuit uses a 6-qubit chain — five edges. Compute the relevant survival for the best available chain and compare it against the chip-wide median. Which is the right number for you?

30.11 ★★★ Two vendors quote 99.25% and 99.20% two-qubit fidelity. Construct two calibration records consistent with those quotes where the second device is dramatically better for a 15-gate linear circuit.

Randomized benchmarking

30.12 ★ Implement rb_circuit(n, m, rng) and confirm that with no noise it returns $|0\dots0\rangle$ every shot.

30.13 ★ Run 1-qubit RB against a known 0.2% depolarizing error and reproduce the decay table.

30.14 ★★ Fit $P(0) = A p^m + B$ and recover the error per Clifford. How close is it to the injected value, and why is it not exactly equal?

30.15 ★★ Show that RB is insensitive to readout error: add 10% readout error to the noise model and confirm the fitted decay $p$ barely changes while $A$ and $B$ do.

30.16 ★★ Run 2-qubit RB. Why does the same decay $p$ imply a larger error per Clifford on two qubits than on one?

30.17 ★★★ Implement interleaved RB: alternate random Cliffords with a specific gate under test, and extract that gate's error from the ratio of decays. Validate it against a known injected error on that gate alone.

30.18 ★★★ Construct a noise model with a purely coherent error (a small over-rotation) of the same average fidelity as a depolarizing one. Confirm RB reports the same number for both, then show your circuit does not.

What benchmarks miss

30.19 ★★ Build the "same mean, different device" comparison from §30.5 and reproduce the 7.8× infidelity ratio when one edge can be avoided.

30.20 ★★ For a circuit that must use every edge, are the two devices distinguishable? What does that tell you about when averaged benchmarks are adequate?

30.21 ★★ Take a transpiled circuit, extract final_index_layout(), and look up the readout error of exactly those qubits. How does it compare to the chip median?

30.22 ★★★ Write a regression check that would have caught Case Study 2's failure: track the readout error of the qubits your circuit actually uses, week over week.

Quantum Volume

30.23 ★ For $n = 2 \ldots 8$, compute the two-qubit gate count of a QV circuit and the implied survival at the median error rate. At what $n$ does it fall below the heavy-output threshold's neighbourhood?

30.24 ★★ Implement a QV circuit generator: $n$ layers of random SU(4) on random pairs. Confirm the gate count matches $n \lfloor n/2 \rfloor$.

30.25 ★★ QV uses random pairs across the whole chip. Modify it to use only the best 6-qubit chain and compare the two scores. What does the difference measure?

30.26 ★★★ Chapter 28's Grover circuit is 5 qubits and 257 two-qubit gates; Chapter 29's ansatz is 6 qubits and 15. Compute what QV would predict for each and compare against the measured fidelities. How useful is QV as a predictor here?

Project

30.27 ★★ (Project Checkpoint) Build vqelab/benchmarking.py with error_distribution, ErrorDistribution, quoted_fidelity, predict_survival, readout_summary, fit_rb_decay, and error_per_clifford. Write tests asserting:

  1. ErrorDistribution has no .error attribute — there is no single "the error."
  2. Nine two-qubit edges are dead, and 135 are live.
  3. The mean exceeds the median for gates and for readout (2.1× for readout).
  4. statistic rejects an unknown name; error_distribution rejects an unknown instruction.
  5. The same chip supports a factor of 9 in quoted two-qubit error.
  6. Including dead edges multiplies the mean by ~7.1×.
  7. Implied 100-gate survival ranges from above 45% to below 1%.
  8. quoted_fidelity has no default for statistic or include_dead.
  9. The median predicts Chapter 28's measured 0.1290 to within ~12%.
  10. A pinned layout makes the prediction trustworthy is False, and the summary quotes 0.6790.
  11. Survival falls monotonically with gate count; Shor(15)'s 3,368 gates do not survive.
  12. 12 of 127 qubits exceed 10% readout error and one is at exactly 0.5000.
  13. Readout median exceeds two-qubit gate median.
  14. fit_rb_decay recovers a known decay from synthetic data.
  15. error_per_clifford(0.99544, 1) ≈ 0.00228 — the measured run.
  16. The dimension factor makes the same decay mean more error on more qubits.
  17. error_per_clifford's docstring says AVERAGE and names what the twirl discards.
  18. Two devices with identical mean error differ by more than 7× in infidelity once one edge can be avoided.

Tests 5, 8, 9, 10 and 12 are the ones this chapter exists to encode.

30.28 ★★★ Extend the module with circuit_specific_error(backend, transpiled) that reads the layout, looks up the errors of exactly the edges and qubits used, and returns a survival estimate. Compare it against the chip-wide median for several circuits — when do they diverge?

Going further

30.29 ★★ Look up CLOPS (Circuit Layer Operations Per Second). What does it measure that none of this chapter's benchmarks do, and why did it become necessary?

30.30 ★★★ Read a quantum supremacy paper and its strongest classical-simulation rebuttal. What exactly was claimed, what was simulated to verify it, and how much of the gap has closed?

30.31 ★★★ Design an application benchmark for this book: a suite of circuits from Chapters 21, 23, 24 and 29, each with a classically-verifiable output, scored on a single device. What does your suite measure that RB and QV do not, and what does it still miss?