Quiz: Benchmarking Quantum Hardware

Answers with explanations at the end.

1. What does a backend's Target actually hold for each instruction?

2. How many of FakeSherbrooke's 144 two-qubit edges are dead, and what does "dead" mean?

3. Why does the mean exceed the median for every error channel on this device?

4. Give the spread (max/min) for two-qubit gates, single-qubit gates, and readout.

5. Describe the randomized benchmarking protocol in two sentences.

6. RB was run against a known 0.2% depolarizing error. What did it report, and what does that demonstrate?

7. Why is RB insensitive to state-preparation and measurement error?

8. What is the Clifford twirl, and why is it simultaneously what makes RB valid and what limits it?

9. The same chip supports quoted two-qubit errors of 0.00750 and 0.07205. Explain how, and say whether anyone is lying.

10. What are the four questions to ask before comparing two vendors' quoted fidelities?

11. $(1-0.00750)^{257} = 0.1445$; Chapter 28 measured 0.1290. Why is a chip-wide median that good a predictor of one circuit?

12. When does the median stop being predictive? Give the measured example.

13. Complete: "A summary statistic is a description of ______."

14. Name four things randomized benchmarking averages away.

15. On this device, which is larger — median readout error or median two-qubit gate error? By how much?

16. Two devices have identical mean edge error. Under what circumstance are they indistinguishable, and under what circumstance do they differ by 7.8×?

17. Define Quantum Volume and give its two main limitations.

18. Why might a device with a few excellent qubits and many poor ones score badly on QV and still run your circuit well?

19. Give the two structural objections to cross-entropy benchmarking as evidence that a device will run your program.

20. What is the only benchmark that answers "will my program work?", and what do you do when you cannot simulate its reference?

Answers

1. A distribution — a per-qubit or per-qubit-pair error and duration for every instruction. Not a number. Two-qubit gates alone have 144 entries spanning 34× between best and worst live edge.

2. Nine — 6% of the connectivity. "Dead" means error exactly 1.0: uncalibrated or failed in that snapshot. Chapter 29 §29.4 pinned a layout across two of them and the circuit returned 0.6790 against an expected 0.9116.

3. Because the distributions are right-skewed — a few very bad elements pull the average up while the median resists them. For readout the mean is 2.1× the median (0.0415 against 0.0198).

4. Two-qubit 34× (0.00347 to 0.11736), single-qubit 124× (0.00011 to 0.01333), readout 171× (0.00293 to 0.50000). Chapter 12 measured 288× on a different snapshot.

5. Apply $m$ random Clifford gates, then the single Clifford inverting their product; a perfect device returns $|0\dots0\rangle$ every shot. Fit $P(0) = Ap^m + B$ across several $m$, and the decay $p$ gives the average error per Clifford.

6. Fitted decay 0.99544, error per Clifford 0.00228, against an injected 0.002 per gate. It demonstrates the method is sound — each Clifford is roughly two physical gates, so an error per Clifford near the per-gate figure is exactly right.

7. Because SPAM error changes the fit's amplitude $A$ and offset $B$, while the reported quantity is the exponent $p$. The method is self-calibrating by construction.

8. Twirling over the Clifford group converts an arbitrary noise channel into a depolarizing channel with the same average fidelity. That is what makes a single exponential fit valid regardless of the actual noise — and it is exactly what discards coherent errors, correlated errors, and gate-dependence. The averaging is the mechanism, not a flaw.

9. Median with dead edges excluded gives 0.00750; mean with dead edges included gives 0.07205. Nobody is lying — excluding uncalibrated edges is defensible (you would not route through them), and reporting a median is defensible (it resists outliers). Every choice is defensible and the combination spans a factor of 9.6, with implied 100-gate survival from 47% to 0.06%.

10. Median or mean? Dead elements included or excluded? Which gate? Averaged over all qubits or over the best subset? Two vendors making different defensible choices produce numbers that cannot be compared, and 99.25% against 99.20% sits well inside that range.

11. Because the transpiler picks good qubits. VF2Layout scores candidate embeddings against the full error record, so a transpiled circuit lands preferentially on the better part of the distribution. The median describes the qubits you get, not the qubits that exist.

12. The moment you pin a layout yourself. Chapter 29 §29.4's hand-chosen chain crossed two dead edges and returned 0.6790 against an expected 0.9116 — the tail of the distribution rather than its middle.

13. "...a description of a sampling process." Change how the sampling happens and the statistic stops describing anything.

14. Any four of: readout/SPAM error (cancels by design); coherent errors (the twirl converts them to a depolarizing rate); crosstalk (standard RB benchmarks qubits in isolation); gate-dependent errors (error per Clifford mixes however many physical gates each uses); per-qubit structure (one number for a 34× distribution); drift (the number is from whenever calibration last ran).

15. Readout, by roughly 2.6× — median 0.0198 against 0.0075 for two-qubit gates. It is the largest error channel on this device, has the widest spread, and the standard gate benchmark is designed not to see it.

16. They are indistinguishable when the circuit must use every edge (survival 0.9228 against 0.9207). They differ by 7.8× in infidelity when the circuit can avoid one bad edge (0.9303 against 0.9910). Which is better depends entirely on your circuit's shape — which is what Chapter 29 §29.2 engineers for and what a single averaged number cannot express.

17. The largest $2^n$ for which random square circuits ($n$ qubits, $n$ layers of random SU(4) on random pairs) beat a $2/3$ heavy-output threshold. Limitations: (a) it measures width and depth in a fixed ratio, and most real circuits are not square; (b) it uses random pairs across the whole chip, sampling parts your circuit will never touch. It is also quantized in powers of two, so real degradations may not register.

18. Because QV samples the whole connectivity graph with random pairs, so the poor qubits drag the score down — while a hardware-aware circuit (Chapter 29) deliberately uses a short chain of the good ones and never touches the rest.

19. (a) It requires simulating the ideal distribution, which is the thing being claimed intractable — so verification happens where simulation is possible and the claim is extrapolated. (b) It benchmarks a random circuit, the most favourable case for a quantum device and the least for a classical simulator — the same objection Chapter 21 §21.7 raised about Grover's strawman baseline and Chapter 24 §24.5 about QAOA versus Goemans–Williamson.

20. Your circuit's $1-\text{TVD}$ against a noiseless reference, with a standard error over transpiler seeds, at several sizes. When the reference cannot be simulated, fall back to Chapter 26 §26.8: verify classically — Chapter 23's Shor checks its factors and Chapter 24's QAOA checks its cut. A result you can verify is a result you can benchmark, at any size.