Chapter 30 — Key Takeaways (Benchmarking Quantum Hardware)

Where every error rate this book has quoted comes from, and how much of your decision it can carry.

The Target holds distributions, not numbers

   two-qubit (ecr):     144 entries, 9 DEAD (error = 1.0)
     min 0.00347   p25 0.00567   median 0.00750   mean 0.01018   p95 0.01999   max 0.11736  (34x)

   single-qubit (sx):   127 entries, 0 dead
     min 0.00011                 median 0.00024   mean 0.00056   p95 0.00112   max 0.01333  (124x)

   readout (measure):   127 entries, 0 dead
     min 0.00293                 median 0.01978   mean 0.04148   p95 0.15859   max 0.50000  (171x)

Nine of 144 two-qubit edges are dead — 6% of the connectivity, and Ch. 29 §29.4 routed through two of them. The mean exceeds the median everywhere (readout: 2.1×) because the tails are long.

Randomized benchmarking

Apply $m$ random Cliffords, then the Clifford inverting their product. Fit $P(0) = A p^m + B$.

   injected 0.002 depolarizing, 1 qubit
      m:   1      2      4      8     16     32
   P(0): .9955  .9928  .9883  .9803  .9605  .9303
   fitted decay p = 0.99544    error per Clifford = 0.00228

It recovers the injected error. RB is self-calibrating: SPAM affects the fit's amplitude and offset, not its exponent.

⚛️ Why the fit is valid, and what that costs. The Clifford twirl converts an arbitrary noise channel into a depolarizing one of the same average fidelity. That is what makes a single exponential correct regardless of the noise — and it is the mechanism that discards the structure, not a limitation of the analysis.

★★ The same chip, nine different fidelities

   statistic                              value   implied 100-gate survival
   median, dead edges EXCLUDED          0.00750                      0.4710
   mean,   dead edges excluded          0.01018                      0.3593
   median, dead edges INCLUDED          0.00779                      0.4576
   mean,   dead edges INCLUDED          0.07205                      0.0006
   p95,    dead excluded                0.01999                      0.1328
   worst live edge                      0.11736                      0.0000

A factor of 9.6, same chip, same day. Including the dead edges multiplies the mean by 7.1× and drops implied 100-gate survival from 47% to 0.06%.

⚠️ Nobody is lying. Excluding uncalibrated edges is defensible; reporting a median is defensible. Every choice is defensible and the combination spans an order of magnitude.

Before comparing devices ask: median or mean? dead included? which gate? all qubits or the best ones? 99.25% against 99.20% is well inside that range.

★★ ...and yet the median is predictive

$$(1 - 0.00750)^{257} = 0.1445 \quad\text{against Chapter 28's measured}\quad 0.1290$$

Ratio 1.12 — for a one-line estimate ignoring readout, decoherence, crosstalk and single-qubit error.

Why it works: the transpiler picks good qubits. VF2Layout scores embeddings against the full error record, so a transpiled circuit lands on the better part of the distribution. The median describes the qubits you GET, not the qubits that EXIST.

🔬 It stops predicting the moment you pin a layout yourself. Ch. 29's hand-chosen chain sampled two dead edges and returned 0.6790 against an expected 0.9116 — the tail, not the middle.

A summary statistic is a description of a sampling process. Change the sampling and it stops describing anything.

★ What RB cannot see

   readout error: median 0.0198, mean 0.0415, max 0.5000
   12 of 127 qubits above 10%    |    1 qubit at 0.5000 -- A COIN FLIP

   readout / SPAM error     cancels in the fit BY DESIGN
   coherent errors          the twirl converts them to a depolarizing rate
   crosstalk                standard RB benchmarks qubits in isolation
   gate-dependent errors    error per CLIFFORD mixes however many gates each uses
   per-qubit structure      one number for a distribution spanning 34x
   drift                    the number is from whenever calibration last ran

A benchmark that is robust to a class of errors is, from your circuit's perspective, blind to them. Robustness and blindness are the same property from two directions.

Readout is the largest error channel here — median 0.0198 against 0.0075 for two-qubit gates — with the widest spread (171×), and the standard gate benchmark is designed not to notice it.

★ Same mean, different device

   device      mean    median    worst      ALL ten edges    avoiding the worst
   uniform   0.0080    0.0080   0.0080             0.9228                0.9303
   skewed    0.0080    0.0010   0.0710             0.9207                0.9910

Identical mean. Indistinguishable if your circuit uses every edge; 7.8× better in infidelity if it can avoid one. Which device is better depends on your circuit's shape — precisely what Ch. 29 §29.2 engineers for, and precisely what a single averaged number cannot express.

Quantum Volume

The largest $2^n$ for which random square circuits ($n$ qubits, $n$ layers, random SU(4) on random pairs) beat a $2/3$ heavy-output threshold.

Virtues: holistic (gates, connectivity, routing, compilation, measurement together), hard to game, single ordered number.

Limits, both from "square":

  • It measures width and depth in a fixed ratio. Ch. 28's Grover circuit: 5 qubits, 257 gates. Ch. 29's ansatz: 6 qubits, 15 gates. Neither is square.
  • Random pairs sample the whole graph, including the parts a hardware-aware programmer avoids. A device with a few excellent qubits and many poor ones scores badly on QV and may run your circuit well.
  • It is quantized — moves in powers of two, so a real degradation may not register.

🗝️ Version Note. QV has been reported to $2^{20}$ and beyond, and vendors increasingly de-emphasize it for application benchmarks and CLOPS-style speed metrics — partly because it saturates, partly because square circuits stopped resembling anyone's workload.

XEB and the supremacy claims

Scores sampled output by cross-entropy against the simulated ideal distribution.

  • It requires simulating the thing claimed intractable. Verification happens where simulation is possible and the claim is extrapolated — which is why every supremacy claim has been followed by classical simulation work, some of it substantially narrowing the gap.
  • It benchmarks a random circuit — the most favourable case for a quantum device and the least for a classical simulator. Same objection as Ch. 21 §21.7 (Grover vs a strawman) and Ch. 24 §24.5 (QAOA vs Goemans–Williamson).

Not dishonest — it measures what it measures well. A poor guide to whether a device runs your program, because your program is not a random circuit.

The benchmark that answers your question

   RB / calibration data  -> planning, and choosing between devices
   Quantum Volume         -> a coarse ordering, holistically
   XEB                    -> research claims about random-circuit sampling
   YOUR CIRCUIT'S 1-TVD   -> whether your program will work

Run your circuit: simulate noiselessly for a reference, run on the device or its noise model, report $1-\text{TVD}$ with a standard error over transpiler seeds, repeat at several sizes.

When you cannot simulate the reference, fall back to Ch. 26 §26.8: verify classically. Ch. 23's Shor checks its factors; Ch. 24's QAOA checks its cut. A result you can verify is a result you can benchmark, at any size.

The protocol

   1. PULL THE FULL DISTRIBUTION and COUNT THE DEAD ELEMENTS.
   2. COMPUTE THE STATISTIC YOU NEED -- 5 edges of your chain, not 144 of the chip.
   3. USE THE MEDIAN FOR PLANNING. Stop the moment you pin a layout.
   4. CHECK READOUT SEPARATELY. RB cannot see it.
   5. TREAT QV AS A COARSE ORDERING.
   6. BENCHMARK YOUR CIRCUIT: 1-TVD, with an error bar, at several sizes.
   7. RE-PULL THE CALIBRATION. It is a timestamp, not a property.

Common pitfalls

  • Comparing two vendors' quoted fidelities without knowing how each was computed.
  • Reading a median as a property of the device rather than of a sampling process.
  • Using the median after pinning a layout.
  • Assuming a gate benchmark covers readout.
  • Treating Quantum Volume as predictive for a non-square circuit.
  • Tracking device benchmarks as a regression alarm — stable summaries do not move when something local breaks.
  • Forgetting that calibration data has a timestamp.

Project piece added this chapter

vqelab/benchmarking.py — ErrorDistribution with no .error attribute, reporting min/p25/median/mean/p95/max plus dead_count; quoted_fidelity requiring both the statistic and the dead-element decision as explicit arguments (no defaults, because both defaults are wrong for somebody); predict_survival carrying a trustworthy flag that is False when the layout was pinned; readout_summary, which exists because RB does not provide it; and fit_rb_decay / error_per_clifford, whose docstring states in capitals that it is an AVERAGE. 28 tests pass, including test_the_same_chip_supports_a_factor_of_nine_in_quoted_error, test_the_median_predicts_chapter_28s_MEASURED_result, test_a_pinned_layout_makes_the_prediction_UNTRUSTWORTHY, and test_rb_cannot_see_the_coin_flip_qubit.