Case Study 2: The Benchmark That Could Not See the Failure

Clean benchmarks, dead results

A team runs a weekly regression: the same 8-qubit circuit, the same analysis, tracked over time. It has been stable for months. Then one week the results collapse — output fidelity drops from around 0.85 to 0.31, and stays there.

Their first move is correct: check whether the device changed. They pull the gate benchmarks.

   randomized benchmarking, this week vs last week:
     1-qubit error per Clifford:   0.00024  ->  0.00025
     2-qubit error per Clifford:   0.00751  ->  0.00758
   Quantum Volume:                     128  ->      128

Nothing moved. Gate fidelity is unchanged to three significant figures. Quantum Volume is identical.

So the device is fine, and the problem must be theirs. They spend two weeks auditing their own code — re-running Chapter 26's bisection, Chapter 27's property suite, Chapter 28's equivalence checks. Every test passes. The circuit is correct, the transpilation is verified, the unitary is preserved to $10^{-12}$.

The device really had changed. The benchmark could not see it.

What RB is blind to

   readout error: median 0.0198, mean 0.0415, max 0.5000
   12 of 127 qubits have readout error above 10%
   1 qubit at 0.5000 -- A COIN FLIP

A qubit whose measurement is a coin flip carries no information at all. Their circuit measured eight qubits; the transpiler's layout had shifted onto one of the bad ones after a recalibration, and the weekly regression pinned no layout, so it followed.

And randomized benchmarking cannot see any of this, by design.

RB works by applying $m$ random Cliffords and then the single Clifford inverting their product, then fitting $P(0) = A p^m + B$. State-preparation and measurement error change the amplitude $A$ and the offset $B$. The fit reports the exponent $p$. SPAM cancels.

That property is advertised as a feature, and it is one: it is why RB gives a clean gate-quality number independent of how good your readout happens to be. It is also exactly why RB was silent here.

⚛️ A benchmark that is robust to a class of errors is, from your circuit's perspective, blind to them.

Robustness and blindness are the same property described from two directions. Every benchmark that isolates one quantity does so by making itself insensitive to the others — and your circuit is insensitive to nothing.

The full list

RB averages away more than SPAM:

   readout / SPAM error     cancels in the fit BY DESIGN
   coherent errors          the Clifford twirl converts them to a depolarizing rate
   crosstalk                standard RB benchmarks qubits in isolation
   gate-dependent errors    error per CLIFFORD mixes however many gates each uses
   per-qubit structure      one number for a distribution spanning 34x
   drift                    the number is from whenever calibration last ran

The twirl deserves emphasis because it is the mechanism, not a flaw. Twirling over the Clifford group converts an arbitrary noise channel into a depolarizing channel with the same average fidelity. That is what makes a single exponential fit valid regardless of what the noise actually is — and it is what discards the structure.

You cannot have both. A benchmark whose answer does not depend on the noise model cannot tell you about the noise model.

Quantum Volume was silent for a different reason

QV did not move either, and that is also expected.

QV tests random square circuits — $n$ qubits, $n$ layers, random SU(4) on random pairs — and scores against a heavy-output threshold. It does include measurement, unlike RB. But:

It uses random pairs across the whole chip, so one bad readout qubit among 127 barely shifts the aggregate. QV is a coarse, holistic, single-integer summary; it is designed to be robust to exactly the kind of local variation that destroyed this team's circuit.

And it is quantized. QV moves in powers of two. A degradation that halves your circuit's fidelity may not be enough to drop $2^7$ to $2^6$.

A benchmark designed to be a stable summary of a whole device is, necessarily, insensitive to anything local. That is what "stable summary" means.

What they should have been tracking

Not the device benchmarks. Their circuit.

   1. Simulate the circuit noiselessly -> the reference distribution.
   2. Run it on the device (or a noise model built from it).
   3. Report 1 - TVD, with a standard error over transpiler seeds.
   4. Track THAT number weekly.

This is what Chapters 27, 28 and 29 all did, and it would have flagged the regression in the same week it happened — because it is a measurement of the thing that broke.

Alongside it, two cheap device-side checks that RB does not provide:

   * readout error on the qubits your transpiled circuit ACTUALLY USES
     (t.layout.final_index_layout(), then look them up in the Target)
   * the number of DEAD elements -- 9 of 144 edges here

Both are a handful of lines against the Target, and both would have named the failure immediately.

The deeper pattern

This is the fifth time in six chapters that the finding has this shape.

  • Chapter 25: a QEC test storing an eigenstate of its own failure mode reported zero error.
  • Chapter 26: a bisection whose default input could not see the bug reported no divergence.
  • Chapter 27: three of four oracle-free properties passed a broken circuit; a distribution test loosened until it stopped flaking became 0% sensitive.
  • Chapter 29: a layout chosen by connectivity alone routed through two dead edges.
  • Chapter 30: a benchmark robust to readout error was silent when readout error broke everything.

The recurring failure in this book is not wrong code. It is measurements that cannot detect the thing that is wrong — and in every case the blindness was a documented, deliberate property of the method, sitting in plain sight in its own description.

The lessons

Track your circuit, not the device. Device benchmarks answer questions about devices. You have a question about a program.

Read what a benchmark is insensitive to — it is always stated, usually as a selling point. "SPAM robust" and "cannot see your readout error" are the same sentence.

Check readout separately. It is the largest error channel on this device (median 0.0198 against 0.0075 for two-qubit gates), it has the widest spread (171×), and the standard gate benchmark is designed not to notice it.

And a stable benchmark is a poor alarm. QV and RB are built to summarize a whole device robustly. Robust summaries do not move when something local breaks — which is precisely when you want an alarm.

Reproduce it: code/example-01-the-distribution-not-the-median.py prints the readout distribution and the coin-flip qubit; code/example-02-randomized-benchmarking.py implements RB, validates it against a known error, and enumerates what it averages away. readout_summary in code/vqelab/benchmarking.py exists because RB does not provide it, and test_rb_cannot_see_the_coin_flip_qubit asserts the 12-of-127 and the 0.5000.