Case Study 1: The Device That Was Better on Paper
The procurement decision
A research group has budget for time on one of two quantum devices. They do the responsible thing and compare published specifications.
Device A Device B
qubits 127 133
two-qubit fidelity 99.25% 99.20%
Quantum Volume 128 256
Device B looks better. Marginally worse on gate fidelity, meaningfully better on Quantum Volume, slightly more qubits. They book time on B.
Their workload is Chapter 29's hardware-aware ansatz: a 6-qubit variational circuit with a linear entanglement pattern, 15 two-qubit gates, designed specifically to use a short chain of good qubits.
It runs badly.
Two numbers that were not comparable
The published two-qubit fidelities were computed differently, and neither vendor said so.
Here is what one chip's calibration record actually supports:
statistic value implied 100-gate survival
median, dead edges EXCLUDED 0.00750 0.4710
mean, dead edges excluded 0.01018 0.3593
median, dead edges INCLUDED 0.00779 0.4576
mean, dead edges INCLUDED 0.07205 0.0006
p95, dead excluded 0.01999 0.1328
worst live edge 0.11736 0.0000
The same chip, on the same day, supports "two-qubit error 0.00750" and "two-qubit error 0.07205." A factor of 9.6, and the implied survival of a 100-gate circuit ranges from 47% to 0.06%.
Nobody is lying. Excluding uncalibrated edges is defensible — you would not route through them anyway. Reporting a median rather than a mean is defensible — it resists outliers. Every choice is defensible and the combination spans an order of magnitude.
⚠️ A quoted device fidelity is a statistic, and the choice of statistic is rarely published.
Before comparing two devices, ask: median or mean? dead elements included or excluded? which gate? averaged over all qubits, or over the best subset?
Two vendors making different defensible choices produce numbers that cannot be compared at all — and 99.25% against 99.20% is well inside the range those choices span.
Quantum Volume measured the wrong shape
Device B's QV advantage was real, and irrelevant.
QV tests random square circuits: $n$ qubits, $n$ layers, random SU(4) on random pairs. It is holistic and hard to game, which are genuine virtues. But:
It measures width and depth in a fixed ratio. The group's circuit is 6 qubits and 15 two-qubit gates — shallow and narrow. Chapter 28's Grover circuit was 5 qubits and 257 gates — deep and narrow. Neither is square, and QV says nothing about either.
And it uses random pairs, so it samples the entire connectivity graph — including the parts a hardware-aware programmer deliberately avoids. Chapter 29 §29.2 measured that a well-designed circuit uses a short chain of good qubits and never touches the rest of the chip.
A device with a handful of excellent qubits and many poor ones scores badly on QV and may run this group's circuit very well. Device B's higher QV meant it was more uniformly decent. Device A had a better chain.
Same average, different device
The general form of the problem, with numbers:
device mean median worst
uniform 0.0080 0.0080 0.0080
skewed 0.0080 0.0010 0.0710
device survival using ALL ten edges avoiding the WORST edge
uniform 0.9228 0.9303
skewed 0.9207 0.9910
Identical mean error. Using every edge, the two devices are indistinguishable — 0.9228 against 0.9207. Avoiding one bad edge, the skewed device is 7.8× better in infidelity.
Which device is better depends entirely on whether your circuit can route around the bad part. A single averaged number cannot express that, and randomized benchmarking — the source of most quoted gate fidelities — reports a single averaged number by construction: the Clifford twirl converts whatever the noise actually is into a depolarizing channel of the same average fidelity.
That averaging is not a limitation of the analysis. It is the mechanism that makes the exponential fit valid.
What they should have done
Pull the full distribution, not the summary. It is public and one API call away:
two-qubit (ecr): 144 entries, 9 DEAD (error = 1.0)
min 0.00347 p25 0.00567 median 0.00750 p95 0.01999 max 0.11736
Compute the statistic that matches their circuit. Their circuit uses a 6-qubit chain — five edges. The relevant number is not the median over 144 edges; it is the survival product over the best available chain:
best 6-chain on this device: survival 0.9764 over five edges
That is a number about their workload, and it can be computed for both devices before booking anything.
And run the circuit. Both vendors expose noise models built from real calibration data. Chapter 29's entire measurement — six configurations, five seeds each — takes minutes on a laptop and answers the actual question.
The good news, which is also a warning
The published median was not useless. Chapter 30 §30.4 measured that the median predicted Chapter 28's circuit fidelity to within 12%: $(1-0.00750)^{257} = 0.1445$ against a measured 0.1290.
It works because the transpiler picks good qubits — VF2Layout scores embeddings against the full
error record, so a transpiled circuit lands on the better part of the distribution. The median
describes the qubits you get, not the qubits that exist.
A summary statistic is a description of a sampling process. Change how the sampling happens and the statistic stops describing anything — which is exactly what Chapter 29 §29.4 measured when a hand-pinned layout hit two dead edges and returned 0.6790 against an expected 0.9116.
So the median is a good planning number and a bad comparison number, for the same reason: it depends on a selection process that both devices perform differently.
The lessons
Two quoted fidelities are not comparable unless you know how each was computed. The choices span an order of magnitude on a single chip.
Quantum Volume is a square-circuit benchmark. Your circuit is not square, and QV samples parts of the device your circuit will never touch.
Compute the statistic that matches your workload. Five edges of a chosen chain, not 144 edges of a whole chip.
And benchmark your circuit. Every vendor ships a noise model. The measurement that answers your question takes minutes and is the only one that does.
Reproduce it: code/example-01-the-distribution-not-the-median.py prints all six statistics and
their implied survivals; quoted_fidelity in code/vqelab/benchmarking.py requires both the
statistic and the dead-element decision as explicit arguments, and
test_the_same_chip_supports_a_factor_of_nine_in_quoted_error asserts the 9.6× span.