Case Study: Reading a Benchmark — Quantum Volume and Its Discontents
Executive Summary
Two vendors publish results in the same week. Vendor A: 1,121 qubits. Vendor B: quantum volume $2^{20}$ on 32 qubits. Which machine is better?
The question has no unqualified answer, and understanding why is the point. Benchmarks encode a definition of "better," and every definition is partial. This case study takes apart quantum volume — currently the most widely cited single-number benchmark — establishes what it measures, demonstrates concretely how it can be inflated without improving general capability, and assembles the small set of numbers you would actually want before choosing a device.
Skills applied
- Interpreting quantum volume's definition and its implicit weighting (§18.16).
- Recognizing benchmark-specific optimization.
- Comparing holistic benchmarks against component metrics.
- Selecting metrics appropriate to a stated workload.
Phase 1: What quantum volume measures
QV is the largest $2^m$ such that the device successfully runs random square circuits on $m$ qubits with $m$ layers. "Successfully" means the heavy-output probability exceeds 2/3 with statistical confidence.
Two features follow from the definition:
It is holistic. Qubit count, gate fidelity, connectivity, and compiler quality all affect the score. A device with many poor qubits scores no better than one with few excellent ones — a genuine improvement over raw qubit counts.
It is square. Width equals depth. A device with 1,000 qubits and depth-10 coherence scores $2^{10}$ at best, no matter how many qubits it has. This is deliberate: the benchmark encodes the view that usable computation needs both dimensions.
Phase 2: The saturation problem
QV is exponential in $m$, so each doubling requires one more qubit and one more layer of depth simultaneously. As $m$ grows, the random circuit contains $\sim m^2/2$ two-qubit gates, and the required fidelity grows accordingly:
| $m$ | QV | 2q gates | Required per-gate fidelity |
|---|---|---|---|
| 5 | 32 | ~12 | ~0.97 |
| 10 | 1,024 | ~50 | ~0.993 |
| 15 | 32,768 | ~112 | ~0.997 |
| 20 | $10^{6}$ | ~200 | ~0.9985 |
| 30 | $10^{9}$ | ~450 | ~0.9994 |
| 50 | $10^{15}$ | ~1,250 | ~0.99985 |
Beyond $m \approx 30$ the fidelity requirement exceeds anything achievable without error correction. QV saturates before it reaches interesting scales, which is why the metric's proponents have largely moved on to other benchmarks and why very high QV numbers come from small, excellent devices rather than large ones.
Phase 3: How a score can be inflated
QV is defined on random circuits, but the compiler is allowed to optimize them. That opens three legitimate-but-distorting avenues:
1. Compiler specialization. QV circuits have a known structure — random SU(4) blocks on random qubit pairs, layer by layer. A compiler tuned for that structure finds better decompositions than a general-purpose one. The improvement is real for QV circuits and does not transfer to arbitrary workloads.
2. Qubit selection. The benchmark uses $m$ qubits; a 100-qubit device picks its best $m$. A device with 5 excellent qubits and 95 poor ones posts the same QV as a uniformly excellent 5-qubit device — and a far better score than a uniformly good 100-qubit device would suggest is warranted for its usable width.
3. Heavy postprocessing. The 2/3 threshold is on heavy-output probability, and permissible error mitigation shifts that. Different vendors have applied different amounts.
None of this is cheating; all of it is optimizing for the metric. The general lesson — a benchmark that becomes a target stops measuring what it measured — is not specific to quantum computing.
Phase 4: The numbers you actually want
For any real decision, ask for the components rather than the composite:
| Metric | Why it matters | Typical good value (2026) |
|---|---|---|
| Two-qubit gate error (median and worst) | Dominates circuit fidelity | $10^{-3}$ median |
| $T_1$, $T_2$ (median) | Sets depth budget in time | 100+ μs |
| Readout error | Often the largest single error | $10^{-2}$ |
| Gate time | With $T_2$, gives depth in layers | 300 ns 2q |
| Connectivity (degree, diameter) | Determines routing overhead | Higher degree better |
| Usable qubit count | Qubits meeting a fidelity bar | — |
| Mid-circuit measurement + reset | Required for dynamic circuits | Yes/no |
| Calibration cadence and drift | Reproducibility | — |
The median-versus-worst distinction matters more than it appears: a circuit touching all $n$ qubits is limited by the worst one it uses, not the median. Vendors quote medians.
Phase 5: Answering the original question
Vendor A (1,121 qubits) versus Vendor B (QV $2^{20}$ on 32 qubits):
- For a shallow, wide circuit — a large GHZ state, a sampling demonstration, a hardware-efficient ansatz spanning many qubits — Vendor A may genuinely be better, if its qubits are usable.
- For anything requiring depth — the overwhelming majority of algorithms — Vendor B is better by a wide margin. Depth is the binding constraint.
- For error-correction research — Vendor A, because surface-code experiments need many physical qubits and tolerate moderate per-qubit fidelity.
The honest answer: the question is malformed. "Better" is defined by workload, which is exactly what a single-number benchmark cannot capture.
Phase 6: A benchmarking protocol
- State the workload first. Depth-limited or width-limited? What observable, what precision?
- Get component metrics, including worst-case not just median.
- Run an application-proxy benchmark — a small instance of your actual algorithm — rather than trusting a synthetic score.
- Include mirror circuits or randomized benchmarking to separate coherent from incoherent error.
- Repeat across calibration cycles to capture drift (Chapter 8).
- Report the raw data, so that others can re-evaluate under their own definition of better.
Discussion Questions
- QV is square by design. Argue for and against that choice as a measure of capability.
- Compiler specialization improves QV without improving general capability. Is that a flaw in the benchmark or in how it is reported?
- Vendors quote median error rates; circuits are limited by the worst qubit used. Design a metric that captures this.
- QV saturates around $m \approx 30$. What should replace it, and what would that successor's failure mode be?
Your Turn: Extensions
- Estimate the QV of a device from its published error rates and compare to its claimed score.
- Compute the required per-gate fidelity for QV $2^{25}$ and check it against current hardware.
- Design an application-proxy benchmark for a workload you care about.
- Compare a device's median and worst two-qubit error and compute the fidelity impact on a circuit spanning all qubits.
Key Takeaways
- Quantum volume is holistic — it weighs qubits, fidelity, connectivity, and compiler together — which makes it better than raw qubit counts and still insufficient alone.
- Its square definition means depth-limited devices cannot hide behind qubit count, and it saturates near $m \approx 30$ before reaching useful scales.
- Scores can be inflated through compiler specialization, best-qubit selection, and mitigation, none of which are cheating and all of which reduce transferability.
- Ask for component metrics — especially worst-case two-qubit error — plus an application-proxy benchmark on your actual workload.
- "Which machine is better" is only answerable relative to a stated workload; any single-number answer has smuggled one in.