Case Study 1: The 9% Improvement That Wasn't

The optimization

A team is preparing a Grover-like search for a hardware run. It is the largest circuit they have attempted, and every gate matters — Chapter 12 measured a median two-qubit error around $8 \times 10^{-3}$, so a circuit with 273 of them is fighting a survival probability of about $(1 - 0.0078)^{273} \approx 0.12$ before anything else goes wrong.

So they tune the transpiler. Level 1 is the default in their tooling; they try level 2:

   level 1:  273 two-qubit gates
   level 2:  257 two-qubit gates

A 9% reduction in the dominant error source. They switch to level 2, note the improvement in the sprint review, and move on. It is a small, sensible, entirely defensible change.

What they did not look at

The other column.

   level 1:  depth  931   2q 273
   level 2:  depth 1080   2q 257

Level 2's circuit is 16% deeper.

That is not a regression and not a bug. Depth and two-qubit count are both standard optimization objectives, and the transpiler resolves them differently at different levels. Level 2's TwoQubitPeepholeOptimization and CommutativeCancellation find gate reductions that cost serialization.

And the two objectives govern different physics:

   DEPTH drives DECOHERENCE      -- the circuit must finish inside T1 and T2
   2Q COUNT drives GATE ERROR    -- Chapter 12's median ecr error, 288x spread

Both are real. On this circuit they point in opposite directions. The team optimized one of them and reported the result as an improvement, without checking the other or measuring the outcome.

The measurement

Run both circuits on a noise model built from the actual backend. Eight transpiler seeds, 20,000 shots each, tracking the probability of the correct answer:

    level   depth mean   ecr mean   P(top) mean      std     best
        1        956.4      279.8        0.0917   0.0119   0.1031
        2       1098.6      254.0        0.0945   0.0140   0.1072

    difference (L2 - L1) = +0.0028 +/- 0.0065     NOT SIGNIFICANT at 2 sigma

The 9% gate-count reduction bought a fidelity change of $+0.0028 \pm 0.0065$ — a mean in the right direction, an error bar twice its size, and no result.

Level 2 is not worse. It is not better either. The depth increase and the gate-count decrease cancel, which is exactly what you would expect from two mechanisms of comparable magnitude pulling opposite ways — and exactly what nobody checks.

🔬 Honest Assessment: a gate-count improvement is not a performance claim.

"Level 2 reduced two-qubit gates by 9%" is true. It is also what almost every optimization report in the field says, and on this circuit it corresponds to no measurable change in the answer.

Optimizing a proxy is useful exactly insofar as the proxy predicts the outcome. Here two standard proxies predicted opposite outcomes, which means at most one of them was predictive and neither was checked.

What they should have reported

Not "we improved gate count by 9%." Either:

"Level 2 gave $+0.0028 \pm 0.0065$ in output fidelity — no measurable change," which is a real result and takes ten minutes of simulator time, or

"We switched to level 2 for a 9% gate-count reduction; we did not measure the effect on output," which is honest and lets the reader weigh it correctly.

The first costs almost nothing. The circuit was already going to be simulated for validation; running it twice under two transpiler settings is a loop.

The finding they missed

The same experiment contains a result worth reporting, and it is much larger:

   level 0: P(11111) = 0.0390  vs noiseless 0.6027  ->  6.5% of the signal retained
   level 1: P(11111) = 0.0779  vs noiseless 0.6027  -> 12.9% of the signal retained
   level 2: P(11111) = 0.0778  vs noiseless 0.6027  -> 12.9% of the signal retained

Level 0 to level 1 doubles the surviving signal, for milliseconds of transpiler time. That is unambiguous, large, and free.

The team's tooling already defaulted to level 1, so they never saw it — they spent their attention on the margin above a good default rather than on confirming the default was good. The big win is usually the one you inherited, and it is worth measuring once so you know how much you are standing on.

And the number nobody quotes

12.9%.

Even at the best setting, this circuit returns about an eighth of its noiseless signal. Optimization is real and worth doing; it does not rescue the circuit. Chapter 25 §25.9 explains why — the device is on the wrong side of the error-correction threshold, and no amount of gate-count tuning moves a threshold.

Optimization is a constant-factor improvement to an exponentially decaying quantity. Worth having. Not a strategy.

The lessons

Check the other column. Optimization has multiple objectives and they are not aligned. A report that quotes one metric has, by construction, not looked at whether the others moved against it.

Measure the outcome, not the proxy. The fidelity measurement here is a loop over eight seeds. If you cannot afford it, say you did not do it.

Report an error bar. $+0.0028$ and $+0.0028 \pm 0.0065$ are different claims, and only one of them is a claim. This is Chapter 27 §27.5's discipline applied to optimization rather than to testing, and Chapter 24 §24.3's before that.

Verify your default before optimizing past it. The level 0 → 1 result was ten times larger than anything the team was investigating, and it was already switched on.

And keep the absolute number in view. A 9% relative improvement to a quantity sitting at 12.9% of its ideal is not where the problem is.

Reproduce it: code/example-03-does-it-actually-help.py runs the eight-seed comparison and prints the significance verdict; OptimizationReport.better_than in code/vqelab/optimization.py raises rather than ranking circuits without measured fidelity, and test_a_difference_inside_the_error_bars_is_not_better asserts that a higher mean is not enough.