Case Study 2: The Result That Was Not Significant

"The p-value answers a question. Make sure it is the one you asked."

Executive Summary

Two error mitigation techniques are compared on real hardware. Technique A gives 94.2% fidelity; technique B gives 96.1%. The write-up concludes that B is better and recommends it.

The data do not support that conclusion. This case study takes the analysis apart, shows that the difference is inside the noise, computes how many shots would have been needed to make the comparison, and then examines two further errors that are less obvious and more common: comparing against the wrong baseline, and looking at enough variants that something was bound to win.

Every failure here is one this book's own examples could have committed. That is the point — these are not exotic statistical sins, they are the default behavior of a careful person who has not thought about power.

Skills applied: sampling error (§5.4); expectation values and their uncertainties (§5.5); chi-squared testing and statistical power (§5.8); matching precision to systematic error (§5.9).

The Report

Comparing error mitigation strategies for a 4-qubit GHZ state

We prepared a 4-qubit GHZ state on ibm_example and measured the fidelity (fraction of shots returning 0000 or 1111) under two mitigation strategies.

Strategy	Shots	Fidelity
A: readout mitigation only	1,024	94.2%
B: readout mitigation + dynamical decoupling	1,024	96.1%

Strategy B outperformed A by 1.9 percentage points. We recommend B for production workloads.

Read it once. It looks like a competent small experiment. Now take it apart.

Problem 1: No Error Bars

The first thing missing is the only thing that would make the comparison interpretable.

At 1,024 shots, the standard error of a fidelity near 0.95 is

$$\sigma = \sqrt{\frac{p(1-p)}{N}} = \sqrt{\frac{0.95 \times 0.05}{1024}} = 0.0068$$

So the honest table is:

The difference is 1.9 points. The standard error of the difference is $\sqrt{0.0068^2 + 0.0060^2} = 0.0091$, so about 0.9 points.

$$\text{difference} = 1.9 \pm 0.9 \text{ percentage points}$$

That is about 2.1 standard errors — a two-sigma effect, p ≈ 0.04. It is suggestive. It is not the clean result the report claims, and reporting "96.1% vs 94.2%" with four significant figures and no uncertainty implies a precision that does not exist.

Rule: an estimate without an error bar is not a result. Four extra characters would have made this table honest.

Problem 2: A Single Run of Each

Worse than the missing error bar: each strategy was run once.

The error bar above accounts for sampling noise — the randomness of which 1,024 shots you happened to draw. It does not account for anything that varies between runs:

• Device calibration drifts over hours.
• The transpiler may choose different physical qubits (Chapter 4 Case Study 1 — a factor of eight lived there).
• Queue position determines when you ran, and the device at 3 a.m. is not the device at noon.

Those sources are often larger than the sampling error, and a single run of each cannot separate them from the effect you are trying to measure. If A ran an hour after a calibration and B ran just before one, the entire 1.9 points could be drift.

The fix is cheap: interleave. Run A, B, A, B, A, B in one session, five times each, and report the mean and the spread across repetitions. That controls for drift because both strategies experience the same drift.

results = {"A": [], "B": []}
for _ in range(5):
    for strategy in ("A", "B"):
        results[strategy].append(run_experiment(strategy, shots=1024))

for s, vals in results.items():
    mean, sd = np.mean(vals), np.std(vals, ddof=1)
    print(f"{s}: {mean:.4f} +/- {sd/np.sqrt(len(vals)):.4f}  (spread {sd:.4f})")

If the run-to-run spread turns out to be 2 percentage points, the 1.9-point difference is inside the noise of the measurement process itself, and no amount of extra shots within a single run would have revealed that.

Problem 3: How Many Shots Would Have Been Enough?

The question to ask before running.

To detect a difference $\delta$ between two proportions, with 95% confidence and reasonable power, you need roughly

$$N \approx \frac{2 \, z^2 \, p(1-p)}{\delta^2}$$

per condition. For $p \approx 0.95$ and $\delta = 0.019$:

$$N \approx \frac{2 \times 3.84 \times 0.0475}{0.019^2} \approx 1{,}010$$

So 1,024 shots was, in fact, almost exactly the right order of magnitude for detecting a 1.9-point difference — if there were no other noise sources.

That is a genuinely interesting finding, and it sharpens the criticism. The experiment was not under-powered against sampling noise; it was under-powered against run-to-run variation, which was never measured. Adding shots would not have helped. Adding repetitions would.

  to detect 1.9 points against sampling noise alone:  ~1,000 shots    (had it)
  to detect 1.9 points against run-to-run drift:      ~5-10 repetitions (did not)

Identify which noise source dominates before deciding what to buy more of. This is the same lesson as §5.9's step 5, in a different costume.

Problem 4: The Wrong Baseline

Neither strategy was compared to no mitigation at all.

Without that, the table cannot answer the question a reader actually has: is mitigation worth it? Suppose the unmitigated fidelity is 93.8%. Then:

Now the story is different and more useful. A is barely doing anything; B is doing real work. The comparison the report made (A vs B) obscured the comparison that mattered (is any of this worth the overhead?).

And there is a cost side the report omits entirely. Mitigation is not free — readout mitigation needs calibration circuits, dynamical decoupling adds pulses and therefore duration. A fidelity comparison without the overhead is only half the decision, and Chapter 13 §13.8 is entirely about that trade.

Problem 5: The Comparison You Did Not Report

The subtlest problem, and the most common.

If the team tried eight mitigation configurations and reported the two most interesting, the statistics are no longer what they appear. With eight comparisons, the chance that at least one shows a two-sigma difference by chance alone is

$$1 - (1 - 0.05)^8 \approx 34\%$$

A one-in-three chance of a spurious "significant" result, from a procedure that looks rigorous at every individual step.

This is the multiple-comparisons problem, and it does not require any dishonesty — it is what happens when you explore your data and then report the interesting part, which is the most natural thing in the world to do.

Two defenses. Say how many configurations you tried, and correct for it (Bonferroni: divide your threshold by the number of comparisons, so 0.05 becomes 0.006). Or better: use the exploratory run to form a hypothesis, then test it on fresh data. The second is stronger and it is cheap here — another run costs minutes.

🔬 Honest Assessment — What the report should have said.

We compared two error mitigation strategies on a 4-qubit GHZ state, interleaved across five repetitions to control for calibration drift, at 4,096 shots each, against an unmitigated baseline. We tried three configurations in total and report all three.

Strategy	Fidelity	Gain	Overhead
none	93.8% ± 0.4%	—	1×
A: readout only	94.2% ± 0.4%	+0.4 ± 0.6 pts	1.2× circuits
B: readout + DD	96.1% ± 0.4%	+2.3 ± 0.6 pts	1.2× circuits, 1.4× duration

B improves fidelity by 2.3 ± 0.6 points over no mitigation (p < 0.001). A's improvement of 0.4 ± 0.6 points is not distinguishable from zero with this data. The B-versus-A difference is 1.9 ± 0.6 points.

Run-to-run spread was 0.8 points, comparable to the sampling error, so both were controlled by interleaving. These results are specific to this device, this layout (qubits 12, 13, 14, 15), and this calibration window.

Longer, more caveated, and far more useful. It tells a reader what to do, what it costs, what the uncertainty is, and where the conclusion stops applying. It also honestly reports that strategy A did nothing — which the original write-up quietly obscured by never testing against no mitigation.

Lessons

1. An estimate without an error bar is not a result. $\sqrt{p(1-p)/N}$ takes one line.
2. The error bar on a difference combines both: $\sqrt{\sigma_A^2 + \sigma_B^2}$.
3. Sampling error is not the only error. Calibration drift, layout changes, and time of day are often larger, and a single run of each cannot separate them.
4. Interleave and repeat. A, B, A, B, A, B controls for drift almost for free.
5. Work out what dominates before buying more. Here, more shots would not have helped; more repetitions would.
6. Always include the do-nothing baseline. Without it you cannot answer whether any of this is worth doing.
7. Report cost alongside benefit. Fidelity without overhead is half a decision.
8. Say how many things you tried. Eight comparisons make a one-in-three chance of a spurious two-sigma result, with no dishonesty required.
9. Confirm an exploratory finding on fresh data. It is the cheapest way to earn a strong claim.
10. State where the conclusion stops applying — this device, this layout, this calibration window.

Questions

1. Recompute the error bars if both strategies had used 16,384 shots instead of 1,024. Does the 1.9 point difference become significant against sampling noise? Does that resolve the study's actual problem?

2. Write compare_proportions(k1, n1, k2, n2) returning the difference, its standard error, and a p-value. Test it on the case study's numbers, then on a case where the difference is genuinely large.

3. The multiple-comparisons calculation used $1 - 0.95^8$. Redo it for 3, 5, and 20 comparisons. At what number does a spurious two-sigma result become more likely than not?

4. Design the interleaved experiment concretely: how many repetitions, how many shots each, in what order, and what would you do if the run-to-run spread turned out to be 5 percentage points?

5. The rewritten report says strategy A's improvement is "not distinguishable from zero." Is that the same as saying A does not work? Explain the difference, and say what experiment would settle it.

6. The report's fidelity metric counts only 0000 and 1111. Chapter 4 §4.9 warned that this metric cannot detect loss of coherence. Propose a better metric for comparing mitigation strategies, and say what it costs in additional measurements.

7. Hardest. Suppose the team runs the interleaved experiment and finds B better by 2.3 ± 0.6 points on this device and layout. How far does that conclusion generalize — to another layout on the same chip, to another chip of the same generation, to a trapped-ion device? Design the smallest experiment that would justify each broader claim, and say which of them you think is actually worth running.