Case Study 39.2: The Result That Would Not Come Back

The situation

A university group publishes a preprint reporting an error-mitigation technique that improves expectation value accuracy by 34% on a 12-qubit observable, run on a public 127-qubit device. The code is on GitHub, the notebook runs, the plots regenerate.

Four months later a second group tries to reproduce it and measures a 7% improvement. They write to the authors.

The authors re-run their own notebook on the same backend. They get 11%.

Nobody has done anything wrong, and nothing in either codebase explains the discrepancy.

Step 1: what the repository does not contain

The first group's repository has the circuit construction, the mitigation implementation, the analysis, a pinned requirements.txt, and a fixed random seed. By classical software standards it is a good reproducibility package.

>>> from vqelab.platform import ExecutionRecord
>>> record = ExecutionRecord(backend="ibm_device", shots=4096,
...                          optimization_level=1, seed_transpiler=42,
...                          qiskit_version="2.1.0")
>>> record.is_reproducible
False
>>> record.missing_provider_fields
('job_id', 'execution_timestamp', 'calibration_snapshot', 'physical_qubits')

Everything the repository captures is in the first five fields. Everything that varies is in the last four, and all four come from the provider at execution time.

The pinned requirements.txt and the fixed seed create a strong impression of determinism. They pin everything on the client side, which is the side that was never the problem.

Step 2: how much the missing fields are worth

The second group runs the diagnostic the first group did not.

Calibration spread on the device:

   cz         n= 265  min 1.79e-03   median 3.66e-03   max 1.00e+00
   T1 (us)    n= 133  min 15.2       median 174.9      max 483.0

Some links are dead (error 1.00). T1 varies by a factor of 32 across the chip. Which physical qubits you receive is a first-order determinant of the result — and the platform assigns them at execution time.

Layout roulette on a comparable circuit:

   worst  fidelity 0.5755   seed 20   109 2q gates   depth 247
   median fidelity 0.6396
   best   fidelity 0.7911   seed 23    49 2q gates   depth 149
   error ratio worst/best: 2.03x

Twice the error from the transpiler seed alone, driven by a 49-to-112 spread in two-qubit gate count. The first group did fix their seed — but a fixed seed produces a fixed layout only against a fixed backend configuration. When the device's calibration changes, optimization_level=1 routes differently for the same seed, because the cost function it minimizes has changed.

So the fixed seed did not fix the layout. It fixed one input to a function whose other inputs move.

Step 3: reconstructing what happened

With the device's calibration history the second group reconstructs it. Between the original run and the reproduction attempts:

• The device was recalibrated many times, as scheduled.
• Two qubits in the original layout's neighbourhood had degraded, and the transpiler routed around them.
• The mitigation technique's benefit depends on the error profile it is correcting — it helps most when errors are large and structured, less when they are small or when routing has already spread the circuit across better hardware.

The 34% was real. So was the 11%, and the 7%. They are three measurements of a quantity that moves, reported as though it were a constant.

The finding is not that the first group's result was wrong. It is that the paper reported a point estimate of something that varies substantially, without the metadata that would let anyone determine which point they had sampled.

Step 4: what the paper should have reported

>>> record.job_id = "cx7k2p9"
>>> record.execution_timestamp = "2026-04-02T14:22:11Z"
>>> record.calibration_snapshot = "2026-04-02T06:00:00Z"
>>> record.physical_qubits = (30, 31, 32, 33, 37, 52, 51, 50)
>>> record.is_reproducible
True
>>> record.describe()
'ibm_device job cx7k2p9 at 2026-04-02T14:22:11Z, 4,096 shots on qubits
(30, 31, 32, 33, 37, 52, 51, 50) -- re-runnable'

Four fields. A dictionary written alongside every result.

And one methodological change that matters more: repeat the measurement across calibration cycles and report the distribution. "34% (n=1)" becomes "18% ± 9% across 12 runs over three weeks, range 7–34%" — which is a weaker headline and a claim that survives contact with a second group.

Step 5: the correction

The groups publish jointly. The revised claim:

The technique improves expectation-value accuracy by 18% on average (range 7–34%) across 12 executions spanning three weeks, with the benefit strongly dependent on the error profile of the assigned physical qubits. Improvement correlates with baseline two-qubit error rate on the assigned layout ($r = 0.71$), which is consistent with the mechanism: the technique corrects structured error, so it helps most where there is most to correct.

That last sentence is the scientific contribution, and it was not visible from a single run. The original point estimate concealed the mechanism it was evidence for.

What this case study is about

This is the book's seventh instance of a measurement being too small, and it is the one where "too small" does not mean "too few samples."

The first group sampled once from a distribution whose width they had no reason to suspect, because classical software reproducibility habits actively concealed it. A pinned environment and a fixed seed are the right instincts, and they pin the client side of a computation whose variance lives on the server side.

The general habit: on hardware, "I fixed the seed" is not the same claim as "the experiment is determined." Ask what else is an input, and whether you control it.

Questions

1. The first group fixed their transpiler seed and still got a different layout. Explain the mechanism precisely.
2. [measure] Reproduce the layout sweep on a 14-qubit circuit. Then do it on a 4-qubit circuit and explain why the second tells you nothing.
3. The revised claim reports $r = 0.71$ between improvement and baseline error rate. Why is that correlation more scientifically valuable than a larger headline number?
4. Design a reproducibility standard for hardware results that a journal could actually enforce. What is the minimum, and what would you like but cannot require?
5. The hard one. The second group's 7% and the authors' 11% are also single measurements. Are they making the same error? If so, does that change the case study's conclusion — and what should the second group have done before writing to the authors?