Case Study 2: The Result That Could Not Be Reproduced

"Same code. Same device. Same afternoon. Different answer."

Executive Summary

A group reports a hardware benchmark. Three months later a colleague reruns the identical script on the identical device and gets a materially different number. Nothing changed — not the code, not the Qiskit version, not the backend.

The script did not set seed_transpiler. SABRE, the layout and routing heuristic, is randomized, so each run compiled the circuit differently: different physical qubits, different gate counts, different fidelity. The reported number was one sample from a distribution nobody knew they were sampling from.

This case study measures the distribution, traces how the variation propagates from seed to layout to gate count to fidelity, and derives the two-line fix — plus a third line that turns the problem into an optimization.

Skills applied: the seed (§10.9); layout and routing methods (§10.6); the layout-quality result (Ch. 4 Case Study 1); reproducibility discipline (Ch. 5 Case Study 2).

The Distribution Nobody Measured

The script transpiles a routing-limited circuit at optimization level 3 and runs it. Here is what happens across eight seeds:

    seed | 2q gates | depth | layout
  ---------------------------------------------------
       2 |       18 |    66 | [58, 61, 59, 53, 60]
       6 |       18 |    66 | [58, 61, 59, 53, 60]
       7 |       18 |    66 | [58, 61, 59, 53, 60]
       5 |       18 |    69 | [58, 61, 59, 53, 60]
       3 |       19 |    73 | [58, 61, 59, 53, 60]
       0 |       20 |    74 | [58, 53, 61, 60, 59]
       1 |       20 |    74 | [58, 53, 61, 60, 59]
       4 |       21 |    77 | [58, 59, 53, 61, 60]

Two-qubit counts from 18 to 21 — a 17% spread — across three distinct layouts. The original run happened to draw a seed near one end; the reproduction drew one near the other.

How the Variation Propagates

The 17% spread in gate count is not where the story ends. It is where it begins, because each step amplifies.

Seed → layout. SABRE's search is randomized, so different seeds settle on different physical qubits. Above, three distinct layouts across eight seeds.

Layout → gate count. Different physical qubits have different neighbors, so routing costs differ. 18 to 21 gates here.

Layout → error rate. This is the amplifier, and it is Chapter 4's Case Study 1: physical qubits differ enormously in quality. That case study measured an eightfold difference in the correctness of a GHZ result from layout alone — and found a completely dead qubit on the same chip.

Gate count and error rate → fidelity. Both feed in multiplicatively.

So a 17% spread in the compiler's output can become a much larger spread in the experimental result, depending on whether a given seed happened to route through good qubits or bad ones.

⚠️ Common Pitfall — "I ran it several times and got consistent results."

This is the reassurance that makes the problem invisible, and it is worth taking apart.

Running the same script several times in one session usually gives consistent results, because many workflows transpile once and reuse the compiled circuit. The sampling varies; the compilation does not. So you observe shot noise, conclude the result is stable, and never see the larger variation sitting one level up.

The compilation variation only appears when something re-triggers transpilation: a fresh session, a different machine, an edited script, a library upgrade. Which is to say — exactly when someone tries to reproduce your work.

Repeating an experiment is not the same as reproducing it. Repetition varies the shots; reproduction varies everything you did not pin down.

The Fix

Two lines, both free.

pm = generate_preset_pass_manager(optimization_level=3, backend=backend,
                                  seed_transpiler=42)          # 1. PIN IT
isa = pm.run(circuit)
print(isa.layout.final_index_layout())                          # 2. RECORD IT

And the third line, which turns the problem into an advantage:

# 3. SAMPLE IT DELIBERATELY, and keep the best
best = min((pm_with_seed(s).run(circuit) for s in range(8)),
           key=lambda c: (two_qubit_count(c), c.depth()))

   n_seeds | best 2q | depth |  seed |     time
  ----------------------------------------------
         1 |      20 |    74 |     0 |     8 ms
         2 |      20 |    74 |     0 |    18 ms
         4 |      18 |    66 |     2 |    43 ms
         8 |      18 |    66 |     2 |    74 ms
        16 |      18 |    66 |     2 |   147 ms

Four seeds: 10% fewer gates and 11% less depth, for 35 extra milliseconds. Beyond four, nothing more on this circuit — which is itself worth knowing, and is a measurement rather than a guess.

Note what best-of-N does to reproducibility: it makes the result deterministic given N, because you have replaced "whatever seed I happened to get" with "the best of seeds 0 through N−1." Record N and the winning seed and the compilation is fully specified.

What a Reproducible Record Looks Like

The minimum for a hardware result someone else can regenerate:

The project's Compilation record (this chapter's checkpoint) captures the middle four automatically:

opt=3 seed=2 2q=18 depth=66 qubits=[58, 61, 59, 53, 60]

One line, attached to every result.

🔬 Honest Assessment — How common is this?

Common enough that you should assume it unless told otherwise.

seed_transpiler is optional, it defaults to unset, and nothing warns you. Tutorials omit it because it is noise in a teaching example. Papers omit it because it is not in the habitual list of things to report. And the failure is invisible until someone tries to reproduce the work — at which point the discrepancy gets attributed to calibration drift, which is also real and much harder to rule out.

The honest framing is that this is a young-field problem, not a negligence problem. Classical computing took decades to converge on norms for reporting a benchmark, and quantum computing has an unusually long list of things that need pinning: versions, seeds, layouts, calibration windows, shot counts, and mitigation settings. Most of them did not exist as concepts five years ago.

What you can do is straightforward: pin every source of randomness you can find, record what you pinned, and state the calibration window. It costs nothing and it is the difference between a result and an anecdote.

Lessons

1. seed_transpiler is randomized by default. Eight seeds gave 18–21 two-qubit gates and three layouts on one circuit.
2. The variation propagates and amplifies: seed → layout → gate count and error rate → fidelity. Chapter 4 measured an 8× fidelity swing from layout alone.
3. Repeating is not reproducing. Repetition varies shots; reproduction varies compilation, and most workflows transpile once so the larger variation stays hidden.
4. Pin it and record it. Two lines, free.
5. Best-of-N is both an optimization and a reproducibility fix — deterministic given N, and 10% better here for 35 ms.
6. Record the full provenance: versions, backend, seed, optimization level, layout, transpiled gate count, shots, calibration window.
7. This is a young-field problem. The list of things needing pinning is long and recent. Pin what you can and say what you pinned.

Questions

1. Reproduce the seed sweep on your own circuit and backend. What is your spread? Is it larger or smaller than 17%, and what property of your circuit explains that?

2. Run the same circuit with a fixed seed but on two different days (or against two fake backends from different snapshots). Does the layout change? What does that tell you about which parts of the provenance record are stable?

3. Construct a circuit whose transpilation is insensitive to the seed — same gate count for every seed. What must be true of it? (Chapter 10 §10.3 has the answer.)

4. Best-of-N is deterministic given N. Is it stable across Qiskit versions? Design an experiment that would tell you, and say what you would do if it were not.

5. The provenance table lists eight fields. Rank them by how much a change in each would perturb a typical hardware result. Which one would you drop if you could only record four?

6. A colleague reports a result and, when asked, says they ran it five times and got consistent answers. Write the two-sentence reply that explains why that is not the reassurance they think it is, without being condescending.

7. Hardest. Suppose you must compare two ansätze on hardware. Compilation randomness means each gets a different layout, so part of any observed difference is layout luck rather than ansatz quality. Design a protocol that separates the two. (Consider: matched layouts, best-of-N applied equally, paired repetitions, and what you would hold fixed versus randomize. Chapter 30 §30.8 formalizes this.)