Case Study: Building a Reproducible Quantum Experiment Harness

Executive Summary

Six months after publishing, a reviewer asks you to reproduce Figure 3. The backend has been recalibrated two hundred times, Qiskit has had a major release, the transpiler's stochastic routing has changed, and your script hardcodes least_busy(). You cannot reproduce your own result, and neither can anyone else.

Quantum experiments have more reproducibility hazards than ordinary software: the hardware itself is a time-varying instrument. This case study builds a harness that records enough to make a run reconstructible — and, where exact reproduction is impossible in principle, makes that explicit rather than pretending otherwise.

Skills applied

• Identifying every source of run-to-run variation (§8.14).
• Pinning transpilation with seeds and explicit layouts (§8.10).
• Capturing backend calibration state alongside results (§8.13).
• Reporting statistical and systematic uncertainty separately.

Background: seven sources of variation

Rows 1, 4, and 7 are irreducible: shot noise is physics, drift is physics, and hardware gets decommissioned. The harness cannot eliminate them, so its job is to record them precisely enough that a later reader can tell whether a discrepancy is expected.

Phase 1: Pin everything pinnable

import json, hashlib, sys
from datetime import datetime, timezone
from qiskit import transpile, qpy
from qiskit_ibm_runtime import QiskitRuntimeService, SamplerV2 as Sampler

CONFIG = {
    "backend_name": "ibm_brisbane",       # pinned, not least_busy()
    "initial_layout": [12, 13, 14],       # pinned
    "optimization_level": 3,
    "seed_transpiler": 20260727,
    "shots": 8192,
}

service = QiskitRuntimeService()
backend = service.backend(CONFIG["backend_name"])

isa = transpile(circuit, backend,
                initial_layout=CONFIG["initial_layout"],
                optimization_level=CONFIG["optimization_level"],
                seed_transpiler=CONFIG["seed_transpiler"])

Three of the seven hazards are closed by this block alone. The most important is the pinned backend: least_busy() is convenient and makes a script non-reproducible by construction.

Phase 2: Archive the transpiled circuit

The transpiled circuit is the actual experiment. Transpiler behavior changes between releases, so re-transpiling later is not guaranteed to reproduce it — even at a fixed seed.

with open("run_isa_circuit.qpy", "wb") as f:
    qpy.dump(isa, f)                       # exact, version-stable serialization

circuit_hash = hashlib.sha256(
    qpy_bytes_of(isa)).hexdigest()[:16]    # identity you can cite

QPY is Qiskit's binary circuit format and survives across versions. Archiving it means a future reader can execute exactly what you executed, rather than an approximation of it.

Phase 3: Snapshot the instrument

This is the step that distinguishes quantum experiments from ordinary software runs. The device is part of the experiment.

props = backend.properties()
snapshot = {
    "last_calibrated": props.last_update_date.isoformat(),
    "qubits": {
        q: {
            "T1_us":  props.t1(q) * 1e6,
            "T2_us":  props.t2(q) * 1e6,
            "readout_error": props.readout_error(q),
            "frequency_GHz": props.frequency(q) / 1e9,
        } for q in CONFIG["initial_layout"]
    },
    "two_qubit_gate_errors": {
        f"{a}_{b}": props.gate_error("ecr", [a, b])
        for a, b in backend.coupling_map
        if a in CONFIG["initial_layout"] and b in CONFIG["initial_layout"]
    },
}

Without this, a reader who reruns your script in a year and gets a different number cannot tell whether your result was wrong or the device simply has different $T_2$ today. With it, they can check.

Phase 4: Run, and record provenance

sampler = Sampler(mode=backend)
job = sampler.run([isa], shots=CONFIG["shots"])
counts = job.result()[0].data.c.get_counts()

record = {
    "config": CONFIG,
    "circuit_sha256": circuit_hash,
    "circuit_depth": isa.depth(),
    "two_qubit_gates": sum(1 for i in isa.data if i.operation.num_qubits == 2),
    "backend_snapshot": snapshot,
    "job_id": job.job_id(),
    "submitted_utc": datetime.now(timezone.utc).isoformat(),
    "versions": {
        "python": sys.version.split()[0],
        "qiskit": __import__("qiskit").__version__,
        "qiskit_ibm_runtime": __import__("qiskit_ibm_runtime").__version__,
    },
    "raw_counts": counts,
}
json.dump(record, open(f"run_{circuit_hash}.json", "w"), indent=2)

Raw counts are the primary datum. Store them always, unprocessed. Mitigated, normalized, or fitted values are derived products and should be recomputable from the raw counts by anyone.

The job_id is worth keeping: providers retain job records, so a reader can often retrieve the original result server-side independently of your file.

Phase 5: Report uncertainty in two parts

A single error bar hides the distinction that matters.

Statistical — shot noise, shrinks as $1/\sqrt N$, and is fully under your control:

$$\sigma_{\text{stat}} = \sqrt{\frac{p(1-p)}{N}}$$

Systematic — readout bias, coherent gate errors, drift. Does not shrink with more shots. Estimate it by repeating the identical experiment across several calibration cycles and taking the spread of the means.

# same circuit, five separate days
means = [0.812, 0.798, 0.834, 0.805, 0.789]
import statistics
print(f"systematic spread: {statistics.stdev(means):.3f}")   # ~0.017

A result quoted as $0.808 \pm 0.004$ (statistical only) when the day-to-day spread is $0.017$ is misleading by a factor of four. Report both: $0.808 \pm 0.004\ (\text{stat}) \pm 0.017\ (\text{sys})$.

This is the most common reporting failure in the field. Shot-noise error bars are easy to compute and are routinely presented as though they were the whole uncertainty.

Phase 6: The reproducibility statement

Publish, alongside the figure:

Circuits were transpiled for ibm_brisbane with optimization_level=3, seed_transpiler=20260727, and initial_layout=[12,13,14]; the exact transpiled circuits are archived in QPY format (SHA-256 a3f2…). Backend calibration data at submission time is included in the record. Raw counts are provided; all derived quantities are recomputable from them. Results were collected across five calibration cycles between 2026-07-14 and 2026-07-21; the day-to-day spread of the mean is reported as a systematic uncertainty. Exact reproduction is not possible in principle, as device parameters drift and the calibration state of a given day cannot be restored.

That last sentence is the honest one, and including it costs nothing.

Discussion Questions

1. Three of the seven variation sources are irreducible. Does that make quantum experiments unreproducible, or does it change what reproducibility means?
2. Why archive the transpiled circuit rather than the source circuit plus a seed?
3. A colleague reports $0.91 \pm 0.003$ from 100,000 shots in one session. What is your first question?
4. What is lost if only mitigated results are published, and not raw counts?

Your Turn: Extensions

• Implement the harness and run one circuit through it; inspect the JSON record.
• Run the same circuit on five separate days; plot the means and compute the systematic spread.
• Reload an archived QPY circuit and confirm it matches the recorded hash.
• Compare the transpiled output of the same source circuit under two Qiskit versions.

Key Takeaways

• The hardware is part of the experiment: record calibration data and timestamps with every result.
• Pin the backend, layout, seed, and versions; archive the transpiled circuit in QPY, since that is what actually ran.
• Raw counts are the primary datum. Everything else must be recomputable from them.
• Separate statistical from systematic uncertainty. Shot-noise bars alone routinely understate total error several-fold.
• Where exact reproduction is physically impossible, say so explicitly instead of leaving the reader to discover it.