Case Study 2: Three Failures That Looked Alike

The premise

A three-qubit GHZ state should return 000 and 111 in roughly equal measure and nothing else. Here are three runs of "the same" experiment. All three are wrong. One is a bug, one is ordinary noise, and one is a bad layout — and the question is whether you can tell which is which from the histogram, before you know the answer.

  Run A:  {'000': 2001, '111': 1983, '110': 35, '101': 20, '100': 17,
           '001': 14, '011': 14, '010': 12}          correct: 0.9727

  Run B:  {'111': 1165, '110': 1060, '011': 939, '010': 932}
                                                     correct: 0.2844

  Run C:  {'000': 2001, '011': 1988, '010': 44, '001': 37,
           '100': 17, '111': 5, '101': 4}            correct: 0.4897

Try it before reading on. Then watch how little of the answer comes from staring at the numbers.

Run A: the easy one

Correct fraction 0.9727, a smooth tail of low-probability wrong answers, both peaks nearly equal.

This is what healthy looks like. But "looks healthy" is not a finding — §12.7 step 4 asks whether the deviation is the right size, and that is answerable:

  [4] error budget: 2 ecr gates at mean error 0.0051,
      readout 0.0065 on 3 qubits
      predicted correct fraction: 0.9706
      measured:                   0.9727
      shortfall: -0.0020

Predicted 0.9706, measured 0.9727. Two parts in a thousand.

That is a much stronger statement than "looks fine." The error budget was built from calibration data before looking at the result, using Chapter 2's Case Study 2 method, and it landed. The remaining 2.7% of shots are accounted for, gate by gate and readout by readout. Nothing is left over that needs explaining.

Verdict: noise, and explained. This is the only one of the three where you can say explained.

Run B: the one that looks worst and is easiest

Correct fraction 0.2844. Alarming — and it is the fastest of the three to diagnose, because §12.7's step 1 is free.

  [1] ran on: fake_sherbrooke, qubits [83, 84, 85]
      *** PREFLIGHT FAILURE: [84, 'pair(83, 84)', 'pair(84, 85)']

Qubit 84 is Case Study 1's stuck qubit, and both two-qubit links in the chain have a calibrated error of exactly 1.0. The circuit was submitted onto three qubits, one of which reports 1 no matter what and two of whose connecting gates do not work.

Look again at the histogram with that in hand:

  {'111': 1165, '110': 1060, '011': 939, '010': 932}

Only four outcomes appear, out of eight. And every one of them has the middle bit set — the middle clbit reads physical qubit 84, which is stuck at 1. The four missing outcomes are exactly the four with a middle 0. The histogram was telling you which qubit was broken, if you knew to read the bit positions rather than the totals.

The remaining two bits are close to uniform, which is what two dead entangling gates produce: no correlation was ever established, so the outer qubits are independent coin flips.

The 0.2844 "correct fraction" is not a partially-working GHZ state. It is the fraction of random outcomes that happen to look like 111 — about a quarter, as you would expect from two free bits.

The trap in step 4. The error budget predicts 0.0000, because $(1 - 1.0)^2 = 0$. The measured value 0.2844 is higher than predicted, and §12.7's step 4 lists "much smaller deviation than predicted" as a sign you accidentally ran a simulator. That branch would have misled you here.

The resolution is that step 1 already fired. The procedure is ordered for a reason: the cheap, unambiguous checks come first precisely so that the subtler ones are never load-bearing. When an early step gives a clean answer, stop — do not keep walking down the list collecting evidence that needs interpreting.

Verdict: a bad layout — caught before any physics was considered.

Run C: the one that looks mild and is fatal

Correct fraction 0.4897. The least alarming number of the three, and structurally the most deceptive: two clean peaks of almost exactly equal height, with a small tail. That is the shape of a healthy GHZ state. It is what Run A looks like.

Except the peaks are 000 and 011.

  {'000': 2001, '011': 1988, '010': 44, '001': 37, '100': 17, '111': 5, '101': 4}

Not 111. 011. One bit off — and 111 appears five times out of 4096, which is to say never. A "correct fraction" of 0.4897 is entirely 000; the other peak is being scored as an error even though it holds a quarter of all shots and is manifestly not noise.

Preflight is clean. It ran on qubits [122, 123, 124], the best triple on the device. So the layout is right, the device is healthy, and half the shots are landing on a single specific wrong answer. At this point the temptation is to reach for physics: crosstalk, maybe, or an unusually bad calibration window.

§12.7 step 2 costs one line:

  [2] noiseless simulation: {'000': 2072, '011': 2024}   correct 0.5059
      *** FAILS WITHOUT NOISE -> IT IS A BUG. Stop here.

It fails with the noise removed — and the noiseless run is the same two peaks, now perfectly clean. That is as unambiguous as diagnosis gets: the wrong answer is not merely surviving the removal of noise, it is sharpening.

The circuit had a cz where a cx belonged. A cz whose target sits in $|0\rangle$ does nothing at all — it applies a phase only when both qubits are $|1\rangle$. So qubit 2 was never touched:

$$H_0,\; \text{CX}_{0,1} \;\Rightarrow\; \frac{|00\rangle + |11\rangle}{\sqrt2} \text{ on } q_0 q_1, \qquad q_2 = |0\rangle$$

Measuring gives $q_2 q_1 q_0 = $ 000 or 011. Exactly the two peaks observed. The state is a perfectly good Bell pair with an idle third qubit — which is why it looks so healthy. It is a correct answer to a different circuit.

A failure that survives the removal of noise is not a noise failure. That inference is airtight, it needs no physics, and it takes one line.

Verdict: a bug. Nothing about the device was ever relevant.

The scoreboard

The worst-looking run was the easiest to diagnose, and the mildest-looking one was the only real bug. Ranking the three by how bad the histogram looks gets the priority order exactly backwards.

Note also which steps did the work: step 1 and step 2 resolved two of three cases, and both are essentially free. The error budget — the most technically involved step — was needed only for the run that turned out to be fine.

The lessons

Severity does not indicate cause. A catastrophic histogram meant a five-second fix; a mild one meant the code was wrong. Diagnose by procedure, not by alarm.

Run the cheap checks first, and stop when one fires. Step 1 caught Run B before any modeling, and the modeling would have misled on Run B. An ordered procedure is not bureaucracy — later steps have failure modes that earlier steps protect you from.

Read bit positions, not just totals. Both failures were named by their histograms and hidden by their summary statistics. Run B's four missing outcomes all shared a middle bit of 0 — that is the stuck qubit, spelled out. Run C's second peak was 011 rather than 111 — one bit, and it says "qubit 2 was never entangled." "Correct fraction 0.2844" and "correct fraction 0.4897" throw both diagnoses away, which is Case Study 1's lesson arriving from the other direction: a scalar summary of a distribution discards the structure that identifies the fault.

A corollary worth internalizing: a large peak on a single wrong outcome is never noise. Noise spreads across many outcomes (Chapter 11 §11.7). One tall wrong peak is a deterministic error — which means it is in your circuit, your layout, or your bit ordering.

"Explained" is a higher standard than "looks fine," and it is reachable. Run A's 2.7% error was predicted to two parts in a thousand from calibration data alone. Until you have done that, you do not know whether the healthy-looking result is healthy or is hiding a second effect that happens to be small.

And the theme, for the fifth time: check the pipeline before the physics. Chapter 7's layout trap, Chapter 8's parameter ordering, Chapter 8's Case Study 1 team with their four physics hypotheses, Chapter 10's routing overhead, and now Run C. Every one would have been caught in seconds by a noiseless simulation, and in every case that was tried late or not at all.

Reproduce it: code/example-05-noise-or-bug.py runs all three cases and prints the full procedure for each.