Case Study 1: The Debugger That Certified a Broken Circuit

DataField.Dev

Case Study 1: The Debugger That Certified a Broken Circuit

The setup

A team maintains a quantum arithmetic library. Their modular exponentiation routine — Chapter 23's expensive part — has been rewritten for depth, and they need to confirm the new version matches the old one.

They write exactly the right tool. Circuit bisection: binary search for the first instruction at which the new circuit's state diverges from the reference's. It is a good idea, correctly reasoned, and it turns a 3,368-gate haystack into thirteen comparisons.

def diverged(k):
    return state_fidelity(
        Statevector.from_instruction(prefix(new, k)),
        Statevector.from_instruction(prefix(old, k))) < 1 - 1e-9

They run it. No divergence. They run it on a smaller instance. No divergence. They run it on three more test circuits. Nothing.

The rewrite ships.

What was actually true

The circuits were different. Not subtly — process_fidelity between them was 0.9498, and Operator(new) == Operator(old) was False. A rotation angle had been mistyped during the rewrite.

The bisection was comparing states, and Statevector.from_instruction starts from $|0\dots0\rangle$. From the all-zeros state, every control reaching the mistyped gate is in $|0\rangle$, so the controlled-phase gate is the identity and the wrong angle never fires. The states are genuinely, exactly identical. The measurement was correct and the conclusion was backwards.

The part that makes this hard

The natural response is "well, test with a different input." So try the other obvious one:

   state |000>   ->  NOTHING FOUND
   state |+++>   ->  NOTHING FOUND

$|{+}{+}{+}\rangle$ is blind too, for a different reason: the Hadamard earlier in the circuit maps $|+\rangle$ to $|0\rangle$ on the relevant wire before the buggy gate arrives. The two states everyone reaches for are both blind, by two unrelated mechanisms.

At this point it is tempting to conclude that you just need to think a little harder about which input to use. That is also wrong, and here is the measurement that settles it:

   computational basis states blind:   4/8    ['000', '010', '100', '110']
   states over {0, 1, +} blind:       11/27   (41%)
   random states blind:               0/100

Forty-one percent of structured inputs are blind to this bug. And while writing the project checkpoint for this chapter, $|1{+}0\rangle$ was added to the list of states that "obviously" would catch it — it has a $|+\rangle$ and a $|1\rangle$, after all. The test failed. $q_0$ is in $|0\rangle$, so the gate never fires.

You cannot reason your way to a safe test input, because choosing a good one requires already knowing where the bug is.

That is not a solvable problem. It is a reason to stop trying to solve it and use an input- independent test instead.

The fix

Two lines, and a change in what the tool is allowed to say.

Compare operators, not states, whenever the circuit is small enough to build one:

def diverged(k):
    return Operator(prefix(suspect, k)) != Operator(prefix(reference, k))

This is input-independent and cannot be blind. It costs $2^n \times 2^n$ memory, so it works to about 12–14 qubits. Past that, use random_statevector — and use more than one, since 0 of 100 random states were blind but that is a statistical statement, not a guarantee.

And make the tool refuse to say "pass" when it cannot tell. The project module returns one of three verdicts:

   DIVERGED  -- found the first differing instruction. Trustworthy.
   AGREE     -- the circuits really are equal, verified input-independently.
   BLIND     -- this input cannot see a difference that DOES exist. NOT A PASS.

When a state-based bisection finds nothing, it cross-checks against the operator before concluding anything. If the circuits differ and the input could not tell, the verdict is BLIND. A team reading BLIND runs it again with a better input. A team reading AGREE ships.

The uncomfortable generalization

This is the fourth appearance of one failure mode in eight chapters:

Chapter 19 — an oracle "worked" because it was only ever tested in the computational basis, where a phase oracle is indistinguishable from doing nothing.
Chapter 24 — a VQE result reproduced perfectly on an exact simulator, which hides shot noise entirely.
Chapter 25 — a QEC test reported zero logical error at a noise rate where zero is impossible, because the stored state was an eigenstate of the failure mode.
Chapter 26 — a debugging tool reports no divergence because its default input cannot see the defect.

Each time the measurement was precise, reproducible, and about something other than the question being asked. And each time the tell was the same: a result that is impossible in principle, or too clean to be true.

What makes this instance worse than the others is what the tool is for. A blind QEC test produces a wrong number in a paper. A blind debugger produces a certification — the entire point of running it was to answer "is this correct?", and it answered "yes."

The tool you use to find bugs is itself a program, and it can have the bug you are looking for.

Debugging tools need tests. The project checkpoint for this chapter has more assertions about what bisect refuses to claim than about what it finds.

The lessons

Prefer input-independent comparisons. Operator and process_fidelity compare the whole map. A state comparison only ever tells you about one input, and you do not get to know in advance whether that input is the informative one.

When you must use inputs, use random ones. Structured states have structured blind spots, and the most structured states are the ones you type first. 41% versus 0% is not a close call.

Make tools report uncertainty as uncertainty. "I found nothing" and "there is nothing" are different claims, and a tool that conflates them is worse than no tool, because it converts an open question into a false answer.

And treat "the tests pass" as a claim requiring evidence. The bisection ran clean on five separate circuits. All five runs were correct. All five conclusions were wrong.

Reproduce it: code/example-02-circuit-bisection.py runs the same bisection across seven inputs and then counts the blind fraction; bisect in code/vqelab/debugging.py returns Verdict.BLIND rather than Verdict.AGREE, and test_bisecting_from_zero_reports_BLIND_not_AGREE in code/project-checkpoint.py is the assertion that keeps it honest.