Case Study 1: Green for a Year
The suite
A team maintains a quantum transforms library — QFT, its approximate variant, phase estimation built on top. They take testing seriously, and they know the central difficulty: for most of what they ship there is no oracle. You cannot assert the correct output of a QFT on an arbitrary input without already having a correct QFT.
So they do the right thing and write metamorphic properties — relations that must hold, requiring no reference implementation:
def test_adjoint_round_trip():
assert Operator(qft.compose(qft.inverse())).equiv(Operator(identity))
def test_is_unitary():
m = Operator(qft).data
assert np.linalg.norm(m.conj().T @ m - np.eye(len(m))) < 1e-9
def test_qft_of_zero_is_uniform():
p = Statevector.from_instruction(qft).probabilities()
assert np.abs(p - 1 / 2**n).max() < 1e-9
Three properties, all genuinely true of the QFT, all checkable without a reference, all running in microseconds. They run on every commit across every qubit count from 2 to 10. Roughly 300 assertions, green for a year.
What was in the library
A wrong rotation angle. One controlled-phase gate carried $\pi/3$ where it should have carried $\pi/2$ — a transposition made during a refactor.
Operator(good) == Operator(bad): False
process_fidelity between them: 0.949760
A 5% infidelity, shipped for a year, under a green suite of 300 assertions.
Why every property passed
Take them in turn, because each fails for a different reason and the pattern only emerges from all three.
$U U^\dagger = I$ — structurally incapable of catching this.
good: U U^dag == I ? True
bad: U U^dag == I ? True
qc.inverse() inverts your circuit. If your circuit has a wrong angle, the inverse has the
matching wrong angle, and they cancel exactly. This property is not weak, it is blind by
construction — no error that inverse() can reproduce will ever violate it, which is every error in
the gate parameters.
This is the most-cited property in quantum testing. It appears in tutorials, in framework documentation, in nearly every quantum testing talk. It cannot detect a wrong gate parameter, ever.
Unitarity — testing the framework, not the code.
good: ||U^dag U - I|| = 1.47e-15
bad: ||U^dag U - I|| = 1.72e-15
Every circuit assembled from Qiskit gates is unitary by construction. This assertion verifies that
Qiskit's gate definitions are unitary and that Operator multiplies matrices correctly. Both are
true, neither is in question, and the team's own code is not involved.
The known answer — a blind input.
good: max|p - 1/8| = 5.55e-17 PASS
bad: max|p - 1/8| = 5.55e-17 PASS
$\text{QFT}|0\dots0\rangle$ really is the uniform superposition, and this really does test the team's code. But from $|0\dots0\rangle$ every control is in $|0\rangle$, so every controlled-phase gate is the identity and the wrong angle never fires.
This is exactly Chapter 26 §26.4's blind spot — the same failure that made a debugger report no divergence — arriving a second time as a test. Chapter 26 measured it: 4 of 8 computational basis states are blind to this bug, 11 of 27 structured states (41%), and 0 of 100 random ones.
The property that would have caught it
The shift theorem: $\text{QFT}|x+1\rangle$ equals $\text{QFT}|x\rangle$ with amplitude $k$ multiplied by $e^{2\pi i k/N}$.
good: worst deviation = 0.0000 PASS
bad: worst deviation = 0.1830 FAIL
It needs no reference implementation. It runs in microseconds. It is a genuine structural property of the transform. And it catches the bug on the first input it tries.
What makes it different is that it relates different inputs to each other. The three failed properties each constrain the circuit's behaviour at a single point, or constrain a global algebraic feature that any assembled circuit satisfies automatically. The shift theorem constrains the relationship between $\text{QFT}|x\rangle$ and $\text{QFT}|x+1\rangle$ — and a wrong controlled-phase angle is precisely a wrong relationship between inputs that differ in the controlling bit.
⚛️ A property is only useful if the bug can violate it.
"It is self-inverse" and "it is unitary" are satisfied by an enormous space of wrong circuits. They are necessary conditions, and necessary conditions make poor tests.
Write down the failure mode first, then the property that would break under it. Reversing that order produces properties that are true, cheap, elegant, and worthless.
The scoreboard
property needs a reference? catches the bug?
U U^dag == I no NO
unitarity no NO
QFT|000> uniform no NO
shift theorem no YES
random inputs vs QFTGate YES YES
Three of four oracle-free properties pass a circuit with 5% infidelity.
Note also the last row. The team had a reference available the whole time — qiskit.circuit.library.
QFTGate — and comparing against it on 20 random inputs takes 5.75 ms and catches the bug with a worst
infidelity of 0.0665. The hardest case, "no oracle exists," was not their case. They had reasoned
their way into treating an available reference as unavailable, because the library QFT was the thing
they were trying to replace.
What they should have done
Assess each property's discriminating power. For every property, construct a deliberately perturbed circuit and confirm the property rejects it:
result = check_property("U U^dag == I", adjoint_round_trip, circuit, perturbed=broken)
assert result.discriminating, "this property cannot distinguish correct from broken"
This is mutation testing, and it takes one extra line per property. Applied to their suite it would have flagged all three properties as non-discriminating on day one — before a year of green builds established false confidence.
Compare against a reference whenever one exists. "We are reimplementing X" does not mean X is unavailable as a test oracle. It means X is the best oracle you will ever have for this code.
And never let $|0\dots0\rangle$ be your only input. It is the most structured state available, the first one anyone types, and blind to a large class of real bugs.
The lessons
Green is not evidence. Three hundred passing assertions established only that the code satisfied three conditions an enormous space of wrong circuits also satisfies.
Necessary conditions make poor tests. $UU^\dagger = I$ and unitarity are both true and both nearly content-free. The useful question is not "what is true of my circuit?" but "what would stop being true if my circuit were wrong?"
Test your tests. A property that passes a deliberately broken circuit is not a test, and finding out costs one line.
And notice this is the fifth time. Chapter 19's oracle tested only in the computational basis; Chapter 24's exact simulator hiding shot noise; Chapter 25's QEC test storing an eigenstate of its own failure mode; Chapter 26's bisection blind from $|000\rangle$; and now a property suite blind by construction. The recurring failure in this book is not wrong code. It is tests that cannot fail.
Reproduce it: code/example-02-testing-without-an-oracle.py runs all five properties against both
circuits and prints the scoreboard; check_property in code/vqelab/testing.py returns
discriminating=False for the three that pass a broken circuit, and
test_three_of_four_oracle_free_properties_pass_a_broken_circuit asserts the count.