Case Study 2: The Flaky Test That Got Fixed
The annoyance
A team has a distribution test. It compares their state-preparation circuit's measured output against the exact distribution, at 1,000 shots, with a tolerance of 0.05 on total variation distance.
It fails roughly once in every six or seven hundred runs.
Not on any particular change — it fails on merges to unrelated files, on documentation commits, on re-runs of builds that just passed. Everyone knows the drill: hit retry, it goes green, move on.
With around 200 similar tests in the suite, a 0.15% per-test rate produces a red build every four or five commits. Not a crisis; just steady friction, several retries a week, and a slow erosion of the habit of reading a red build as meaningful. Someone eventually files the obvious ticket: fix the flaky test.
The fix
The diagnosis is straightforward and correct. The test fails because shot noise makes the measured distribution wander, and 0.05 is close enough to the noise floor to be crossed occasionally:
shot noise TVD on a CORRECT circuit at 1,000 shots
mean 0.01313 std 0.01003 max over 40 runs 0.03700
The tolerance is only about 3.7 standard deviations out. Crossings are rare but inevitable.
So they loosen it to 0.10 and measure the result:
shots tolerance false failures rate 95% upper bound
1,000 0.05 3/2,000 0.150% 0.320%
1,000 0.10 0/2,000 0.000% 0.150%
Zero false failures in 2,000 runs. The flakiness is gone. The ticket closes. Builds are green.
(Note they did the measurement properly, at 2,000 runs rather than 200. A rate near 0.15% cannot be resolved by 200 runs at all — the standard error would exceed the rate. That part of their work was sound, which is what makes the rest of the story worth telling.)
What the fix cost
Nobody measured the other error rate. Here it is — detection against a real bug, a rotation error of size $\varepsilon$ in the state preparation:
bug size eps true TVD detected @1k/tol.10 detected @10k/tol.02
0.00 0.0000 0% 0%
0.02 0.0100 0% 1%
0.05 0.0250 0% 86%
0.10 0.0499 0% 100%
0.20 0.0993 52% 100%
0.40 0.1947 100% 100%
At 1,000 shots and tolerance 0.10, the test detects a bug of size $\varepsilon = 0.10$ zero percent of the time. That is a real defect with a true TVD of 0.0499 — half the tolerance, by construction undetectable. The test does not begin to notice anything until the bug is twice that size, and even then catches it only half the time.
The test stopped flaking because it stopped being able to fail.
The two error rates
Every statistical test has two, and they trade against each other along the tolerance axis:
tolerance TIGHT -> fails on correct code (flaky)
tolerance LOOSE -> passes on broken code (blind)
Moving the tolerance does not improve the test. It slides the test along a curve, exchanging one failure mode for the other. The team traded a 0.15% chance of a spurious red build for a 100% chance of missing a 5% error.
And the trade is asymmetric in a way that hides it: flakiness is loud and blindness is silent. A flaky test files its own ticket. A blind test never says anything at all, and its silence is indistinguishable from correctness.
🔬 Honest Assessment: a test has two error rates, and tuning one blindly destroys the other.
"It stopped flaking" is not evidence that a test improved. It is equally consistent with the test having been disabled.
The fix that works
Move along the other axis. Increase the shots:
shots tolerance false failures detects eps=0.05 detects eps=0.10
1,000 0.10 0.0% 0% 0%
10,000 0.02 0.0% 86% 100%
10,000 shots at tolerance 0.02 also has a zero false-failure rate — it does not flake either — and it catches a 5% error 86% of the time and a 10% error always.
The cost, from §27.2's table:
simulated counts, 1,000 shots 85.43 ms 730 assertions per CI-minute
simulated counts, 10,000 shots ~200 ms ~300 assertions per CI-minute
Roughly a factor of two to seven in wall-clock, depending on circuit size. For 200 tests that is tens of seconds of CI time, in exchange for a suite that can detect the errors it exists to detect.
The correct fix for a flaky distribution test is more shots. Shots cost CPU seconds. Tolerance costs detection, and does not send an invoice.
The rule that would have prevented it
A distribution test is not fully specified by its tolerance. It needs three numbers:
1. the tolerance
2. the shot count
3. THE SMALLEST BUG IT CAN DETECT
The third is the one that makes the test a test, and it is the one nobody writes down. It is also cheap to compute — construct a perturbed circuit, sweep the perturbation, and record where detection crosses 50%.
The project module makes this structural. DistributionTest cannot be constructed without a shot
count, computes both error rates on request, and says so explicitly:
1,000 shots, tolerance 0.100 (measured max 0.037) | false-failure rate 0.0% |
detection rate 0.0% | THIS TEST NEVER FLAKES BECAUSE IT IS BLIND.
Increase SHOTS, not tolerance.
And assert_distribution raises a ValueError — not an AssertionError — when the tolerance is
below the shot-noise floor, with the shot count you would need:
ValueError: tolerance 0.0100 is below the shot-noise floor 0.0949 at 1,000 shots
-- this test would fail on CORRECT code. Raise the shot count to at least 90,000,
or loosen the tolerance and accept the loss of detection.
A broken test is a different kind of failure from a broken circuit, and it deserves a different
exception. An AssertionError says "your code is wrong." A ValueError says "your test is
impossible."
The lessons
"It stopped flaking" is not a result. It is consistent with a fixed test and with a disabled one, and distinguishing them requires measuring detection.
Quote both error rates, always. A tolerance and a shot count describe what a test costs. Only the detection threshold describes what it does.
Fix flakiness with shots, not tolerance. One costs CPU time you can see on a dashboard; the other costs coverage that nothing reports.
Compute the noise floor before choosing a tolerance. $\sim 3/\sqrt{N}$ is the level below which a correct circuit fails — it takes one line, and it turns tolerance selection from taste into arithmetic.
And note where this one sits in the book. Chapter 24's shot budget said 10,000 shots was not enough for chemical accuracy; the same $1/\sqrt N$ wall now sets the sensitivity of a CI suite. Chapter 25's team read an improvement factor when they needed a breakeven. The recurring error is reporting the number that is easy to get instead of the number that answers the question.
Reproduce it: code/example-03-two-error-rates.py measures the shot-noise floor, both error rates,
and the detection sweep; DistributionTest.summary() in code/vqelab/testing.py prints the
never-flakes-because-blind warning, and test_a_test_that_never_flakes_because_it_is_BLIND asserts
both rates are zero at once.