Case Study 2: The Comparison That Gave One Side Seventeen Times the Budget
The evaluation
A team is deciding how to measure observables on quantum states. They have seventeen quantities they care about — the terms of a Hamiltonian, plus magnetizations — and two options:
Direct estimation. One circuit per observable, measured in the observable's eigenbasis. Simple, obvious, and what everyone does.
Classical shadows. Measure in randomly-chosen Pauli bases, store the outcomes, and reconstruct every observable from the same record.
They run a careful comparison. Same state, same observables, same number of shots for each method, and twelve repetitions so the numbers are stable rather than one draw — exactly the discipline Chapter 33 §33.3 established.
shots direct |err| shadow |err| ratio
1,000 0.0202 0.0542 2.7x
5,000 0.0088 0.0218 2.5x
20,000 0.0050 0.0113 2.3x
Shadows are consistently about 2.5× less accurate. The result is stable across three shot counts and twelve repetitions each. They conclude that direct estimation is better and drop the shadow implementation.
Every number is correct. The methodology is careful. The conclusion is wrong.
What "same number of shots" meant
Direct estimation runs one circuit per observable.
direct at 1,000 shots: 1,000 shots x 17 observables = 17,000 total
shadows at 1,000 shots: 1,000 shots x 1 record = 1,000 total
The comparison gave direct estimation seventeen times the total budget.
This is not a subtle error, and it is easy to make: "1,000 shots each" sounds like a controlled comparison. It is a controlled comparison of per-observable cost, and the whole point of shadows is that they do not have a per-observable cost.
The comparison that answers the question
Give both methods the same total budget:
total budget direct (split 17 ways) shadows (all on one record) winner
1,700 0.0645 0.0431 SHADOWS 1.50x
8,500 0.0284 0.0172 SHADOWS 1.65x
34,000 0.0157 0.0087 SHADOWS 1.80x
Shadows win, at every budget, by 1.5–1.8×.
The arithmetic is clean. Statistical error scales as $1/\sqrt{N}$, so:
splitting a budget 17 ways costs sqrt(17) = 4.1x in error
shadows pay only ~2.5x for reusing one record
the difference -- about 1.6x -- is the win
⚛️ The Physics Underneath: shadows trade accuracy per estimate for estimates per shot.
A random Pauli-basis measurement produces a classical snapshot whose expectation over the random basis choice reconstructs the state. Any observable can then be estimated from the same snapshots — you are not measuring the observable, you are measuring the state and asking the record afterwards.
The snapshot count grows with the locality of the observables, not with how many there are. That is the entire reason the trade is favourable.
And the advantage grows
The $\sqrt{n}$ splitting penalty scales; the ~2.5× shadow penalty does not:
observables sqrt(n) penalty for splitting shadow penalty
4 2.0x ~2.5x
17 4.1x ~2.5x
50 7.1x ~2.5x
200 14.1x ~2.5x
1,000 31.6x ~2.5x
At four observables direct estimation is competitive. At a thousand it is not close.
Which tells you when to use each: shadows are for when the observables outnumber a handful, and the crossover is around four to eight — low enough that most realistic workloads are past it.
Why this matters beyond shadows
Chapter 24 §24.3 established the shot budget as the binding constraint on essentially everything in Parts IV and VI, and every technique since has run into $1/\epsilon^2$:
Ch. 24 mitigation costs shots, and can lose at fixed budget
Ch. 27 the fix for a flaky test is more shots, not more tolerance
Ch. 32 barren plateaus x shots -- a product, both terms growing
Ch. 34 concentration x n^2 -- another product
Every one of those tried to reduce the shots per estimate, and hit the same wall.
Shadows are the only technique in this book that attacks it from the other side: more estimates per shot. That does not repeal $1/\epsilon^2$ — the error per observable still scales that way — but it changes what a fixed budget buys, and it is why the quantum-data case of §35.4 is practical at all.
What the evaluation should have controlled for
1. FIX THE TOTAL BUDGET, not the per-item budget. Ask "given N shots,
which method gives lower error?" -- which is the question you have.
2. STATE THE OBSERVABLE COUNT. The answer depends on it, and the
crossover is around four to eight.
3. CHECK THE SCALING, not just the point. sqrt(n) versus a constant is
the whole result, and it is invisible at a single n.
The project module puts the fair comparison in the signature:
def shadow_advantage(n_observables, total_budget, direct_error, shadow_error):
"""Compare shadows against direct estimation AT EQUAL TOTAL SHOTS.
Comparing at equal shots PER OBSERVABLE makes shadows look ~2.5x worse and is
the wrong comparison -- it silently gives direct estimation n times the total
budget.
"""
total_budget is a required argument, and splitting_penalty returns $\sqrt{n}$ so the reason for
the win is visible next to the win.
The lessons
Control the resource you actually have. The team had a shot budget, not a per-observable budget, and the comparison should have fixed the thing that is fixed.
"Same shots each" is not a controlled comparison when the methods have different shot structures. One method's cost is per-observable and the other's is not; equalizing per-observable silently multiplies one side by $n$.
Check the scaling. $\sqrt{n}$ against a constant is the result. At $n=17$ it is a 1.6× win; at $n=1{,}000$ it is a 12× win; and at $n=4$ it is a loss. A single measurement point cannot show any of that.
And notice the shape. Chapter 21 compared Grover against a strawman classical search. Chapter 24 compared QAOA against a measured ratio rather than a proven guarantee. Chapter 28 read a gate-count improvement as a performance claim. Here the comparison was scrupulous and the resource being controlled was the wrong one — which is a subtler failure and produces the same kind of confident wrong answer.
Reproduce it: code/example-02-classical-shadows.py runs the per-observable comparison first
(where shadows lose), then the equal-budget comparison (where they win), then the scaling table;
shadow_advantage in code/vqelab/hybrid.py requires the total budget, and
test_shadows_lose_per_observable_and_win_at_equal_budget asserts both directions.