Case Study 1: The Demonstration That Could Not Fail
The result
A team builds a variational quantum classifier and writes it up. The circuit is 4 qubits, angle
encoding, two StronglyEntanglingLayers, 24 parameters. The dataset is iris, restricted to two
classes. The result:
VQC test accuracy: 1.0000
One hundred percent. The write-up notes that a quantum model with 24 parameters classifies the data perfectly, includes the training curve, and describes the approach as promising.
Every number in it is correct.
The comparison that was not run
model train time test accuracy
LogisticRegression 2.2 ms 1.0000
SVC (rbf) 0.6 ms 1.0000
RandomForest 60.8 ms 1.0000
All three classical baselines also get 100%. Iris-binary — setosa versus versicolor — is linearly separable. It has been the textbook example of a trivially separable dataset since Fisher published it in 1936.
The quantum result is not wrong. It is unfalsifiable in this setting: every method gets 100%, so no method can be distinguished from any other.
🔬 Honest Assessment: matching a baseline is not evidence of anything.
The result demonstrates that a parameterized quantum circuit can express a linear decision boundary — which was never in doubt, and which a single perceptron has done since 1958.
A demonstration on a dataset the baseline solves perfectly cannot distinguish the two methods. It can only fail to.
And the cost, which was also not reported
The comparison gets worse when timing is included.
model train time train acc test acc
LogisticRegression 2.2 ms 1.0000 1.0000
VQC (4q, 2 layers, exact) 50.2 s 0.9714 1.0000
Roughly 20,000× slower — and that is on an exact simulator, with no shots, no sampling, and no noise. It is the most favourable possible condition for the quantum model, and it also achieves lower training accuracy (0.9714 against 1.0000).
Then the hardware cost:
TOTAL = (2n+1) x samples x steps x shots
= 49 x 70 x 60 x 10,000
= 2,058,000,000 shots
= 57.2 QPU HOURS = 2.38 DAYS
Two billion shots and 2.38 days of continuous QPU time, to match a model that trained in 2.2 milliseconds on a laptop.
That number is one line of arithmetic. It was never computed, because the paper reported qubit count, circuit depth, parameter count, accuracy, and iteration count — the standard set — and total shots is not in it.
Why this keeps happening
Not dishonesty. Three structural reasons, and they are worth naming because they are fixable.
The easy datasets are the ones that fit. A 4-qubit device can encode 4 features. Iris has 4 features. The datasets small enough to run are, almost by construction, the datasets classical methods solve trivially — so the demonstration that is possible is the demonstration that proves nothing.
The baseline is not the interesting part. A researcher building a quantum classifier is thinking
about ansätze, encodings, and optimizers. Running LogisticRegression().fit() takes one line and is
nobody's contribution, so it gets skipped — and Chapter 21 §21.7 found exactly this with Grover
against a strawman search, and Chapter 24 §24.5 with QAOA against Goemans–Williamson.
And the accepted reporting format does not include the decisive number. Total shots determines whether an experiment is possible at all. It is absent from the conventional results table, so its absence is not noticed.
What the write-up should have said
Not "our quantum classifier achieves 100% accuracy." Either:
"Our quantum classifier matches three classical baselines that also achieve 100%, at roughly 20,000× the training cost on a simulator and an estimated 2.38 QPU-days on hardware. The dataset is linearly separable, so this result establishes expressibility, not advantage."
That is honest, it is not embarrassing, and it is a real contribution — demonstrating that a circuit trains at all, on hardware-realistic assumptions, is worth publishing.
Or: pick a harder dataset. The moment the baseline drops below 100%, the comparison starts carrying information.
The project module's refusal
compare_to_baseline will not report a verdict when the baseline is already perfect:
if classical_accuracy >= perfect_threshold:
return BaselineComparison(
verdict=Verdict.UNINFORMATIVE_TIE,
reason=(f"{classical_name} already reaches {classical_accuracy:.4f}. "
f"The problem is solved classically, so a tie distinguishes nothing... "
f"Use a dataset the baseline does NOT solve, and tune the baseline "
f"before comparing."),
)
It returns UNINFORMATIVE_TIE, not a number. The verdict names what to do instead, and the
threshold is a parameter with a strict default rather than a hidden constant.
This is the same shape as Chapter 31's evaluate_dynamical_decoupling returning NOT_EVALUABLE, and
Chapter 30's predict_survival flagging itself untrustworthy after a pinned layout. A tool that
cannot answer should say so, and say why.
The lessons
Run the classical baseline first, and tune it. It is one line, and it determines whether your result can mean anything.
Check whether the baseline already solves the problem. If it does, no comparison on that dataset will distinguish the methods, and you should choose a different dataset before running the experiment rather than after.
Report total shots. $(2n+1) \times \text{samples} \times \text{steps} \times \text{shots}$ — one line, and it is the number that decides feasibility.
Report the simulator/hardware distinction prominently. An exact simulator hides shot noise entirely, which Chapter 24 §24.3 and Chapter 27 §27.5 both measured as the dominant error term.
And notice this is the third time. Chapter 21's Grover against a strawman classical search. Chapter 24's QAOA against a proven 1995 guarantee it does not beat at executable depth. Now a classifier against a baseline that already scores 100%. The recurring failure is not measuring the alternative.
Reproduce it: code/example-02-vqc-versus-sklearn.py runs all three classical baselines before
the quantum model, then prints the training shot budget; compare_to_baseline in
code/vqelab/qml.py refuses the tie, and test_compare_to_baseline_REFUSES_an_uninformative_tie
asserts the refusal.