Case Study: Benchmarking a Quantum Classifier Honestly

Executive Summary

A team reports a variational quantum classifier achieving 94% accuracy on a binary classification task and presents it as evidence of quantum machine learning's promise. The number is real. The evidence it constitutes is close to zero, because the baseline was never established.

This case study runs the comparison that should have accompanied the claim: same data, same splits, against logistic regression, a support vector machine, and a small neural network. Then it examines what happens as the dataset grows. The pattern that emerges — competitive on tiny data, falling behind immediately as data scales — is the characteristic signature of current QML results.

Skills applied

• Constructing appropriate classical baselines for a learning task (§21.14).
• Measuring accuracy against training-set size.
• Accounting for the true cost of quantum model evaluation.
• Distinguishing model expressiveness from trainability and generalization.

Phase 1: The task and the quantum model

Data: binary classification, 8 features, class-balanced. The team used 60 training and 40 test points.

Model: angle encoding of 8 features onto 8 qubits, followed by 3 layers of parameterized rotations and CNOT entanglers — 48 trainable parameters. Trained with parameter-shift gradients and Adam.

Reported result: 94% test accuracy.

Phase 2: The baselines nobody ran

Identical data, identical splits, five-fold cross-validation:

The quantum classifier is not the best model and takes roughly six orders of magnitude longer to train. Logistic regression — nine parameters, three milliseconds — is within 1.5 points.

First finding. On a 60-point dataset, essentially every model achieves 92–95%, because the task is easy and the data is small. Reporting 94% without baselines communicates nothing about the method.

Phase 3: Scale the data

The revealing experiment is varying training-set size:

Two clear patterns.

The quantum model wins at 20 points. With 48 parameters and a restricted hypothesis class, it is well regularized and resists overfitting where the MLP struggles. This is a genuine, reproducible effect — and it is an argument about model capacity on small data, not about quantum computation. A comparably regularized classical model shows the same behaviour.

It stops improving from ~200 points. Classical models keep gaining as data grows; the VQC plateaus at ~94%. Its limited parameter count and expressiveness cap it, and adding layers to increase capacity runs into barren plateaus and noise.

At 5,000 points the VQC is simply infeasible: the parameter-shift rule needs $2\times48 = 96$ circuit evaluations per gradient step, times thousands of steps, times shots per evaluation.

Phase 4: The cost accounting nobody publishes

Per gradient step:

$$96 \text{ circuits} \times 1{,}024 \text{ shots} = 98{,}304 \text{ circuit executions}$$

At 2,000 steps: $\approx 2\times10^{8}$ executions. On hardware at ~1,000 executions/second including overhead: ~55 hours — for a model that logistic regression beats within 1.5 points in 3 milliseconds.

The structural issue is that classical backpropagation computes all gradients in one backward pass, while parameter-shift requires two circuit evaluations per parameter. Gradient cost is $O(1)$ passes classically and $O(P)$ evaluations quantumly. This gap is not an engineering deficiency — measurement destroys the state and no-cloning forbids storing intermediate activations, so backprop as practiced classically is unavailable in principle.

Phase 5: How the claim should have been made

Not defensible: "Our quantum classifier achieves 94% accuracy, demonstrating QML's potential."

Defensible: "On a 60-point, 8-feature binary task, a variational quantum classifier achieved 94.0% ± 2.1% test accuracy, compared with 95.0% for an RBF-kernel SVM and 92.5% for logistic regression. At 20 training points the quantum model outperformed all classical baselines (86.5% vs. 85.5% best classical), consistent with its restricted hypothesis class acting as regularization. Performance plateaued beyond 200 training points while classical models continued improving. Training required ~$2\times10^{8}$ circuit executions versus milliseconds classically. We do not claim advantage."

The second version is a more useful scientific contribution than the first, because it identifies a specific regime and a specific mechanism rather than an unsupported general claim.

Phase 6: The benchmarking checklist

1. Always report classical baselines — at minimum logistic regression, SVM, and a small MLP, tuned with the same effort as the quantum model.
2. Report the trivial baseline (majority class). Many published QML accuracies are close to it.
3. Vary training-set size — the scaling curve is far more informative than any single number.
4. Report total circuit executions and wall-clock, not just accuracy.
5. Use standard splits and cross-validation; small test sets have enormous variance (40 points → ±8% at one sigma).
6. State what is being claimed. "Competitive on small data" is defensible and interesting; "demonstrates quantum advantage" is not.

Discussion Questions

1. The VQC won at 20 training points. Is that a quantum effect? Design an experiment to find out.
2. Parameter-shift costs $O(P)$ evaluations against backprop's $O(1)$ passes. Why is this structural rather than an engineering gap?
3. A 40-point test set gives ±8% standard error. How many of the QML papers you have seen report error bars?
4. What experimental result would genuinely demonstrate a QML advantage? Be specific about task, data, and baseline.

Your Turn: Extensions

• Reproduce this comparison with scikit-learn and a Qiskit variational classifier on a synthetic 8-feature dataset.
• Plot accuracy against training-set size for all four models and identify the crossover.
• Add a classically regularized model with 48 parameters and check whether it reproduces the small-data advantage.
• Count circuit executions for one full training run and convert to hardware wall-clock.

Key Takeaways

• An accuracy number without baselines is not a result; on small easy datasets nearly every model scores similarly.
• Current quantum classifiers can win on very small training sets through implicit regularization — a capacity effect, reproducible classically, not evidence of quantum advantage.
• They plateau as data grows while classical models keep improving, and become infeasible at moderate dataset sizes.
• Parameter-shift gradients cost $O(P)$ circuit evaluations against backprop's $O(1)$ passes, a structural consequence of measurement and no-cloning.
• Report baselines, scaling curves, error bars, and total executions — and state precisely what is and is not being claimed.