Case Study 1: The Result That Was a Lucky Split

A genuinely good idea, well executed

A researcher is working on data re-uploading. The technique is elegant, the universality result is proven, and the practical question is real: does re-injecting the data actually beat a conventional multi-qubit encoding?

They do the work properly. They avoid Chapter 32's mistake and choose a dataset with room in it — moons at noise 0.30, where classical models spread from 0.768 to 0.949. They implement both models, train both with the same optimizer for the same number of steps, and evaluate on the same held-out split.

   model                                   params   qubits    test
   data re-uploading, 1 qubit, 2 layers         6        1   0.9091
   VQC, 4 qubits, 2 layers                     24        4   0.7778

One qubit with six parameters beats four qubits with twenty-four, by 0.13.

That is a striking, theoretically-motivated, correctly-measured result. The dataset was chosen well. The comparison is like-for-like. Every number is right.

The test set has 99 samples

A gap of 0.1313 on 99 test points is thirteen samples.

Chapter 27 §27.5 established what a small sample does to an estimate — a 2/200 measurement of a rate near 0.15%. Chapter 28 §28.4 found two circuits agreeing and concluded, wrongly, that two optimization levels were identical. Both failures had the same fix: run it again.

Ten independently generated datasets, ten independent splits:

   model                          mean      std     min     max
   kNN                          0.8970   0.0377  0.8283  0.9495
   SVC                          0.8889   0.0310  0.8283  0.9394
   LogReg                       0.8414   0.0463  0.7677  0.9091
   1-qubit reupload (2L)        0.8343   0.0407  0.7677  0.9091
   4-qubit VQC (2L)             0.8142   0.0336  0.7778  0.8889

   1-qubit - 4-qubit VQC  = +0.0202 +/- 0.0170   NOT SIGNIFICANT at 2 sigma

The 0.13 gap becomes $+0.0202 \pm 0.0170$.

And the mechanism is visible in the min/max columns:

   the 1-qubit model's result on split 0:  0.9091  =  its MAXIMUM over ten splits
   the 4-qubit model's result on split 0:  0.7778  =  its MINIMUM over ten splits

The single split paired one model's best result with the other's worst. Not by anyone's choice — split 0 is random_state=0, the first thing anybody types. It was simply the draw that came up.

🔬 Honest Assessment: a result from one split is a draw from a distribution, not a measurement.

The standard deviations are around 0.04, so a difference of 0.13 between two single draws is about two standard deviations of each — an unremarkable outcome that looks like a large effect.

The correct report is not the mean. It is the mean with an error bar over independent replicates, and the error bar here covers zero.

What survived

The interesting part: something real is left.

   1-qubit reupload:  6 parameters,  1 qubit
   4-qubit VQC:      24 parameters,  4 qubits
   difference in accuracy: +0.0202 +/- 0.0170  (not significant)

Indistinguishable accuracy at a quarter of the parameters and a quarter of the qubits.

On a device where qubits are the scarce resource and two-qubit gates dominate the error budget — Chapter 12's 288× spread, Chapter 29's routing overhead, Chapter 31's coherence budget — that is exactly the trade you want. A model that matches on a quarter of the hardware is a better model for this hardware.

It is a weaker claim than "one qubit beats four," and it is the one the data supports. It is also more useful, because it is about resources rather than a percentage point.

What the write-up should have said

Not "a single qubit outperforms a four-qubit classifier." Instead:

"Across ten independent splits, a single-qubit data re-uploading classifier matches a four-qubit variational classifier to within $+0.0202 \pm 0.0170$, using a quarter of the parameters and a quarter of the qubits. Both are outperformed by k-nearest-neighbours by $+0.0626 \pm 0.0067$."

Three sentences, every claim supported, and the resource result — which is the genuinely valuable one — survives intact.

The project module's refusal

compare_models will not compare two models from a single split:

if n < min_splits:
    return SplitComparison(
        ..., significance=Significance.INSUFFICIENT_REPLICATES,
        reason=(f"{n} split(s) is not a measurement, it is a draw from a "
                f"distribution. Section 33.3 saw a 0.13 gap on one split become "
                f"+0.0202 +/- 0.0170 across ten. Use at least {min_splits} "
                f"independent datasets AND splits, and report an error bar."),
    )

Note "independent datasets AND splits." Re-splitting the same 300 points ten ways is better than one split and still shares the sample; the measurement above regenerates the data at each seed.

The lessons

Run it again. One split is a draw. Five is a minimum, ten is comfortable, and it costs seconds when the model trains in under a second.

Report the error bar, not the difference. $+0.13$ and $+0.0202 \pm 0.0170$ are different kinds of statement, and only the second is a claim.

Check whether your headline split is an extreme. The min and max columns take one line and would have caught this immediately — split 0 was simultaneously one model's best and the other's worst.

Vary the data, not just the split. Independent datasets test the model; independent splits of one dataset test the split.

And notice what survived. The weaker, resource-based claim is fully supported and more useful than the headline would have been. Replication does not usually destroy a result. It usually reveals what the result actually was.

Reproduce it: code/example-02-one-qubit-against-four.py runs the single split, prints the striking gap, then runs ten and prints the collapse; compare_models in code/vqelab/classifiers.py refuses fewer than five, and test_the_single_split_result_does_not_replicate asserts both that the gap was 0.1313 and that split 0 was the 1-qubit maximum paired with the 4-qubit minimum.