Case Study 1: The Result That Would Not Reproduce

The simulator result

A group implements VQE for a small molecule. On an exact simulator it works beautifully:

   VQE:    -1.857275030 Ha
   exact:  -1.857275030 Ha
   error:   8.88e-16 Ha

Machine precision. Every random start converges to the same value. The ansatz is expressive enough, the optimizer is well-behaved, and the result beats chemical accuracy by thirteen orders of magnitude.

They write it up, book hardware time, and run it.

The hardware result is off by tens of milli-Hartree and varies between runs.

The first diagnosis, and why it is incomplete

"It is hardware noise." Reasonable, and they respond with everything Part II taught:

Chapter 12's layout scoring — they select the best qubit pair by calibration data rather than accepting the default. Genuine improvement.

Chapter 13's readout mitigation — 64% of the readout error removed.

Chapter 13's ordering — readout mitigation first, then ZNE. Another improvement.

It is still not within chemical accuracy, and it still varies between runs.

At this point the natural conclusion is that current hardware simply is not good enough. That conclusion happens to be true, and they have not yet identified the largest term.

The measurement they had not taken

Take the optimal parameters — the answer already found on the simulator — and evaluate the energy with a finite shot budget. On a noiseless simulator. No hardware, no decoherence, no readout error.

Averaged over 25 independent repetitions:

       shots   mean |error|         std   0.5/sqrt(N)   chem acc?
         100       2.09e-02    1.48e-02      5.00e-02          NO
       1,000       6.52e-03    4.57e-03      1.58e-02          NO
      10,000       1.75e-03    1.04e-03      5.00e-03          NO
     100,000       6.80e-04    5.45e-04      1.58e-03         yes
       exact       8.88e-16           -             -         yes

At the exactly-correct parameters, with no noise of any kind, ten thousand shots is not enough.

They had been running at 1,000 shots per evaluation — where the mean error is 6.5 mHa, four times chemical accuracy, before hardware contributes anything.

And look at the standard deviation. At 1,000 shots it is 4.6 mHa, comparable to the mean. That is the "varies between runs" they had attributed to drifting calibration. It was sampling.

Why this was easy to miss

The simulator result had no shot noise at all. An exact simulator computes $\langle\psi|H|\psi\rangle$ by linear algebra — there is no sampling, so the entire error term they were about to encounter was invisible during development.

Shot noise looks like hardware noise. Both produce run-to-run variation and a bias away from the true value. Chapter 12 §12.7 step 2 — does it fail in noiseless simulation? — is the discriminator, and it works here only if you run the noiseless simulation with a realistic shot count. A noiseless exact simulation hides the problem completely.

This is a new variant of a familiar failure. Chapter 12's decision procedure asks whether a failure survives the removal of noise. Shot noise is not "noise" in that sense — it survives the removal of all hardware noise and disappears only when you remove sampling, which the default simulator does silently.

The corrected check has three levels, not two:

text exact simulator -> tests the ALGORITHM noiseless + finite shots -> tests the SHOT BUDGET noisy + finite shots -> tests the HARDWARE

Skipping the middle level is what happened here, and it is the default.

The arithmetic they should have done first

$$N \sim \left(\frac{\sigma}{\epsilon}\right)^2$$

For chemical accuracy with $\sigma \approx 0.5$:

$$N \approx \left(\frac{0.5}{0.0016}\right)^2 \approx 97{,}656 \text{ shots per evaluation}$$

which matches the measurement — 100,000 is the first row to clear the threshold.

And that is one evaluation. The full run:

   params   iterations   shots/eval        TOTAL SHOTS     QPU time
        4          200       97,657        175,782,600       4.9 hr
       20          200       97,657        800,787,400      22.2 hr
       50          200       97,657      1,972,671,400     2.3 days
      100          200       97,657      3,925,811,400       4.5 days

Their four-parameter ansatz needed 176 million shots and about five hours of continuous QPU time for one molecule at one geometry — and they had budgeted a few thousand shots per evaluation.

This calculation takes two minutes and requires no hardware. It would have set the expectation correctly before any time was booked.

What the mitigation stack was actually doing

Nothing they did was wasted — but the ordering of effort was inverted.

They optimized the second-largest term first, because it was the one they had a name for.

And the two interact. Mitigation costs shots — readout mitigation needs calibration circuits, ZNE needs 3–5× the circuits (Chapter 13 §13.9). So applying the full stack at a fixed total budget reduces the shots available per evaluation, which increases the shot noise. It is possible to mitigate your way to a worse answer if the budget is fixed and shot noise already dominates.

The lessons

An exact simulator hides the largest error term in a variational algorithm. Development on one tests the algorithm and nothing about the budget.

Test at three levels, not two. Exact → noiseless-with-shots → noisy-with-shots. The middle level is the one that is skipped and the one that carries the dominant term here.

Shot noise is indistinguishable from hardware noise by inspection, and both produce run-to-run variation. The standard deviation of repeated evaluations tells you which — if it matches $\sigma/\sqrt N$, it is sampling.

Do the $1/\epsilon^2$ arithmetic before booking hardware. Two minutes, no QPU, and it sets the expectation correctly.

Measure the terms before optimizing them. They applied three real techniques to the second-largest error source. Chapter 13 §13.9's procedure opens with "diagnose first", and this is what happens when that step is skipped — the fifth time in this book that a correct technique was applied to the wrong dominant term.

And mitigation can make things worse at a fixed budget, because it spends the shots that precision needs.

Reproduce it: code/example-02-the-shot-budget.py produces the table and the full bill; shot_budget() in code/project-checkpoint.py does the arithmetic before you run anything.