Case Study 1: The Result That Wouldn't Reproduce

"It worked on the old cluster. That is the whole problem."

Executive Summary

A colleague published a diffusion result last year — a steady-state temperature at a probe point, quoted as $47.31°$ — from a Fortran solver much like yours. The department has since upgraded its cluster, and now the same code, on the same input, prints $47.28°$. The lab's one regression test, which compared the output field bit-for-bit against a stored golden file, is bright red. Someone suspects a bug crept in. You are asked to find it.

There is no bug. What you will find instead is a reproducibility failure: the original run's build and environment were never recorded, so a routine upgrade silently changed the last digits of a non-associative floating-point computation — a change that is not an error but that a bit-for-bit test cannot tell apart from one. This case study is the diagnosis, and its lesson is the chapter's spine: a result you cannot reproduce is not yet a result, and the fix is to record provenance and to compare numbers the way numbers deserve — within a tolerance.

Skills applied: distinguishing bit-for-bit from tolerance comparison (§37.2); reading a regression failure that is not a bug; recording run provenance for reproducibility (§37.5); the compiler-matrix value of CI (§37.3). It leans on floating-point non-associativity from Chapter 20, compiler flags from Chapter 30, and reduction ordering from Chapter 33.

Background

The published number came from a run whose command line no one saved. The lab notebook says only "compiled with optimizations, run on 32 cores." The regression test lives in test/ and looks reasonable at a glance:

! the lab's ONE regression test -- and the source of the false alarm
if (all(field%u == golden)) then
  print '(a)', 'PASS regression'
else
  error stop 1
end if

It asserts the field is bit-for-bit identical to a stored golden file. On the machine and build where the golden was captured, it passed. Everywhere else, eventually, it fails — and today it is failing. Your job is to decide whether $47.28°$ is a broken $47.31°$ or an equally-correct one.

Phase 1 — What Actually Changed?

Start where §37.5 says: enumerate everything that could move the numbers, and check what was recorded. It is a short, damning list.

Only the code was pinned. Everything about how it was built and run — the part §37.2 warned changes the last bits — was left to memory. The git tag proves the source is unchanged, which is the crucial clue: if the source is identical and the answer moved, the cause is not in the code. It is in the build or the environment.

Phase 2 — The Mechanism: Floating-Point Is Not Associative

Before accusing the upgrade, reproduce the kind of thing that is happening, in ten lines you fully control. The root cause is the fact from Chapter 20 that floating-point addition is not associative: the order of a sum can change its result. Here is the cleanest possible demonstration, with values chosen so you can compute both answers by hand:

program not_associative
  use, intrinsic :: iso_fortran_env, only: dp => real64
  implicit none
  real(dp) :: a, b, c
  a =  1.0e16_dp
  b = -1.0e16_dp
  c =  1.0_dp
  print '(a, f6.1)', '(a + b) + c = ', (a + b) + c     ! group one way
  print '(a, f6.1)', 'a + (b + c) = ',  a + (b + c)    ! group the other
end program not_associative

$ gfortran -std=f2018 -Wall not_associative.f90 -o na && ./na
(a + b) + c =    1.0
a + (b + c) =    0.0

Work it by hand and feel the hazard. Grouping left, $a + b = 0$ exactly, then $0 + 1 = 1$. Grouping right, $b + c = -10^{16} + 1$, but $10^{16}$ is larger than $2^{53}$, so its representable neighbours are $2$ apart and the $+1$ vanishes in the rounding — $b + c$ rounds back to $-10^{16}$, and $a + (b+c) = 0$. Same three numbers, same hardware, and the answer is $1.0$ or $0.0$ depending only on the order of operations. That is not a bug in either grouping; it is the arithmetic. Now scale the intuition down to the last-digit level of a real reduction, and you have the diffusion result's $47.31$ versus $47.28$.

Phase 3 — Pin the Cause

With the mechanism in hand, the specific culprit is easy to name, because two of the unrecorded items reorder exactly this kind of sum:

• The flags. The old build used -Ofast (the notebook's "with optimizations"), which — unlike -O2 — permits the compiler to reassociate floating-point arithmetic (Chapter 30). The new cluster's default build script uses plain -O2. Same source, different grouping of the solver's sums, different last digits.
• The core count. The probe temperature is a global reduction over the plate (a sum of contributions), and a parallel reduction(+:...) (Chapter 33) combines partial sums in an order that depends on how many threads there are. The old run used 32 cores; the new one used 48. Different partition, different summation order, different last digits.

Neither is an error. Both runs solve the same discretized equations to the same accuracy; they differ by an amount on the order of the accumulated round-off, which for a long-running reduction can reach the third or fourth significant digit. The $0.03°$ discrepancy is within the numerical uncertainty of the method itself. The bug is not in the solver. The bug is in the test, which demanded bit-for-bit identity across a changed build and mistook a rounding difference for a regression.

Sanity check. Confirm the discrepancy is round-off-scale, not physics-scale. The probe sits in a field ranging $0$–$100°$; a genuine bug (a wrong stencil, a sign error) would move it by whole degrees, not hundredths. A $0.03°$ shift on a $47°$ value is a relative change of $\sim 6\times10^{-4}$ — far larger than machine epsilon but entirely consistent with reassociated reductions over a large grid. It smells of arithmetic order, not of broken physics, and Phases 2–3 confirm the nose.

Phase 4 — Fix the Test

The regression test must survive a legitimate change of build. Replace bit-for-bit identity with a tolerance chosen from the problem, not from hope:

! the fixed regression test: tolerance, not bit-for-bit
real(dp) :: maxdev
maxdev = maxval(abs(field%u - golden))
if (maxdev < 1.0e-2_dp) then                 ! 0.01 deg: far below physics scale, above round-off
  print '(a, es10.3)', 'PASS regression, maxdev = ', maxdev
else
  print '(a, es10.3)', 'FAIL regression, maxdev = ', maxdev
  error stop 1
end if

The tolerance $10^{-2}$ degrees is deliberate: it is far below the scale at which a real bug would show (whole degrees) yet comfortably above the round-off from reassociated reductions (thousandths). A test with that tolerance passes for both the 32-core -Ofast run and the 48-core -O2 run — because both are correct — and would still fail loudly if someone broke the stencil. This is the §37.2 lesson made concrete: reserve bit-for-bit for a pinned configuration, and for a portable test compare within a physically meaningful tolerance.

Phase 5 — Restore Reproducibility, and Report

The deeper fix is to stop losing provenance. Going forward, every run stamps its own build and environment into its output header — the §37.5 discipline — so "which build?" is never again a question of memory:

# run_provenance.txt  (written automatically by the driver)
code commit : 9f3a1c2         (git rev-parse HEAD)
compiler    : GCC 13.2.0      (compiler_version())
options     : -O2 -std=f2018  (compiler_options())
cores       : 48
seed        : (deterministic; solver is not stochastic)

Your report to the colleague is now precise and reassuring: the solver is correct; $47.28°$ and $47.31°$ are the same result computed with different arithmetic ordering, caused by an unrecorded flag change (-Ofast to -O2) and an unrecorded core-count change (32 to 48), neither of which is a bug. The bit-for-bit regression test was the false alarm and has been replaced with a tolerance test. From now on, each run records its build, so the next upgrade will be a non-event. Had the lab run the §37.3 compiler-matrix CI from the start, the bit-for-bit test would have gone red on the second compiler the day it was written — surfacing the fragility immediately, long before it masqueraded as a regression in a published number.

Discussion Questions

1. The git tag proved the source was unchanged, which was the decisive clue. Explain why "same source, different answer" points immediately away from the code and toward the build or environment — and why a code without a committed tag would have left you unable to make that inference.
2. The chosen tolerance was $10^{-2}$ degrees. Argue for why a tolerance must be set from the problem (physics scale versus round-off scale), and describe the failure mode of a tolerance set too loose — say $10^{2}$.
3. The colleague protests: "But bit-for-bit is the strictest test — surely stricter is safer!" Rebut this using the "cry wolf" pitfall of §37.2. When is stricter genuinely better?

Your Turn: Extensions

• Option A. Run not_associative.f90 yourself, then modify it to sum an array [1.0e16, 1.0, 1.0, -1.0e16] two ways — left-to-right with a loop, and as sum(x) (which the compiler may reassociate) — and report whether they agree. Then rebuild with -Ofast and compare again. You are reproducing the case study's mechanism in miniature.
• Option B. Write the driver code that produces a run_provenance.txt like Phase 5's, using compiler_version() and compiler_options() and a git_commit parameter. Have it print the provenance header before the results, so every output file is self-documenting.
• Option C. Take the lab's bit-for-bit test and convert it into a two-tier test: a strict bit-for-bit check guarded by a "pinned build" flag (compiler, flags, and core count all matching recorded values), falling back to the tolerance test otherwise. State when each tier should run in CI.

Key Takeaways

• "Same source, different answer" is a provenance failure, not necessarily a bug. Pin the code with a commit hash so you can make that inference; then look to the build and environment, not the source.
• Floating-point is not associative, so a changed flag (-Ofast reassociates), a changed core count (reorders a reduction), or a changed library can move the last digits of a correct computation.
• A bit-for-bit regression test across an unpinned build cries wolf — it fails on legitimate changes until people ignore it. Compare within a tolerance set from the problem: below the physics scale, above the round-off scale.
• Record provenance automatically. Compiler, version, flags, core count, seed, and commit, stamped by the run itself, turn the next upgrade from a mystery into a non-event — the difference between a result and an anecdote.