Case Study 30.1: The Benchmark That Lied

"The first principle is that you must not fool yourself — and you are the easiest person to fool." — Richard Feynman

Executive Summary

A research group announces, in a group meeting, that they have "optimized the heat solver by 24×." The slide shows two numbers: an old run at 48.0 s and a new run at 2.0 s. The result is greeted with applause and a plan to submit it as a methods note. You are asked to help write it up — which means you are the first person who has to make the number reproducible, and the moment you look at how it was produced, the 24× begins to dissolve.

This case study is a forensic exercise in reading a build. You will take a headline speedup apart into its honest components, discover that most of it came from removing safety checks rather than from optimization, confirm (using a checksum) that the fast build still computes the right answer, and rebuild the claim as something a reviewer can trust. Nothing here changes the solver's source; everything here is about the flags and the discipline of measuring.

Skills applied - Distinguishing a development build from a release build, and why timing the former misleads (§30.3). - Attributing a speedup to specific flags — checks, -O level, -march=native, -flto (§30.1, §30.3). - Verifying that optimization preserved results, and ruling out -Ofast (§30.1). - Recording a build so the number is reproducible (§30.4).

Background

The "old" and "new" build commands, recovered from two shell-history lines, were:

# "old" (the 48.0 s baseline)
$ gfortran -std=f2018 -g -O0 -fcheck=all -fbacktrace heat.f90 -o heat_old

# "new" (the 2.0 s result)
$ gfortran -std=f2018 -O3 -march=native -flto heat.f90 -o heat_new

Look at what changed between them. It is not one thing; it is five things at once: the run-time checks were removed, -g was dropped, the optimization level jumped from -O0 to -O3, the CPU's native instruction set was enabled, and link-time optimization was turned on. A 24× ratio between these two builds is real in the sense that the stopwatch says so — but as an optimization result it is close to meaningless, because the baseline is a guarded debug build that nobody would ever run in production. The group has measured "debug versus release," dressed it up as "unoptimized versus optimized," and the difference between those two framings is the difference between an honest methods note and one that will not survive review.

Phase 1 — Reproduce, then interrogate, the baseline

The first rule of a suspicious benchmark is to reproduce it before you criticize it. Assume the 48.0 s and 2.0 s are real on the group's machine. The question is not "are the numbers wrong?" but "what do they mean?" — and the answer is in the flags.

The single most important observation is that the baseline carries -fcheck=all. As §30.3 established, those run-time checks — bounds, allocation, pointer status — can slow a loop-heavy code several-fold. A solver spends nearly all its time in array accesses inside the stencil, which is exactly what -fcheck=all instruments most heavily. So a large part of the 48.0 s is the checks, not the absence of optimization.

To interrogate the claim properly, we need to measure the intermediate builds the group skipped. We build the same source at a ladder of settings, each differing from the last by one flag, so each step attributes its own slice of the speedup.

Phase 2 — Decompose the speedup, one flag at a time

Here is the experiment: five builds, each adding one change, timed identically (warm cache, several repetitions, median reported — the methodology from Chapter 28).

All six numbers are illustrative (Tier 2). They are plausible orders of magnitude for a vectorizable stencil, chosen to show the shape of the decomposition, not measured on a specific machine. Your own ladder will differ — which is exactly why you must run it yourself and record it.

Read the table and the story rewrites itself. The honest optimization speedup — the thing a methods note should claim — is B1 → B5: from 16.0 s (no checks, no optimization) to 2.0 s (fully optimized), or 8×. The jump from B0 to B1, a full 3× of the original 24×, is not optimization at all; it is turning off the debugging safety net, which you would never leave on for a production run in the first place. The group's 24× multiplied a legitimate 8× optimization by a 3× "stopped measuring the checks" factor and reported the product as if it were all one achievement.

The rule made concrete: never let the baseline of an optimization claim be a -fcheck=all build. The apples-to-apples comparison is checks-off vs checks-off. Compare -O0 (no checks) against -O3 -march=native -flto (no checks); that ratio — here 8× — is the number you can defend.

Phase 3 — Did the fast build still get the right answer?

A speedup is worthless if the fast build computes something different. Two hazards from §30.1 must be ruled out: that -O3 somehow changed the result (it should not have), and — critically — that nobody sneaked in -Ofast (which could have). The group's -flto line does not contain -Ofast, which is reassuring, but we verify rather than trust, by having each build print a checksum of the final field.

program checksum_probe
  use, intrinsic :: iso_fortran_env, only: dp => real64, compiler_options
  implicit none
  ! Stand-in for the solver's final field: a fixed, known array whose sum we can
  ! hand-verify, so any change across builds is a red flag.
  real(dp) :: field(5) = [100.0_dp, 28.0_dp, 32.0_dp, 28.0_dp, 12.0_dp]
  print '(2a)',      'built with: ', compiler_options()
  print '(a, f9.2)', 'checksum  : ', sum(field)     ! 100+28+32+28+12 = 200
end program checksum_probe

Build and run it at every rung of the ladder:

$ gfortran -std=f2018 -Wall -O0                       checksum_probe.f90 -o c0 && ./c0
built with: -std=f2018 -Wall -O0
checksum  :    200.00
$ gfortran -std=f2018 -Wall -O3 -march=native -flto   checksum_probe.f90 -o c5 && ./c5
built with: -std=f2018 -Wall -O3 -march=native -flto
checksum  :    200.00

The checksum is 200.00 at every level (the array sums to $100+28+32+28+12 = 200$, exactly, at any optimization level — result-preserving flags do not touch it). Identical checksums across the ladder are the evidence that the 8× bought speed, not a different answer. Had any build printed a different checksum, we would have found -Ofast (or -ffast-math) hiding in the recipe and stopped the presses. Notice, too, that the probe prints its own compiler_options() — so the log itself records which build produced which checksum. That is the reproducibility habit of §30.4, applied as a verification tool.

Phase 4 — Rebuild the claim as a reproducible artifact

Now we make the number defensible. The methods note gets a build manifest, not a slide with two bare times. Everything that can change the result or the speed is written down:

Reproducibility manifest — heat solver optimization
---------------------------------------------------
Compiler   : GCC (gfortran) 13.2.0
Baseline   : -std=f2018 -O0                      (checks off; the honest baseline)
Optimized  : -std=f2018 -O3 -march=native -flto  (checks off)
Machine    : <CPU model> — REQUIRED because -march=native encodes this CPU's ISA
Grid/steps : 512 x 512, 1000 steps
Timing     : median of 5 runs, warm cache (Chapter 28 methodology)
Result     : final-field checksum 200.00, identical for baseline and optimized
Speedup    : 8x (optimization), reproducible on the machine above

The line that matters most is the CPU model, precisely because -march=native was used: the optimized binary is specific to that processor, so a reader on a different chip must rebuild rather than copy the binary. That single caveat, recorded honestly, is the difference between "our result" and "a result."

Phase 5 — The corrected announcement

The revised slide reads: "We optimized the solver 8× (-O0 → -O3 -march=native -flto, checks removed), result unchanged (checksum verified), on the recorded machine." It is a smaller headline and a far better one — because it is true, it is reproducible, and it separates the two real effects the group had conflated: removing the debug checks (a 3× you always take, and never call optimization) and optimizing the code (a genuine 8×).

Sanity check. $3\times \times 8\times = 24\times$ — the two honest factors multiply back to the original headline. The 24× was never a lie about arithmetic; it was a lie about attribution. Good benchmarking is mostly the discipline of attribution.

Discussion Questions

1. The group's baseline was the build they had been developing with. Why is it so natural — and so wrong — to reach for your development build as a benchmark baseline?
2. Suppose Phase 3 had shown the optimized build printing a checksum of 199.87 instead of 200.00. What would you look for in the build line, and what would you tell the group before they publish anything?
3. Is the 3× from removing -fcheck=all ever worth reporting? In what document does it belong, if not in an optimization result?
4. The manifest records the CPU model "because -march=native." If the group had used plain -O3 (no -march=native), would the CPU model still matter for reproducibility? For bit-exact reproducibility (recall the FMA note in §30.1)?

Your Turn: Extensions

Option A (analysis). Take the ladder in Phase 2 and compute, for each rung, its share of the honest 8× optimization speedup (B1 → B5). Which single flag contributed most on this illustrative machine? Would you expect the same ranking on a memory-bound problem? Justify with Chapter 28's compute- vs memory-bound distinction.

Option B (verification). Extend checksum_probe.f90 so it also prints a checksum under -Ofast and deliberately choose a field computation (a long alternating sum) where reassociation would change the last digits. Predict, by hand, the strict-IEEE value; then explain what you would expect -Ofast to do and why you cannot predict its exact output.

Option C (protocol). Write the group a one-paragraph "benchmarking checklist" that would have prevented this whole episode: what to record, what the baseline must be, and the one verification (checksum) that must pass before any speedup is announced.

Key Takeaways

• A speedup is a ratio between two builds; if the builds differ in more than optimization, the ratio is not an optimization result. Here, 24× = 3× (checks removed) × 8× (real optimization).
• Never benchmark a -fcheck=all build. The honest baseline is checks-off, unoptimized; the honest result is checks-off, optimized.
• Verify results across the ladder with a checksum. Identical checksums prove you bought speed, not a different answer, and immediately expose a stray -Ofast.
• Record the compiler, flags, and CPU — the CPU especially when -march=native is used. A number without its manifest is a story, not a measurement.
• The easiest person to fool is yourself; the flags are where the fooling happens.