Case Study 30.2: A Reproducible, Portable Release Build for the Solver

"An experiment is reproducible until another laboratory tries to repeat it." — Alexander Kohn (widely quoted; treat the wording as illustrative)

Executive Summary

Case Study 30.1 was diagnostic: you took a dishonest number apart. This one is constructive: you will design the build so the number is honest from the start. The heat solver is finished and fast on your machine. Now it must become a piece of software that another group — on a different CPU, with a different compiler — can build, run, and get the same science from. That is a design problem, and its ingredients are exactly this chapter's: two build profiles, an explicit floating-point contract across compilers, self-recording provenance, a defensible position on profile-guided optimization, and a manifest that ties a result to the recipe that produced it.

You will produce (1) a portable profile table that maps "debug" and "release" onto gfortran, Intel ifx, and NVIDIA nvfortran; (2) a compilable build_info module the solver uses to stamp every run; (3) a cross-compiler agreement test that checks results to a tolerance rather than demanding bit-exactness; and (4) a decision on PGO with the reasoning written down. No solver physics changes — this is entirely about building it right.

Skills applied - Designing debug and release profiles and mapping them across compilers (§30.2, §30.3). - Setting the floating-point knob explicitly so compilers agree (§30.2 Intel default; §30.5). - Stamping a binary with its own provenance for reproducibility (§30.4). - Judging profile-guided optimization on cost/benefit (§30.4). - Trading peak speed for portability with -march=native (§30.5).

Background

The solver builds cleanly under gfortran with -O3 -march=native -flto, validated against the analytical steady state to a tight tolerance. A partner lab will run it on an Intel cluster with ifx, and a third collaborator wants to try the NVIDIA compiler as a step toward GPU work later (Chapter 35). Three compilers, three CPUs, one source, one science. The danger is not that the code fails to compile — it is standard Fortran 2018 and it will — but that it compiles into three subtly different programs because each compiler's defaults differ, and nobody wrote the differences down.

Phase 1 — Define two profiles, portable across three compilers

The foundation is the two-build discipline of §30.3, generalized so that "debug" and "release" mean the same intent on every compiler even though the spellings differ. We write it as a table the whole project agrees on:

Two design decisions are encoded here. First, every profile names an explicit floating-point contract, because §30.2 showed that Intel defaults to a relaxed model — so -fp-model precise is not optional padding, it is what makes ifx agree with gfortran's default. (The gfortran column is blank for the FP row only because its default already is strict; we still state that fact in the manifest so no reader has to know it.) Second, the release row is where portability bites: -march=native/-xHost bakes in the build machine's instruction set, which is fine if we build on each target and a bug if we ship one binary everywhere. We resolve that in Phase 5.

A caution on the specifics: the exact ifx and nvfortran spellings above (-check all, -Mbounds, -Kieee, -fast) are given as orientation from the vendors' documentation and their PGI lineage; they have shifted across versions. Verify each against the compiler you actually have — the table's structure (debug / release / FP-contract, one column per compiler) is the durable design; the cells are details to confirm. This is flagged as version-dependent.

Phase 2 — Make the binary record its own build

Reproducibility begins the moment a run can say how it was built. We give the solver a small module whose one job is to stamp provenance into any output stream — the terminal, a log file, or the header of a results file.

module build_info
  use, intrinsic :: iso_fortran_env, only: output_unit, &
                                           compiler_version, compiler_options
  implicit none
  private
  public :: stamp_build

contains

  ! Write the compiler and flags that built this binary to `unit` (default: stdout).
  subroutine stamp_build(unit)
    integer, intent(in), optional :: unit
    integer :: u
    u = output_unit
    if (present(unit)) u = unit
    write(u, '(a)')  '# --- build provenance ---'
    write(u, '(2a)') '# compiler: ', compiler_version()
    write(u, '(2a)') '# options : ', compiler_options()
  end subroutine stamp_build

end module build_info

program run_with_stamp
  use build_info,  only: stamp_build
  use, intrinsic :: iso_fortran_env, only: dp => real64
  implicit none
  real(dp) :: result

  call stamp_build()                    ! header for every run's log
  result = 3.0_dp * 4.0_dp + 1.0_dp     ! stand-in "science": 13.0, deterministic
  print '(a, f8.2)', 'result = ', result
end program run_with_stamp

$ gfortran -std=f2018 -Wall -O3 -march=native -flto build_info.f90 -o stamped && ./stamped
# --- build provenance ---
# compiler: GCC version 13.2.0
# options : -std=f2018 -Wall -O3 -march=native -flto
result =    13.00

The result line is deterministic — $3 \times 4 + 1 = 13$ — so it reads 13.00 on every correct build, under every compiler. The two # lines are environment-dependent by design: run the same program built by ifx and they will name ifx and its flags instead. Every log file the solver writes now begins with the recipe that produced it. When the partner lab emails you a strange result, the first three lines of their log tell you exactly how their binary was built — often solving the mystery before you read the fourth line.

Phase 3 — Cross-compiler agreement: to a tolerance, not the last bit

Here is the design decision that separates people who have shipped portable numerical code from those who have not: do you require the three compilers to agree bit-for-bit, or only to a scientific tolerance?

Demanding bit-exactness across compilers and CPUs is usually a mistake. As §30.1's FMA note explained, a build that fuses a*b + c into one rounding will differ in the last bit from one that does not — and that is better arithmetic, not a bug. Chase bit-exactness and you must forbid FMA, forbid -march=native, and pin one compiler, throwing away real speed to make a checksum match. The right target for a physics result is agreement to a tolerance you justify from the problem, not from the hardware.

So the validation harness compares each build's result against a reference within a relative tolerance:

program agree
  use, intrinsic :: iso_fortran_env, only: dp => real64
  implicit none
  real(dp) :: gfort_val, ifx_val, tol

  ! Same source, two compilers: last-bit differences from FMA / instruction choice.
  gfort_val = 2.7182818284590452_dp
  ifx_val   = 2.7182818284590455_dp
  tol       = 1.0e-12_dp                       ! justified by the problem, not the hardware

  if (abs(gfort_val - ifx_val) <= tol * abs(gfort_val)) then
    print '(a)', 'builds AGREE within tolerance'
  else
    print '(a)', 'builds DIFFER -- investigate flags'
  end if
end program agree

The two values differ by about $3\times10^{-16}$; the tolerance is $10^{-12}\times 2.718 \approx 2.7\times10^{-12}$. Since $3\times10^{-16}$ is far smaller, the builds agree within tolerance:

$ gfortran -std=f2018 -Wall -O2 agree.f90 -o agree && ./agree
builds AGREE within tolerance

This is the regression-test pattern of Chapter 37 in miniature: pin the science to a tolerance, let the last bits float. If instead the harness printed DIFFER, that is your signal to hunt for a flag mismatch — most likely a compiler running its relaxed FP default when you meant strict, exactly the Intel gotcha from §30.2.

Design rule. Cross-compiler portability means agreement to a stated tolerance with an explicit FP contract, not identical bits. Write the tolerance into the test; write the FP flag into the profile.

Phase 4 — Should this project use PGO?

Profile-guided optimization is on the table because a collaborator read that it "can add 10–20%." We make the decision the way §30.4 says to: by asking where the solver's time goes.

The hot path is the five-point stencil — a branch-free sweep of arithmetic over contiguous arrays. PGO's strengths (branch layout, hot/cold splitting, call-frequency-driven inlining) have almost nothing to work with there. The branchy part of the solver is the per-step boundary-condition dispatch and the occasional I/O checkpoint, which together are a sliver of the runtime. So the expected PGO payoff is small, and it comes at the cost of a three-step instrument-run-rebuild pipeline that every collaborator would have to reproduce — multiplying the build complexity across three compilers.

Decision: defer PGO. We record the reasoning so it is a decision, not an omission:

PGO decision (2026-07): DEFERRED.
Rationale: runtime is dominated by a branch-free stencil; PGO chiefly helps branchy
control flow, of which the solver has little. Expected gain < ~5%, against a 3x build
pipeline replicated over three compilers. Revisit IF profiling (Chapter 28) later shows
a branchy hot spot, e.g. after adaptive time-stepping or per-cell material models land.

That paragraph is worth more than a hasty -fprofile-use: the next person who wonders "why aren't we using PGO?" gets an answer, and a trigger for when to reconsider.

Phase 5 — The manifest and the shipped build

Finally, portability of the distributed artifact. Because release uses -march=native/-xHost, we do not ship one binary. Instead we ship the source plus an fpm-style profile the user opts into (Chapter 16), so each site compiles for its own CPU, and we ship a manifest that pins everything a result depends on:

Heat solver — reproducibility manifest
---------------------------------------
Source     : git commit <hash>, Fortran 2018
Profiles   : debug / release / FP-contract per the Phase 1 table (one column per compiler)
Release    : gfortran -O3 -march=native -flto  |  ifx -O3 -xHost -ipo -fp-model precise
FP contract: IEEE-strict on every compiler (gfortran default; ifx via -fp-model precise)
Build rule : compile release ON the target node (so -march=native/-xHost is safe)
Validation : agree-to-tolerance vs analytical steady state, tol = 1e-12 relative
Provenance : every run stamps compiler_version()/compiler_options() (Phase 2)
PGO        : deferred, with recorded rationale (Phase 4)

The two load-bearing lines are "compile release ON the target node" — which turns -march=native from a portability trap into a portability feature — and "IEEE-strict on every compiler," which is the one sentence that makes three compilers compute the same science. Everything else is bookkeeping, and the bookkeeping is what makes it reproducible.

Discussion Questions

1. Why does naming an explicit floating-point contract matter more for a multi-compiler project than for a gfortran-only one? What single default would silently break agreement if left unstated?
2. The design ships source plus a profile rather than a prebuilt binary. What did we gain, and what did we ask of the user in return? When would shipping a conservative--march binary be the better trade?
3. Phase 3 chose agreement-to-tolerance over bit-exactness. Construct a scenario (hint: a chaotic or ill-conditioned system) where even a tolerance-based test is hard to satisfy across compilers, and say what you would do about it.
4. The PGO decision was "defer, and here is the trigger to revisit." Why is a recorded deferral more valuable to a future maintainer than simply not mentioning PGO at all?

Your Turn: Extensions

Option A (design). Extend the Phase 1 profile table with a fourth column for the LLVM flang compiler. Research (or reason about) its optimization, bounds-checking, and floating-point flags, and mark which cells you are confident about versus which you would verify before trusting. Then state how adding flang to CI would harden the project.

Option B (build). Turn the build_info module into a write_results_header procedure that stamps provenance plus the run's grid size, step count, and a timestamp into the header of the solver's output file — so a results file is self-describing. Sketch the interface and the exact lines it would write.

Option C (optimize). Suppose profiling later reveals that a new adaptive-time-stepping feature added a branchy hot spot (the PGO trigger from Phase 4 has fired). Design the three-command gfortran PGO build for it, define what "representative input" means for a heat run with adaptive stepping, and describe the before/after measurement that would justify keeping PGO in the release profile.

Key Takeaways

• Portability is a design, not an accident: one profile table, mapped across compilers, with an explicit floating-point contract in every column.
• State the FP contract explicitly on each compiler — especially Intel, whose default is relaxed. This one flag is what makes different compilers compute the same science.
• Stamp every binary with its own provenance (compiler_version/compiler_options) so every result carries its recipe.
• Aim for agreement to a justified tolerance, not bit-exactness; chasing identical bits forbids FMA and -march=native and costs real speed for no scientific gain.
• Ship source plus an opt-in profile, and compile -march=native release builds on the target node — turning the portability trap into a portability feature.
• Decide PGO on evidence, and record the decision (including a deferral and its trigger). A written "no, because…" is worth more than silence.