Chapter 37 — Key Takeaways (Testing, Documentation, and Software Engineering)

A one-page reference for testing numerical code, comparing results honestly, wiring up CI, documenting with FORD, and making a result reproducible. Keep it beside you the first time you write a test for a solver.

The three kinds of test

Kind Question it answers Scope In our solver
Unit Is this piece correct in isolation? one procedure laplacian, step, stable_dt
Regression Did anything I didn't mean to change, change? whole solver vs a known-good run 5×5 two-step golden field
Verification Is it right (vs a known-true answer)? whole solver vs an analytical solution linear steady state; separable mode
  • Unit localizes a break; regression guards against change; verification catches wrongness a regression can silently preserve. A good suite has all three.

What to test in numerical code — the four oracles

You rarely know the general answer, so test against a source of truth you do know:

Oracle Example (heat solver)
Exact special case $\nabla^2(x^2+y^2)=4$; $\nabla^2(\text{linear})=0$ (stencil exact for degree ≤ 3)
Invariant / conservation maximum principle: $0 \le u \le 100$ after any CFL-safe step
Symmetry symmetric input → symmetric output (column reversal unchanged)
Convergence order error vs the analytical solution falls $\times 4$ per halving (Ch. 22)

Assertions: compare with a tolerance, never ==

if (abs(got - want) <= tol) then ...      ! RIGHT for floating point
if (got == want) then ...                  ! WRONG: rounding makes equal values differ in the last bit
  • A minimal harness is ~20 lines: assert_close / assert_true, a failure counter, and error stop 1 at the end.
  • The exit code is the contract. error stop 1 (Ch. 13) sets a nonzero exit that fpm test and CI read as failure. A printed "FAIL" that exits 0 is invisible to automation.

pFUnit — the framework

@test
subroutine test_x()
  use funit
  @assertEqual(4.0_dp, lap(2,2), tolerance=1.0e-12_dp)
  @assertLessThanOrEqual(dt/h**2, 0.25_dp)
end subroutine
  • .pf files are preprocessed into .F90 (not compiled by plain gfortran), then run through a generated driver. Understands MPI, so it can test parallel routines (Ch. 34). Named in Ch. 16; concept identical to the hand-rolled harness.

Bit-for-bit vs tolerance

Bit-for-bit (a == b) Tolerance (abs(a-b) <= tol)
Strictness exact, catches one changed bit looser, physically meaningful
Robustness fragile portable
Use when build is pinned (compiler, flags, libs, core count) everything portable — the default

Anything that reorders non-associative floating-point breaks bit-for-bit without a bug:

  • a different compiler or version · different flags (-Ofast/-ffast-math reassociate, Ch. 30) · a different BLAS/LAPACK · a different processor count (reorders a reduction, Ch. 33–34).

Continuous integration (GitHub Actions)

on: [push, pull_request]
strategy:
  matrix:
    gcc: [11, 12, 13]        # build+test on THREE compilers, in parallel
  • Every change → clean machine → build → fpm test → nonzero exit turns the job red and blocks the merge.
  • The compiler matrix is the point: catch non-portable code and compiler bugs the day they appear.
  • Keep tests fast (tiny grids — oracles are exact at any size); slow full-resolution studies run on a schedule.

Documentation (FORD) and git for science

  • FORD reads !> (before) and !! (after) doc comments — ordinary comments, so the source still compiles — and generates a cross-linked API site. Docs in the source cannot drift from the code.
  • git: tie every figure to a commit hash (git rev-parse HEAD); commit code and inputs; tag submissions; do not commit huge outputs — commit how to regenerate them.

Reproducibility — record everything that can move the answer

Record Where
code (exactly) git commit hash
compiler + version compiler_version()
flags compiler_options()
inputs committed namelist/config
library versions pinned in fpm.toml
random seed recorded with output
processor count recorded with output 
use, intrinsic :: iso_fortran_env, only: compiler_version, compiler_options   ! self-report the build
  • A fixed seed reproduces the RNG on the same compiler — but the standard doesn't fix the generator, so not necessarily across compilers.
  • "Works on my machine" = the result depends on an undocumented environment = not yet reproducible.

Project piece added this chapter

The solver's test/ is filled: unit tests of laplacian/step/stable_dt, a regression test vs the golden 5×5 field (tolerance), a verification test vs the analytical steady state, FORD doc comments on the public API, a CI matrix workflow, and a provenance stamp. Same Chapter 24 physics; now trustworthy. This is the software half of the Chapter 38 capstone.

The two things to remember

  1. Test against oracles, not "the right answer" — exact special cases, invariants, symmetry, convergence order — and compare within a tolerance, never ==.
  2. A scientific result you cannot reproduce is not a result. The deliverable is the number plus everything needed to regenerate it: tested code, recorded build, inputs, and seeds.