Case Study 2: From Script to Release — Giving the Solver a Test Suite, Docs, and CI

"The code that runs once and makes a figure is a draft. The code anyone can trust is a release."

Executive Summary

Chapter 36 turned your heat solver into a package — src/, app/, an empty test/. This case study fills the engineering in: you will design and build a real test suite for it, document its public interface with FORD, wire up continuous integration across a compiler matrix, and make each run stamp its own provenance — the full distance from "a program that runs" to "software a reviewer would believe." Where Case Study 1 diagnosed a trustworthiness failure, this one builds trustworthiness in, deliberately, from the oracles up.

The deliverable is the artifact the Chapter 38 capstone presents as research: a solver that proves itself correct on every change, documents its own API, builds on three compilers automatically, and can be regenerated from a commit hash.

Skills applied: designing unit tests around the four oracles (§37.1); regression and verification against a golden field and an analytical solution (§37.2); authoring a GitHub Actions matrix workflow (§37.3); FORD doc comments and git discipline (§37.4); provenance recording (§37.5). It composes directly on the package of Chapter 36.

Background

The solver's public interface, frozen since Chapter 6 and made real in Chapter 24, is three procedures in heat_solver: laplacian(u, dx, dy), step(field, alpha, dt), and stable_dt(alpha, dx, dy, safety). Your task is to build the test/ directory that guards them, plus the docs and CI around them. You will design before you code — because a test suite assembled at random tests whatever was easy, not what matters.

Phase 1 — Design the Test Plan

A good suite is a plan, mapping each procedure to the oracle that pins it. Lay it out as a table before writing a line, so gaps are visible:

Notice the plan spans all four oracle sources — exact cases, invariants, symmetry, convergence — and layers three scopes: unit (one procedure), regression (the whole thing versus a known-good run), and verification (the whole thing versus a known-true answer). That layering is the design: units localize a break to one procedure, the regression catches unintended change, and the verification catches being wrong.

Phase 2 — Build the Unit Tests

Unit tests come first because they are fastest and most localizing. Using the hand-rolled harness of §37.1, each is a few lines and each expected value is hand-computed. Here is the exactness oracle for laplacian, driving the real procedure:

! test/test_unit.f90  (excerpt) -- exact special cases as unit tests
real(dp) :: u(5,5), lap(5,5)
integer  :: i, j

do j = 1, 5
  do i = 1, 5
    u(i,j) = (real(i-1,dp)*0.5_dp)**2 + (real(j-1,dp)*0.5_dp)**2   ! x^2 + y^2, h = 0.5
  end do
end do
lap = laplacian(u, 0.5_dp, 0.5_dp)
call assert_true('laplacian(x^2+y^2) == 4 (all interior)', all(abs(lap(2:4,2:4) - 4.0_dp) < 1.0e-12_dp))

$ ./test_unit
PASS  laplacian(x^2+y^2) == 4 (all interior)

The value $4$ is the true Laplacian $2 + 2$ of $x^2 + y^2$, and the stencil returns it exactly because it is exact for quadratics — an oracle you compute in your head, not by running anything. The linear-field ($\to 0$) and stable_dt ($r \le 1/4$) unit tests follow the same shape; the complete set is in code/project-checkpoint.f90.

Phase 3 — Build the Regression and Verification Tests

The whole-solver tests come next. The regression test pins the two-step golden field against unintended change, comparing within a tolerance (§37.2's lesson, and Case Study 1's hard-won one):

! test/test_regression.f90 (excerpt) -- golden field, tolerance comparison
call f%init(nx=5, ny=5, dx=1.0_dp, dy=1.0_dp)
f%u(1,:) = 100.0_dp
call step(f, 1.0_dp, 0.2_dp)
call step(f, 1.0_dp, 0.2_dp)
call assert_true('5x5 two-step field matches golden', maxval(abs(f%u - golden)) < 1.0e-9_dp)

The verification test is the stronger one — it checks the solver is right, not merely unchanged — by pitting it against an analytical solution. The linear steady state is exact for the stencil, so it is a fixed point of step:

! test/test_verify.f90 (excerpt) -- the analytical steady state is a fixed point
do j = 1, 5
  do i = 1, 5
    s%u(i,j) = 25.0_dp * real(j-1, dp)       ! exact steady state: linear ramp
  end do
end do
before = s%u(3,3)
call step(s, 1.0_dp, 0.2_dp)
call assert_true('linear steady state is a fixed point', abs(s%u(3,3) - before) < 1.0e-12_dp)

$ ./test_verify
PASS  linear steady state is a fixed point

For the transient verification — the plate warming — the oracle is the separable mode $\sin(\pi x)\sin(\pi y)\,e^{-2\alpha\pi^2 t}$, and the check is a convergence study: refine the grid, measure the error against the exact solution, and confirm it falls by a factor of four each halving, the $O(h^2)$ signature the Project Checkpoint of Chapter 22 taught you to measure. That study is the capstone's validation centerpiece; the suite here wires in the exactly checkable steady-state and maximum-principle pieces and leaves a labeled slot for the full convergence run.

Design note — every test must be able to fail. Before accepting a test, break the code on purpose and confirm the test goes red. Drop the / dx**2 from laplacian and the exactness test must fail (it now returns the unscaled neighbour sum, not $4$); flip a sign in the stencil and the symmetry and steady-state tests must fail. A test you have never seen fail is a test you do not yet trust — it may be asserting nothing, like the worthless allocated(u) check of §37.2.

Phase 4 — Document with FORD, and Author the CI Workflow

With the suite in place, make the code understandable and make the suite run automatically. First, FORD doc comments on the public interface — comments to the compiler, an API page to FORD:

!> Advance the field one explicit (forward-Euler / FTCS) time step, in place.
!  Updates interior cells only; the Dirichlet boundary is held fixed.
subroutine step(field, alpha, dt)
  type(field_t), intent(inout) :: field   !! the field, advanced in place
  real(dp),      intent(in)    :: alpha    !! thermal diffusivity
  real(dp),      intent(in)    :: dt       !! time step (keep r = alpha*dt/h^2 <= 1/4)

Then the CI workflow, so every push builds and runs the suite across three gfortran versions:

# .github/workflows/ci.yml
name: CI
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        gcc: [11, 12, 13]
    steps:
      - uses: actions/checkout@v4
      - run: sudo apt-get update && sudo apt-get install -y gfortran-${{ matrix.gcc }}
      - uses: fortran-lang/setup-fpm@v5
      - run: fpm build --compiler gfortran-${{ matrix.gcc }} --flag "-std=f2018 -Wall"
      - run: fpm test  --compiler gfortran-${{ matrix.gcc }}

The chain that makes this work is the exit code: any test's error stop 1 makes fpm test fail, which fails the CI step, which turns the job red and blocks the merge. The three-compiler matrix is the payoff — it would have caught Case Study 1's bit-for-bit fragility on the second compiler, the day the test was written.

Phase 5 — Stamp Provenance and Tie It Together

Finally, make the solver reproducible: the driver records its own build and environment (§37.5), and the whole thing is committed so a result ties to a commit.

use, intrinsic :: iso_fortran_env, only: compiler_version, compiler_options
print '(a)', 'commit  : ' // git_commit          ! injected at build time
print '(a)', 'compiler: ' // compiler_version()  ! self-reported build
print '(a)', 'options : ' // compiler_options()

Commit the source, the fpm.toml (which pins dependency versions), the CI workflow, and the inputs; tag the release; and the solver is now a release, not a script. Run fpm test and the whole suite reports:

$ fpm test
PASS  unit: laplacian(x^2+y^2) == 4
PASS  unit: laplacian(linear)  == 0
PASS  unit: stable_dt gives r <= 1/4
PASS  regression: 5x5 two-step field matches golden (maxdev = 0.0)
PASS  verification: linear steady state is a fixed point (residual = 0.0)
PASS  verification: maximum principle holds (0 <= u <= 100)
--- 6 / 6 checks passed

That green suite, the FORD API page, the compiler-matrix CI, and the provenance stamp are, together, what turns the Chapter 36 package into a trustworthy artifact. Nothing about the physics changed from Chapter 24; what changed is that the solver now proves its physics on every edit, explains its own interface, checks itself on three compilers, and can be regenerated years from now from a single commit. That is the capstone's software half, built.

Discussion Questions

1. The Phase 1 plan mapped each procedure to an oracle before any code was written. What kinds of gap does designing the plan first reveal that assembling tests ad hoc would miss? Give an example of an untested behaviour the table would surface.
2. Phase 3's "every test must be able to fail" note asks you to break the code deliberately. Why is a test you have never watched fail a liability, and how does this connect to the worthless allocated(u) test of §37.2?
3. The CI matrix runs three compilers on every push. Beyond catching non-portable code, how does the matrix change the meaning of a passing bit-for-bit test — and why does it push you toward tolerance comparisons?

Your Turn: Extensions

• Option A. Add the convergence-study verification test in full: initialize the field to the separable mode on grids of $h, h/2, h/4$, step to a fixed time, measure the max error against the exact solution, and assert the successive error ratios lie in $[3.5, 4.5]$ (near the $O(h^2)$ value $4$, with slack for round-off). State why the window, not the exact value.
• Option B. Add a test_symmetry unit test: assert the hot-top plate's field is unchanged under column reversal after one step (maxval(abs(u - u(:,n:1:-1))) below tolerance). Then break the laplacian's $y$-term sign and confirm the symmetry test catches it.
• Option C. Write the fpm.toml that wires pFUnit in as a test dependency and the .pf versions of two of your tests, so fpm test runs the framework suite. Compare the developer experience with the hand-rolled harness — what does the framework buy, and what does the hand-rolled version teach that the framework hides?

Key Takeaways

• Design the suite as a plan — a table mapping each procedure to an oracle — so it tests what matters, spans all four oracle sources, and layers unit, regression, and verification scopes.
• Layer the scopes: units localize a break to one procedure; the golden regression catches unintended change; the analytical verification catches being wrong, which a regression alone can preserve.
• Every test must be able to fail — break the code on purpose and watch it go red, or you cannot trust the green.
• Docs, CI, and provenance finish the job: FORD comments explain the interface, a compiler-matrix workflow proves the code on every push via the exit-code chain, and a self-stamped build ties each result to a commit — the package of Chapter 36, now a trustworthy release.