Exercises: Testing, Documentation, and Software Engineering

Testing is a skill you build by writing tests and breaking code to watch them catch it, not by memorizing definitions. These exercises make you classify tests by kind, choose oracles for numerical code, decide between bit-for-bit and tolerance comparisons, write assertions and a CI workflow, add FORD docs, and reason about what a reproducible result must record. The heart of the set is extending the heat solver's own test suite. Do the compilable ones at a real terminal — a test you did not run is a test you do not yet trust.

Difficulty: ⭐ warm-up · ⭐⭐ standard · ⭐⭐⭐ deeper. Solutions: worked solutions to the daggered (†) and odd-numbered problems are in appendices/answers-to-selected.md; the compilable ones (37.7, 37.12, 37.22) are worked in full as code/exercise-solutions.f90. Every program compiles with gfortran -std=f2018 -Wall. Try each problem before you look.

Part A — Testing Fundamentals ⭐

37.1 † Classify each check as a unit, regression, or verification test: (a) confirms laplacian(x^2+y^2) == 4 on a small grid; (b) compares the full solver's output field to the field stored from the last release; (c) compares the solver to the analytical steady state; (d) confirms stable_dt returns a step with $r \le 1/4$; (e) confirms a modernized routine reproduces the FORTRAN 77 original's output.

37.2 In two or three sentences, define a test oracle and explain why finding one is the hard part of testing numerical code — contrast it with testing a routine that sorts a list.

37.3 † Why must a floating-point assertion compare abs(got - want) <= tol rather than got == want? Give a concrete two-line example (real values) where the == test fails even though the computation is correct.

37.4 Name the four oracle sources for numerical tests from §37.1, and give one concrete example of each for the heat solver.

Part B — Reading and Writing Assertions ⭐⭐

37.5 † (Type, compile, and run — predict first.) Read code/example-01-hand-rolled-assert.f90. Write down its exact output. Then predict what changes if you introduce a bug into laplacian by dropping the / dx**2 scaling (so the neighbour sum is returned unscaled) — which specific check(s) now fail, and which still pass, and why?

37.6 (Find the bug — in the test.) A teammate adds this "test" and reports it passes:

call step(field, alpha, dt)
call assert_true('solver ran', allocated(field%u))

Explain why it is worthless as a test, state the one-sentence rule it violates ("a test that cannot ___ tests nothing"), and rewrite it as a real check of the values.

37.7 † (Design it — compilable.) Using the §37.1 assert harness, write a unit test that the hot-top plate is left–right symmetric after one step: the field must be unchanged when its columns are reversed. State the expected result and why symmetry is a good oracle even though you cannot predict the individual cell values.

37.8 (Write the pFUnit.) Write a pFUnit @test subroutine (a .pf snippet, not compiled by plain gfortran) that checks the maximum principle after two steps of the hot-top plate: every value lies in $[0, 100]$. Which pFUnit assertion(s) do you use?

Part C — Regression and Bit-for-Bit ⭐⭐

37.9 † Give three build-configuration changes that can make a bit-for-bit regression test fail on correct, unchanged source, and in one clause each say why the change reorders floating-point operations. (Recall §37.2 and Chapter 30.)

37.10 (Back of the envelope.) You are tempted to store the golden output of a $1000 \times 1000$ run as a text file at full precision (~24 characters per number). Estimate the file size. Why is a coarse-grid golden, or a checksum, usually a better regression artifact than a giant full-resolution dump?

37.11 † When is bit-for-bit the right comparison? Describe a pinned configuration (name the things you pin) and one concrete scenario — for example, verifying that a refactor changed nothing — in which bit-for-bit is exactly the check you want, and a tolerance would be too weak.

Part D — Verification Against Analytical Solutions ⭐⭐⭐

37.12 † (Design it — compilable.) Write a verification test: initialize the field to the exact linear steady state (the ramp $0, 25, 50, 75, 100$ across columns), take one step, and assert the interior did not change by more than $10^{-12}$. Explain, from the stencil's exactness on linear fields, why the residual is $0$.

37.13 The transient heat equation on the unit square with zero edges has the separable solution $u(x,y,t) = \sin(\pi x)\sin(\pi y)\,e^{-2\alpha\pi^2 t}$. Given the discrete Laplacian is $O(h^2)$, what error ratio should you see against this solution when you halve the grid spacing, and where does that number come from? (Cite Chapter 22.)

37.14 † (Design it.) Sketch — in steps and asserts, not full code — a convergence-study test that refines the grid three times against the analytical solution and confirms the error ratio approaches $4$. What tolerance would you allow on the measured ratio, and why is it wrong to assert the ratio equals exactly $4$?

Part E — Continuous Integration and Automation ⭐⭐

37.15 † Read the GitHub Actions workflow in §37.3. Answer: (a) what events trigger it; (b) what the matrix: gcc: [11, 12, 13] line causes; and (c) trace, step by step, how a failing assert_close becomes a red, merge-blocking job.

37.16 Extend the workflow to add a debug build (-O0 -g -fcheck=all) alongside the release one. Why is running the test suite under -fcheck=all in CI worth the slowdown? (Connect to Chapter 13.)

37.17 † (Back of the envelope.) Your suite runs in $8$ s locally; the matrix has $3$ compilers; each fresh CI runner spends about $1$ minute on setup (checkout, install). Estimate the wall-clock time (jobs run in parallel) and the total machine-minutes per push. Use the result to argue why unit tests belong on tiny grids.

Part F — Documentation, Git, and Reproducibility ⭐⭐⭐

37.18 † Add FORD !> / !! doc comments to the step subroutine, documenting the routine and each of its three arguments. Does the file still compile with gfortran -std=f2018 -Wall? Explain why in one sentence.

37.19 A paper's Figure 3 was produced by your solver. List everything from §37.5 you must record so a reader can regenerate it, and name the single item that pins the code itself.

37.20 † (Find the bug — reproducibility.) A stochastic run (it calls random_number) gives different results every time and "won't reproduce." The code never calls random_seed. Explain the cause, give the two-line fix, and state what you must record with the output — and the honest caveat about reproducing the same seed on a different compiler.

37.21 Why should large binary simulation outputs not be committed to git, and what do you commit instead so the result stays reproducible?

Part G — Interleaved ⭐⭐⭐

37.22 † (Interleaved — Chapter 13 — compilable.) Extend the §37.1 harness with a whole-array assertion assert_all_close(name, got(:,:), want(:,:), tol) that passes when the maximum element-wise deviation is within tol. Use it to test the golden $5\times 5$ two-step field in a single call.

37.23 (Interleaved — Chapter 36.) Explain why a pure physics function such as laplacian is trivially unit-testable, connecting your answer to the five architectural roles and to what a pure procedure may and may not do.

37.24 † (Interleaved — Chapter 20.) The golden values $28$, $32$, $4$ are bit-exact doubles, yet $0.2$ is not exactly representable. Explain how the products $0.2 \times 100$ and $0.2 \times 60$ nonetheless round to exactly $20$ and $12$ — and why, despite this happy accident, a robust regression test still compares with a tolerance rather than ==.

37.25 (Port it.) A pytest test reads numpy.testing.assert_allclose(got, want, atol=1e-9). Translate it to (a) the hand-rolled Fortran harness and (b) a pFUnit @assertEqual, and say what plays the role of atol in each.

37.26 † (Interleaved — Chapter 18.) Chapter 18 used a regression test as the safety net for modernizing legacy code. Explain how the golden-field test of §37.2 is the same idea, and why bit-for-bit comparison is tempting there but often wrong — even when "the answer should be identical."

Solutions to the daggered and odd-numbered problems are in appendices/answers-to-selected.md; the compilable ones (37.7, 37.12, 37.22) are worked in full as code/exercise-solutions.f90. For every "design it" problem, state the oracle you are testing against and the tolerance you chose — they are part of the answer.