Case Study 2: The Safety Net

"The courage to change a working program comes entirely from the test that will catch you if you break it."

Executive Summary

Where the first case study read a legacy routine, this one asks you to build something: the safety net that makes modernizing the PLATE kernel safe. The cardinal rule of Part IV is that you modernize a validated code without changing the science, and the only way to prove you have not changed it is to pin its behavior down before you touch it, with a regression harness — a test that runs the code on known inputs and checks the outputs against a trusted reference. You will design the harness, decide what "still correct" means when floating-point rounding is in play, build it in modern Fortran, and establish the baseline that Chapter 18 will check against after every migration step. This is the engineering that turns a terrifying rewrite into a routine, reversible refactor.

Skills applied: reading the PLATE kernel to find its checkable outputs (§17.6); exploiting the known analytic answer (the 4×4 hot-edge symmetry solution) as a test oracle; distinguishing bit-for-bit from tolerance-based equivalence; designing a comparison in modern Fortran that a future modernization must keep passing.

Background

You are about to modernize PLATE — the FORTRAN 77 Jacobi relaxation kernel from §17.6 — through the eight-step recipe of Chapter 18. Every step (free-form, COMMON to module, structured control, and so on) is supposed to leave the computed temperature field unchanged. But "supposed to" is not good enough for a validated code; a transposed index or an off-by-one in a rewritten loop would corrupt the science silently, exactly the failure mode Part IV exists to prevent. You need a test that fails loudly the instant the answer changes.

PLATE hands you an unusual gift: on its 4×4 test grid — top edge hot at 100°, three edges cold at 0° — its exact steady state is known by symmetry. The two upper interior cells equal $a$ and the two lower equal $b$, with $3a - b = 100$ and $a = 3b$, so $b = 12.5$ and $a = 37.5$ — every value a dyadic fraction, exact in double precision. Most legacy codes have no such closed-form answer and you must snapshot the current output as the reference ("golden output" testing). Here you have something stronger: an analytic oracle, a mathematically correct answer derived independently of the kernel. Use it.

Phase 1 — Decide What "Still Correct" Means

Before writing a line of the harness, resolve the question that trips up every migration: when you compare the modernized kernel's output to the reference, must they match exactly, or only closely?

Standard What it checks When to use it
Bit-for-bit Every bit of every number identical Reordering that provably preserves arithmetic (renaming, free-form, COMMON→module)
Tolerance ("close enough") Values agree within some $\varepsilon$ Any change that reorders floating-point operations (a new loop nest, sum vs a manual loop)

The distinction is not pedantic. Modernization step 3 (COMMON to a module) moves data but performs the same additions in the same order, so it should be bit-for-bit identical. But if a later step replaces the hand-written neighbour sum with an intrinsic, or changes the sweep order, the roundoff can differ in the last bit even though the mathematics is unchanged — and a bit-for-bit test would raise a false alarm. The safe default for a numerical code is a tolerance comfortably larger than roundoff yet far smaller than any physically meaningful change. For our problem, whose exact values are 37.5 and 12.5, a tolerance of $10^{-3}$ degrees is far larger than any roundoff yet thousands of times smaller than a real error. (On this particular grid the values are dyadic and computed with no rounding at all, so a bit-for-bit check also works — but a tolerance is the habit to build for the general case.)

Phase 2 — Build the Oracle

The heart of the harness is the reference field. Because we have the analytic answer, the oracle is a few lines — and, crucially, it shares no code with the kernel, so a bug in the kernel cannot hide by also being in the oracle.

module heat_check
  use, intrinsic :: iso_fortran_env, only: dp => real64
  implicit none
  private
  public :: dp, plate_reference, max_abs_diff, report_check

contains

  ! The analytic oracle: the exact steady state of the 4x4 hot-edge problem,
  ! derived from the symmetry relations 3a - b = 100 and a = 3b (NOT by
  ! relaxation), so a bug in the kernel cannot also hide in the oracle.
  function plate_reference() result(t)
    real(dp) :: t(4, 4), a, b
    b = 100.0_dp / 8.0_dp          ! 8b = 100  ->  b = 12.5
    a = 3.0_dp * b                 ! a = 3b     ->  a = 37.5
    t = 0.0_dp
    t(1, :) = 100.0_dp             ! hot top edge; three cold edges stay 0
    t(2, 2) = a;  t(2, 3) = a      ! upper interior
    t(3, 2) = b;  t(3, 3) = b      ! lower interior
  end function plate_reference

  ! The comparison: the largest absolute difference over the whole field.
  pure function max_abs_diff(a, b) result(d)
    real(dp), intent(in) :: a(:,:), b(:,:)
    real(dp) :: d
    d = maxval(abs(a - b))
  end function max_abs_diff

  ! The verdict: PASS if within tolerance, FAIL otherwise.
  subroutine report_check(name, d, tol)
    character(*), intent(in) :: name
    real(dp),     intent(in) :: d, tol
    if (d <= tol) then
      print '(a, a, f7.3, a)', name, ': PASS (max diff ', d, ')'
    else
      print '(a, a, f7.3, a)', name, ': FAIL (max diff ', d, ')'
    end if
  end subroutine report_check

end module heat_check

Phase 3 — Wire Up the Harness and Prove It Catches a Regression

A test you have never seen fail is not a test; it is a hope. So the harness runs two checks: a faithful candidate (the field the kernel converges to) that must PASS, and a deliberately broken candidate (one interior cell wrong by 5 degrees) that must FAIL. Seeing the FAIL is how you know the net has holes small enough to catch the fish.

program regression_harness
  use heat_check
  implicit none
  real(dp)            :: reference(4, 4), candidate(4, 4)
  real(dp), parameter :: tol = 1.0e-3_dp

  reference = plate_reference()

  ! Check 1 — a faithful candidate: exactly the field the kernel converges to.
  candidate = reference
  call report_check('faithful ', max_abs_diff(candidate, reference), tol)

  ! Check 2 — a regressed candidate: one interior cell corrupted by +5 degrees.
  candidate = reference
  candidate(2, 2) = candidate(2, 2) + 5.0_dp
  call report_check('regressed', max_abs_diff(candidate, reference), tol)
end program regression_harness

$ gfortran -std=f2018 -Wall heat_check.f90 regression_harness.f90 -o harness && ./harness
faithful : PASS (max diff   0.000)
regressed: FAIL (max diff   5.000)

Trace both by hand. The faithful candidate is the reference, so abs(a - b) is zero everywhere and the maximum difference is 0.000, comfortably under the 0.001 tolerance — PASS. The regressed candidate differs from the reference in exactly one cell, by exactly 5.0 degrees, so the maximum absolute difference is 5.000, far above tolerance — FAIL. The harness distinguishes a faithful field from a corrupted one, on a criterion you set deliberately. That is the entire job.

Phase 4 — Establish the Baseline

Now connect the harness to the real workflow. In Chapter 18 you will replace candidate with the actual output of the modernized kernel — read from the file it writes, or computed by calling the modernized solve directly. Today, with only the legacy kernel in hand, you do three things to lock in the baseline:

  1. Save the reference output. Run plate-legacy.f and capture its printed field (converged in 25 iterations, then the interior 37.5000 / 12.5000) into legacy/expected.txt. Because the answer is the analytic steady state you derived by hand, you can also confirm the captured file against the oracle by eye — a rare double check.
  2. Record the tolerance and the standard. Write down, beside the harness, that steps preserving arithmetic order are expected bit-for-bit and steps reordering it are checked to $10^{-3}$. A tolerance nobody wrote down is a tolerance somebody will argue about later.
  3. Make the check runnable in one command. The harness should be a single build-and-run so that after every modernization step you can rerun it in seconds. A safety net you have to assemble by hand is a safety net you will skip on the day you most need it.

Phase 5 — Why This Changes the Economics of Modernization

With the harness in place, the risk profile of the whole Chapter 18 migration inverts. Without it, each edit to a validated code is an act of faith, and the rational response is fear — which is precisely why so much legacy code is never modernized and slowly rots. With it, each edit is cheap and reversible: make one change, rerun the harness, and either it passes (keep going) or it fails (you know, in seconds, that the last edit — and only the last edit — broke something). The eight-step recipe of Chapter 18 becomes a sequence of small, individually verified moves rather than one terrifying leap. This is the concrete machinery behind the ethic legacy code is not a burden; it is an inheritance — the net is what lets you improve the engineering while proving, at every step, that you preserved the science.

Discussion Questions

  1. The oracle here is an analytic answer that shares no code with the kernel. Why is that independence essential, and what weaker form of testing must you fall back on for a legacy code with no closed-form solution?
  2. Phase 1 argued for a tolerance rather than bit-for-bit comparison for numerical code. Give one concrete modernization step that should be bit-for-bit and one that should be tolerance-based, and justify each.
  3. The harness deliberately includes a check that fails. Why is a test suite in which nothing ever fails a warning sign rather than a reassurance?

Your Turn: Extensions

  • Option A. Extend the harness to read the candidate field from a text file (the kernel's actual output) rather than constructing it in code, using the file I/O of Chapter 7. This is what makes the harness test the real modernized program in Chapter 18.
  • Option B. Add a second test problem to the oracle: an all-edges-equal boundary (every edge at 50 degrees), whose exact steady state is a uniform 50 everywhere. Predict the reference field, add it to the harness, and explain why a second independent case strengthens the net.
  • Option C. Generalize max_abs_diff into a relative error check, maxval(abs(a-b)) / maxval(abs(b)), and discuss when a relative tolerance is safer than an absolute one (hint: consider a field whose values are all near a million, or all near a millionth).

Key Takeaways

  • You do not modernize a validated code and then test it; you build the regression harness first, so that every migration step is checked the moment you make it.
  • Decide the comparison standard deliberately: bit-for-bit for order-preserving changes, a stated tolerance for anything that reorders floating-point arithmetic. Roundoff is not a regression.
  • An independent oracle — here the analytic 4×4 symmetry solution (37.5 / 12.5) — is the strongest reference, because a bug in the code under test cannot also hide in it. When no analytic answer exists, snapshot the current output and test against that.
  • A harness must be seen to fail on a real regression and be runnable in one command, or it will not be trusted or used. The net is what converts modernization from an act of faith into a safe, reversible engineering process — the foundation for Chapter 18.