Case Study 2: Building a Regression Harness for a Migration at Scale

"Legacy code is code we have gotten value from that we are afraid to change."

Executive Summary

Modernizing a five-subprogram kernel by eye is one thing; modernizing a 30-module, 20,000-line thermal code across many weeks is another. At that scale you cannot eyeball outputs — you need a tool that, after every one of the hundreds of edits, answers one question mechanically: do the numbers still match the reference? This case study builds that tool. We design the acceptance criterion (bit-for-bit where the arithmetic is preserved, tolerance where it is deliberately changed), write a numerical-comparison harness in modern Fortran, lay out a phased migration plan ordered by module dependencies, and handle the one wrinkle that always appears — a well-intentioned precision change that breaks bit-for-bit and forces a considered choice of tolerance. Where Case Study 1 ran the recipe, this one builds the infrastructure that makes running it safe at scale.

Skills applied: regression testing and numerical equivalence (§18.3); designing a tolerance (bit-for-bit vs "close enough", §18.3); incremental modernization as a managed process (§18.2); building a comparison tool with intent, assumed-shape, and array intrinsics (§18.1); connecting to CI and testing practice (Chapter 37).

Background

You are leading the migration of THERMASIM, a validated FORTRAN 77 thermal-analysis code: ~30 modules' worth of subprograms sharing a dozen COMMON blocks, GOTO control flow throughout, a few EQUIVALENCEs, and — the detail that will bite — a mix of single- and double-precision arithmetic. It produces a report of a few thousand numbers per run. Management wants it modern (readable, testable, ready for the OpenMP work of Part VIII); the regulator wants proof that the science did not change. Both demands are satisfied by the same artifact: a regression harness that turns "did the science change?" into a build-time pass/fail.

Phase 1 — Design the Acceptance Criterion

Before writing any comparison code, decide what "unchanged" means, because it differs by step.

• The mechanical and structural steps — implicit none, free-form, COMMON → modules, intent, assumed-shape, GOTO → do — preserve the arithmetic exactly. For every edit in this class, the criterion is bit-for-bit: max absolute difference exactly zero. Anything else is a bug.
• The deliberate numeric changes — promoting a single-precision variable to real(dp), reordering a reduction, replacing a hand-rolled sum with an intrinsic — change the low-order bits legitimately. For these the criterion is tolerance-based: agreement to within an absolute or relative bound you justify from the code's own validated accuracy.

The harness therefore needs to support both: a strict mode (tolerance zero) and a tolerant mode. The strict mode is the default, because most steps are arithmetic-preserving and you want the tightest net that will hold them.

Phase 2 — Build the Harness

The tool reads the reference numbers (captured once from the trusted legacy build) and the candidate numbers (from the current modern build), and reports the maximum absolute and relative differences and a verdict. Here is the core, exercised on sample data so its output is hand-checkable:

! regress.f90 -- numerical regression comparison for a migration.
! Compile:  gfortran -std=f2018 -Wall regress.f90 -o regress && ./regress
module regress
  implicit none
  integer, parameter :: dp = selected_real_kind(15, 307)
contains
  !> Compare candidate against reference; ok is .true. if within tolerance.
  subroutine compare(ref, cand, atol, maxabs, maxrel, ok)
    real(dp), intent(in)  :: ref(:), cand(:)      ! assumed-shape: sizes travel along
    real(dp), intent(in)  :: atol                 ! absolute tolerance (0 => bit-for-bit)
    real(dp), intent(out) :: maxabs, maxrel
    logical,  intent(out) :: ok
    real(dp) :: adiff(size(ref))
    if (size(cand) /= size(ref)) error stop 'regress: length mismatch'
    adiff  = abs(cand - ref)
    maxabs = maxval(adiff)
    maxrel = maxval(adiff / max(abs(ref), tiny(1.0_dp)))
    ok     = (maxabs <= atol)
  end subroutine compare
end module regress

program run_regress
  use regress
  implicit none
  real(dp) :: ref(3)  = [1.0_dp, 10.0_dp, 100.0_dp]
  real(dp) :: cand(3) = [1.0_dp, 10.0_dp, 100.001_dp]
  real(dp) :: maxabs, maxrel
  logical  :: ok
  call compare(ref, cand, atol = 1.0e-2_dp, maxabs = maxabs, maxrel = maxrel, ok = ok)
  print '(a, f0.6)', 'max abs diff = ', maxabs
  print '(a, f0.6)', 'max rel diff = ', maxrel
  print '(a, l1)',   'within tol   = ', ok
end program run_regress

Verify the output by hand before trusting the tool. The candidate differs from the reference only in the third value, by $100.001 - 100.0 = 0.001$. So the maximum absolute difference is $0.001$, and the maximum relative difference is $0.001 / 100.0 = 10^{-5}$. With an absolute tolerance of $10^{-2}$, $0.001 \le 0.01$, so the verdict is pass:

$ gfortran -std=f2018 -Wall regress.f90 -o regress && ./regress
max abs diff = 0.001000
max rel diff = 0.000010
within tol   = T

Notice what the harness deliberately does not do: it never compares text. It parses the numbers into real(dp) arrays and compares those, so a legacy E exponent versus a modern ES one, or five-column integer padding versus none, cannot produce a false failure. This is the §18.3 warning built into a tool. Set atol = 0.0_dp and the same harness enforces strict bit-for-bit agreement for the arithmetic-preserving steps.

Design note. compare takes its arrays as assumed-shape intent(in) and asks their size with size(); the reference and candidate lengths must match or it is a structural error worth an error stop, not a silent truncation. maxval(abs(cand - ref)) does the whole comparison as two whole-array operations — the modern style the chapter has been teaching, doing real work in the tool that guards the migration.

Phase 3 — Order the Migration by Dependencies

With the harness in hand, sequence the work. Modules must be modernized in dependency order — you cannot turn COMMON /MATERIAL/ into a material module until the modules that read it are ready to use it — so build the dependency graph first and work from the leaves inward. A workable plan:

1. Sweep the whole code with the mechanical steps (free-form, implicit none, GOTO → do), one subprogram per commit, running the harness in strict mode after each. This is safe, high-value, and makes the code readable enough to see its structure.
2. Convert COMMON blocks to modules, leaf blocks first, each followed by a strict-mode harness run.
3. Introduce intent and assumed-shape on the now-modularized routines, again strict mode.
4. Excise EQUIVALENCE case by case, deciding each one's intent (overlay, reinterpret, or group).
5. Add error handling at the input boundaries.

Every step is a commit; every commit is green under the harness; a red run points at exactly one edit. This is incremental modernization (§18.2) scaled up with tooling and version control.

Phase 4 — The Precision Wrinkle

Partway through, you find a module that accumulates a heat balance in single precision — a genuine liability, since the sum of thousands of terms loses accuracy. Promoting it to real(dp) is the right engineering call, but it is not arithmetic-preserving: the moment you do it, the strict-mode harness goes red, because the low-order bits of every downstream number shift.

This is the moment to switch that comparison from bit-for-bit to tolerance-based — deliberately, and with justification. The question is not "do the bits match?" (they should not) but "does the answer still agree to within the accuracy the code was validated at?" If THERMASIM was validated to, say, four significant figures against experiment, then a relative tolerance safely tighter than that — say $10^{-6}$, far below the validation accuracy but well above double-precision noise — is defensible: it certifies the promotion improved precision without moving the answer within the band anyone ever trusted. Record the decision: this module's comparison is tolerance-based at rel $10^{-6}$ because step X changed the precision on purpose.

The judgment that matters: a red regression run is not automatically a failure — it is a question. For an arithmetic-preserving step it means "you broke something, find it." For a deliberate numeric change it means "confirm the change is the improvement you intended, then re-baseline." Knowing which situation you are in — and never blurring them — is the core skill of testing a migration.

Phase 5 — Wire It Into CI

Finally, make the harness automatic. In continuous integration (Chapter 37), every push builds the code, runs the reference case, and pipes the output through regress; a nonzero exit fails the build. Now no change — yours or a collaborator's — can alter the validated numbers without someone choosing to accept it. The migration that began as a risky manual slog ends as a codebase that cannot silently regress, which is a stronger guarantee than the original legacy code ever had.

Discussion Questions

1. The harness's default is atol = 0 (bit-for-bit). Why make the strict mode the default rather than a convenient loose tolerance? What failure would a loose default hide?
2. In Phase 4 you chose a relative tolerance of $10^{-6}$ against a validation accuracy of four significant figures. Argue for a looser tolerance and for a tighter one. What is the risk at each extreme?
3. The plan modernizes COMMON blocks "leaf blocks first." What goes wrong if you convert a heavily-shared block early, before the routines that use it are ready?

Your Turn: Extensions

• Option A. Extend compare to read the reference and candidate numbers from two files (list-directed read), so the harness works on real program output rather than hard-coded arrays. Keep a hand-checked test case.
• Option B. Add per-element reporting: when the verdict is fail, print the index and values of the worst offender, so a red run points at which number moved, not just that one did.
• Option C. Design (in prose) the CI job: what triggers it, what it builds, what reference it runs, and what it does on failure. Tie each part back to reproducibility (record compiler and flags — the bit-for-bit-is-a-build-property point from §18.3).

Key Takeaways

• At scale, the regression harness is the deliverable that makes modernization safe — it turns "did the science change?" into a mechanical, build-time pass/fail you run after every edit.
• Build the criterion into the tool: strict bit-for-bit by default for arithmetic-preserving steps, an explicit and justified tolerance for deliberate numeric changes. Never blur the two.
• A red regression run is a question, not a verdict: on a pure refactoring it means "you broke it"; on a deliberate improvement it means "confirm and re-baseline." Telling them apart is the whole skill.
• Comparing numbers within tolerance — never text — and wiring the check into CI leaves you with a code that cannot silently regress, a stronger position than the legacy original was ever in.