Case Study 2: A Configuration and Restart System You Can Trust

"A long simulation without a restart file is a bet that nothing will go wrong for a week."

Executive Summary

Where the first case study fixed a code, this one asks you to design one component properly from the start: the I/O layer that a real, long-running simulation depends on. A production run must be configurable without recompiling, restartable after a crash without losing days of computation, self-documenting so you know months later exactly what produced a result, and it must fail loudly on bad input rather than compute nonsense. You will build each of these from this chapter's tools — a multi-group namelist for configuration, input validation guarded by error stop, a stream-binary checkpoint that resumes a run bit-for-bit, and a provenance echo that stamps every output with the parameters that made it. This is the difference between a script and an instrument.

Skills applied: multi-group namelist configuration (§7.4); guarded, validated input with iostat/iomsg and error stop (§7.6); exact stream-binary checkpoints (§7.5); formatted echo for provenance (§7.1); the intent and assumed-shape discipline of Chapter 6.

Background

You are writing the I/O layer for the heat solver the book builds toward (Chapter 38). It will run for hours to days on a shared cluster, where jobs are killed by wall-clock limits, nodes fail, and the person reading the results in six months may be you with no memory of the run. Four requirements follow directly, and each maps to a tool from this chapter.

Phase 1 — Group the Configuration by Concern

A real code has many parameters, and cramming them into one namelist group is a maintenance smell. Split them by concern — the grid, the physics, the output — into separate groups in one file. This mirrors how the code is organized (and, later, how it splits into modules in Chapter 8), and it lets a user edit the output settings without scrolling past the physics.

program config_system
  use, intrinsic :: iso_fortran_env, only: dp => real64
  implicit none

  integer  :: nx, ny, write_every
  real(dp) :: alpha, dt
  character(len=64) :: out_prefix
  namelist /grid/    nx, ny
  namelist /physics/ alpha, dt
  namelist /output/  out_prefix, write_every

  integer :: u, ios
  character(len=200) :: msg

  ! Defaults for every parameter, so a short file need only override a few.
  nx = 100;  ny = 100
  alpha = 1.0e-4_dp;  dt = 0.5_dp
  out_prefix = 'field';  write_every = 50

  ! Write a self-contained multi-group config so the demo runs on its own.
  open(newunit=u, file='run.nml', status='replace', action='write')
  write(u, '(a)') '&grid'
  write(u, '(a)') '  nx = 8,'
  write(u, '(a)') '  ny = 8,'
  write(u, '(a)') '/'
  write(u, '(a)') '&physics'
  write(u, '(a)') '  alpha = 1.0e-4,'
  write(u, '(a)') '  dt = 0.1,'
  write(u, '(a)') '/'
  write(u, '(a)') '&output'
  write(u, '(a)') "  out_prefix = 'run',"
  write(u, '(a)') '  write_every = 10,'
  write(u, '(a)') '/'
  close(u)

  ! Read all three groups, guarded against a missing or malformed file.
  open(newunit=u, file='run.nml', status='old', action='read', &
       iostat=ios, iomsg=msg)
  if (ios /= 0) then
     print '(a)', 'cannot open run.nml: ' // trim(msg)
     error stop
  end if
  read(u, nml=grid)
  read(u, nml=physics)
  read(u, nml=output)
  close(u)

  call validate(nx, ny, alpha, dt)

  print '(a)',            'config loaded and validated:'
  print '(a, i0, a, i0)', '  grid        : ', nx, ' x ', ny
  print '(a, es9.2)',     '  alpha       : ', alpha
  print '(a, f6.3)',      '  dt          : ', dt
  print '(a, i0)',        '  write_every : ', write_every
  print '(a, a)',         '  out_prefix  : ', trim(out_prefix)

contains

  subroutine validate(n1, n2, a, d)
    integer,  intent(in) :: n1, n2
    real(dp), intent(in) :: a, d
    if (n1 < 2 .or. n2 < 2) then
       print '(a)', 'invalid grid: nx and ny must be >= 2'
       error stop
    end if
    if (a <= 0.0_dp) then
       print '(a)', 'invalid alpha: diffusivity must be > 0'
       error stop
    end if
    if (d <= 0.0_dp) then
       print '(a)', 'invalid dt: time step must be > 0'
       error stop
    end if
  end subroutine validate

end program config_system

$ gfortran -std=f2018 -Wall -O2 config_system.f90 -o config && ./config
config loaded and validated:
  grid        : 8 x 8
  alpha       :  1.00E-04
  dt          :  0.100
  write_every : 10
  out_prefix  : run

Three sequential reads pull the three groups from one file — namelist reads scan forward for their group name, so the groups may appear in any order. Every parameter had a default, so the file only needs to state what differs from it, and the echo confirms the file's 8 x 8 grid overrode the default 100 x 100.

Phase 2 — Validate Before You Compute

Reading a value is not the same as trusting it. A grid of nx = 0, a negative diffusivity, a zero time step — each is a typo a user will eventually make, and each produces a run that either crashes obscurely much later or, worse, produces plausible-looking garbage. The validate routine above turns bad input into an immediate, specific complaint:

$ ./config          # if run.nml had said  nx = 0
invalid grid: nx and ny must be >= 2
ERROR STOP

error stop halts the program now, at the point of the bad input, with a nonzero exit code the batch system will notice — not three hours into a run when the corruption finally surfaces. This is the Chapter 6 habit of intent(in) arguments turned outward: the same instinct that lets the compiler check a procedure lets you check the world. The full defensive-programming treatment — assertions, -fcheck=all, floating-point traps — is Chapter 13; validating configuration at startup is where it begins.

Phase 3 — Checkpoint and Restart, Exactly

Now the feature that saves a week. A checkpoint dumps enough state — here, the step counter and the temperature field — to resume the run exactly. It must be binary, because a text checkpoint is either lossy (a rounded field restarts a different simulation) or huge; stream-binary is exact and compact, and one write per object is all it takes.

program checkpoint_demo
  use, intrinsic :: iso_fortran_env, only: dp => real64
  implicit none
  integer  :: step
  real(dp) :: field(2,2)
  integer  :: step_in
  real(dp) :: field_in(2,2)
  integer  :: u, i

  step  = 42
  field = reshape([1.0_dp, 2.0_dp, 3.0_dp, 4.0_dp], [2, 2])   ! column-major

  ! WRITE checkpoint: the step counter, then the field, as stream binary.
  open(newunit=u, file='chk.bin', access='stream', form='unformatted', &
       status='replace', action='write')
  write(u) step
  write(u) field
  close(u)

  ! RESTART: read them straight back to resume exactly where we stopped.
  open(newunit=u, file='chk.bin', access='stream', form='unformatted', &
       status='old', action='read')
  read(u) step_in
  read(u) field_in
  close(u)

  print '(a, i0)', 'resumed at step ', step_in
  do i = 1, 2
     print '(*(f5.1))', field_in(i, :)
  end do
end program checkpoint_demo

$ gfortran -std=f2018 -Wall -O2 checkpoint_demo.f90 -o chk && ./chk
resumed at step 42
  1.0  3.0
  2.0  4.0

The step counter and the field come back exactly — field_in(1,1)=1.0, field_in(2,1)=2.0, and so on, in column-major order — because stream binary copies the bits verbatim. A driver reads the checkpoint at startup if one exists (inquire(file='chk.bin', exist=resuming)) and otherwise starts fresh, so the same program both starts and resumes. The order of writes and reads must match exactly — step then field, both sides — which is the one discipline stream I/O demands in exchange for its speed and exactness.

Phase 4 — Stamp Every Output With Its Provenance

Six months from now, a figure on your desk raises a question: what parameters produced this? If the answer lives only in a run.nml you have since edited, the result is unreproducible. The fix is one line at output time: echo the namelist into the output, so the file documents itself.

! At the top of the results/log file, before any data:
write(log_unit, nml=grid)
write(log_unit, nml=physics)
write(log_unit, nml=output)

write(unit, nml=...) dumps each group in the same &group ... / form it reads, so the header of your output is a valid config file — you could feed it straight back in to reproduce the run. (Recall from §7.4 that namelist output formatting is processor-defined, so this is for provenance and restart, not for pretty tables; when you want a human-readable header, format it yourself with §7.1.) This closes the reproducibility loop that Chapter 37 makes a first-class engineering concern.

Phase 5 — The Assembled Design

Put the pieces together and you have an I/O layer worthy of a real code: a driver that reads a multi-group namelist, validates it and error stops on nonsense, checks for a checkpoint and resumes exactly if one exists, writes stream-binary fields every write_every steps, and stamps each output with its own configuration. Sanity check the whole path: run once from scratch to step 42 and let it checkpoint; kill it; run again; confirm it reports resumed at step 42 and continues to the same final field a crash-free run would have produced. If the restart is bit-for-bit identical to the uninterrupted run, the system works — and that bit-for-bit test is exactly the regression test of Chapter 37. Every procedure here — read_config, write_field, write_checkpoint, read_checkpoint — has a stable, intent-checked interface, so when Chapter 8 lifts them into a heat_io module, nothing above them changes.

Discussion Questions

1. Why split configuration into /grid/, /physics/, and /output/ groups instead of one big /config/? Consider adding a new physics parameter a year from now — which design forces fewer edits to old files?
2. The checkpoint writes step and field as two separate writes. What breaks if a future version adds a third quantity to the checkpoint but forgets to update the reader? How does this argue for a self-describing format (Chapter 25)?
3. error stop on bad input halts the run. On a cluster that has already queued for two days, is failing fast the right call, or should the code try to recover with defaults? Argue both sides.

Your Turn: Extensions

• Option A. Add a checksum (a simple sum(field)) to the checkpoint, write it after the field, and have the reader recompute and compare it — printing a warning if the file was truncated or corrupted. This is the first step toward robust binary formats.
• Option B. Make the checkpoint filename carry the step number (chk_000042.bin) using an internal-file write to build the name (a technique Chapter 12 develops), so you keep a history of checkpoints rather than overwriting one.
• Option C. Write the provenance header as both a namelist echo (for machines) and a formatted comment block (for humans), and reflect on why a self-describing scientific format like NetCDF gives you both at once, for free (Chapter 25).

Key Takeaways

• A production I/O layer has four jobs: configure without recompiling, restart without loss, document itself, and reject bad input immediately. Each maps to one tool from this chapter.
• Multi-group namelists keep configuration organized by concern and let short files override only what they must; validation with error stop turns typos into loud, specific failures at startup.
• A checkpoint must be binary to restart a run exactly; stream I/O gives you exact, compact state with one write per object — at the price of keeping the read and write orders in lockstep.
• Echoing the namelist into the output makes results reproducible, closing the loop between what you ran and what you can prove you ran.