Chapter 34 — Teaching Notes

One-line purpose. Take the solver from shared memory (coarrays, OpenMP) to a genuine distributed-memory cluster with MPI, landing one durable pattern — domain decomposition with halo exchange — and the handful of mpi_* calls that implement it correctly (argument order, no deadlock, right collective).

Key ideas to emphasize

  • Private memory is the whole difference. Everything about MPI — send/receive, ghost cells, decomposition — follows from "no process can see another's variables." If students internalise that one fact, the rest is consequence. Contrast explicitly with OpenMP's shared arrays from the previous chapter.
  • The mpi_send argument order is load-bearing. buffer, count, datatype, dest, tag, comm, ierr. Drill it; most MPI bugs are a wrong slot. Have them recite the difference for mpi_recv (source, and a status).
  • The deadlock is the signature MPI bug and the one they will actually hit. Make the "works small, hangs large" mechanism (eager vs rendezvous) visceral, and make mpi_sendrecv the reflex cure. Case Study 1 is built around exactly this.
  • Halo exchange = the pattern. Domain decomposition + ghost cells + exchange-then-update is 80% of distributed scientific computing. The threshold idea — a process runs the unmodified serial stencil once its halo is filled — is the transferable insight; quarantine the parallelism into one routine.
  • mpi_reduce vs mpi_allreduce is a correctness choice, not a style choice (Case Study 2): a distributed stopping test needs allreduce, or non-root ranks loop forever.
  • Column-major decides the layout. Storing the strip as u(nx, 0:nloc+1) so a halo row is contiguous is the Chapter-5 theme paying off in a distributed setting — a nice callback.

Misconceptions to preempt

  • "MPI processes share memory like threads." (No — private memory; coordinate by messages.)
  • "mpi_send then mpi_recv on both sides is fine — it worked in my test." (Standard-mode deadlock; buffering masks it for small messages.)
  • "A speedup means the field changed." (No — the field is deterministic across process counts; only timing and print interleaving are not. This is the honesty anchor.)
  • "Any datatype will do if the count is right." (Datatype mismatch silently copies wrong bytes — no error.)
  • "mpi_reduce is basically mpi_allreduce." (Reduce → root only; using it for a stopping test strands the other ranks.)
  • "Exchange, update — order doesn't matter." (Update-first reads stale ghosts; edge rows drift and compound — Case Study 1's second bug.)
  • "MPI replaces OpenMP." (They compose: hybrid = one MPI rank per node × OpenMP within.)

A live demonstration (6–10 minutes)

Put example-01-hello.f90 on the projector and run it (conceptually) at -np 1, 2, 4 — the point is that ONE program text becomes N processes with different ranks, and that the line order is nondeterministic. Then show the deadlock: the two-send-first snippet, and reason aloud why it hangs (both stuck in send). Fix it live with mpi_sendrecv. Finally, sketch the halo picture (u(nx, 0:nloc+1), ghost rows) and trace on the board that rank 1's row only warms at step 3 because the halo carried rank 0's row across — the 0.8 from the Project Checkpoint. That single trace makes "why halos" click. (Do not present any timing as measured.)

Class-time budget (~55 min)

  • 8 min: the MPI model — private memory, SPMD, ranks, the skeleton, ierr (§34.1).
  • 12 min: point-to-point — argument order, datatype, and the deadlock + mpi_sendrecv (§34.2). This is the highest-value block; do not rush it.
  • 8 min: collectives — bcast, reduce vs allreduce, gather/scatter; why they beat hand-rolling (§34.3).
  • 15 min: domain decomposition + halo exchange — ghost cells, MPI_PROC_NULL, the exchange-then-update loop, the board trace; non-blocking overlap as a preview (§34.4, Project Checkpoint).
  • 6 min: running it — mpif90/mpirun, hybrid MPI+OpenMP, MPI-IO note (§34.5).
  • 6 min: the spaced review (coarray vs OpenMP vs MPI — same solver, three models).

Prerequisites to review

  • The Chapter 24 stencil and two-array FTCS update (the kernel that stays unchanged, and why it's data-parallel). One slide.
  • Chapter 5 column-major order (why the halo row is contiguous). One line.
  • Chapter 31's taxonomy (shared vs distributed vs GPU) and the Amdahl serial-fraction cap on I/O — this chapter is where "distributed" gets built. Chapter 33's data-race worry, for the contrast.

Connections

Backward: Ch. 31 (why distributed; Amdahl I/O cap), Ch. 32 (coarrays — same decomposition, implicit access), Ch. 33 (OpenMP — shared memory, the hybrid partner), Ch. 24 (the kernel), Ch. 5 (column-major). Forward: Ch. 35 (GPU — the last model), Ch. 38 (capstone assembles the hybrid MPI+OpenMP solver, validates, and writes it up). The Case Study 1 debugging discipline and Case Study 2 allreduce/non-blocking design both feed the capstone's scaling study.