Chapter 34 — Key Takeaways (MPI)

A one-page reference to distributed-memory Fortran with MPI: the model, the calls, and the halo-exchange pattern that scales a stencil code across a cluster. Keep it beside you through the capstone.

The MPI model in one breath

Many processes, each with private memory, coordinating by explicit messages. SPMD: mpirun launches N copies of one program; each learns its rank (0…N−1) and specialises. No shared state — if you need a neighbour's data, you send or receive it.

The skeleton (every MPI program)

use mpi
integer :: ierr, rank, nprocs
call mpi_init(ierr)
call mpi_comm_rank(MPI_COMM_WORLD, rank,   ierr)   ! who am I?  (0-based)
call mpi_comm_size(MPI_COMM_WORLD, nprocs, ierr)   ! how many of us?
!  ... work, specialised by rank ...
call mpi_finalize(ierr)

Every mpi_* call ends with ierr in the use mpi interface — it is not optional.

The calls (get the argument order exactly right)

Call Signature (use mpi) Note
mpi_send (buf, count, datatype, dest, tag, comm, ierr) memorize this order
mpi_recv (buf, count, datatype, source, tag, comm, status, ierr) adds status before ierr
mpi_sendrecv (sbuf,cnt,typ,dest,stag, rbuf,cnt,typ,src,rtag, comm,status,ierr) one deadlock-free exchange
mpi_isend/mpi_irecv (buf,count,datatype,dest/src,tag,comm, request, ierr) non-blocking; gives a request
mpi_wait / mpi_waitall (request, status, ierr) / (count, reqs, statuses, ierr) completes non-blocking ops
mpi_bcast (buffer, count, datatype, root, comm, ierr) one → all
mpi_reduce (sbuf, rbuf, count, datatype, op, root, comm, ierr) all → root
mpi_allreduce (sbuf, rbuf, count, datatype, op, comm, ierr) all → all (no root)
mpi_gather/mpi_scatter (sbuf,scnt,typ, rbuf,rcnt,typ, root, comm, ierr) collect / deal out

Datatypes: MPI_DOUBLE_PRECISION for real(dp) (= real64), MPI_INTEGER for integer, MPI_REAL for default real. Reduce ops: MPI_SUM, MPI_MAX, MPI_MIN, MPI_PROD.

Which collective? (decision aid)

You need… Collective
one process's value on all processes mpi_bcast
deal one array out in pieces, one per process mpi_scatter
collect each process's piece onto one mpi_gather
combine (sum/max/…) onto one process mpi_reduce
combine and give the result to every process (e.g. a stopping test) mpi_allreduce

The deadlock, and its cure

! DEADLOCK RISK: both ranks send first (works for small messages, hangs for large ones)
call mpi_send(mine,  n, ..., other, 0, comm, ierr)
call mpi_recv(yours, n, ..., other, 0, comm, status, ierr)

! CURE: one call that does both, guaranteed not to deadlock
call mpi_sendrecv(mine,  n, ..., other, 0,  yours, n, ..., other, 0,  comm, status, ierr)

mpi_send is standard mode: it may buffer small messages (so the bug hides in testing) but blocks on large ones. Never rely on buffering for correctness.

Domain decomposition + halo exchange (the pattern that scales)

  1. Cut the plate into strips, one per process; store each as u(nx, 0:nloc+1) — full width on the first index (contiguous rows, column-major), owned rows 1..nloc, ghost rows at 0 and nloc+1.
  2. Each step: exchange halos first, then run the unmodified Chapter 24 stencil over owned rows.
  3. Exchange with two mpi_sendrecv calls (up, down); use MPI_PROC_NULL for the physical edges → no boundary if, edge ghosts hold the fixed Dirichlet values untouched.
up   = rank-1; if (rank==0)        up   = MPI_PROC_NULL
down = rank+1; if (rank==nprocs-1) down = MPI_PROC_NULL
call mpi_sendrecv(u(:,1),    nx, MPI_DOUBLE_PRECISION, up,   0,  &
                  u(:,0),    nx, MPI_DOUBLE_PRECISION, up,   0,  comm, status, ierr)
call mpi_sendrecv(u(:,nloc), nx, MPI_DOUBLE_PRECISION, down, 0,  &
                  u(:,nloc+1),nx,MPI_DOUBLE_PRECISION, down, 0,  comm, status, ierr)

Non-blocking (overlap): post mpi_irecv/mpi_isend, compute the ghost-independent deep interior, then mpi_waitall and finish the edge rows. Worth it only when strips are fat (surface-to-volume).

Build and run

$ mpif90 -std=f2018 -Wall -O2 heat_mpi.f90 -o heat_mpi   # mpif90 = gfortran + MPI flags
$ mpirun -np 4 ./heat_mpi                                 # launch 4 processes
  • Hybrid MPI+OpenMP: one MPI rank per node (crosses nodes) × OpenMP threads within (fills each node).
  • MPI-IO (via parallel HDF5/NetCDF): parallel output so writing does not funnel through one rank.
  • use mpi_f08 is the modern interface (typed handles, optional ierr); prefer it in new code.

Pitfalls

  • Forgetting ierr (mandatory in use mpi), or swapping rank/ierr in mpi_comm_rank.
  • Send/recv deadlock — both ranks send first; cure with mpi_sendrecv.
  • Datatype mismatch (e.g. MPI_INTEGER for real(dp)) — no error, wrong bytes.
  • Skipping a collective on some ranks — the collective hangs; all must call it.
  • Exchanging halos after the update — edge rows read stale ghosts; exchange first.
  • Touching non-blocking buffers before mpi_wait — a race on data still in flight.

Numbers and rules worth carrying

  • mpi_send order: buffer, count, datatype, dest, tag, comm, ierr — the single most-fumbled thing.
  • Reduce → root only; all-reduce → everyone (use it for global convergence tests).
  • MPI standardised 1994; portable across a laptop and the largest cluster on Earth.
  • A correct decomposition prints the identical field on any process count — that is your check.

Project piece added this chapter

The solver goes distributed: 1D domain decomposition (one strip per process), ghost rows, and a mpi_sendrecv halo exchange each step, with mpi_allreduce for a global convergence measure and mpi_gather to assemble output. The Chapter 24 stencil is unchanged. Hand-computed on a $5 \times 6$ plate, -np 2, $r = 0.2$:

after step 3:  100 100 100 100 100
                 0 32.8 38.4 32.8 0
                 0  7.2  8.8  7.2 0
                 0  0.8  0.8  0.8 0     <- appears ONLY because the halo crossed the process boundary
                 0   0    0    0  0
                 0   0    0    0  0

The parallel solver is assembled into the hybrid MPI+OpenMP capstone in Chapter 38. Next, the GPU: Chapter 35.