Exercises: MPI — Distributed-Memory Parallelism

These exercises move from the MPI model (getting the calls and their arguments exactly right) to the pattern that matters most — domain decomposition with halo exchange. Message passing is unforgiving: a mismatched argument does not usually crash, it silently sends the wrong bytes or hangs, so the single most valuable habit to build here is reading an mpi_* call slot by slot — buffer, count, datatype, rank, tag, communicator, ierr — until it is automatic. Where a problem asks you to run something, launch it with mpirun -np N and predict the output first, noting which parts are process-count-dependent and which are truly nondeterministic (like the interleaving of concurrent prints).

Difficulty: ⭐ warm-up · ⭐⭐ standard · ⭐⭐⭐ deeper. Solutions: worked solutions to the daggered (†) and odd-numbered problems are in appendices/answers-to-selected.md; the code ones are also worked as compilable, hand-checked programs in code/exercise-solutions.f90. Try every problem before you look. Compile MPI code with the wrapper: mpif90 -std=f2018 -Wall.

Part A — The MPI Model ⭐

34.1 † In one or two sentences each: what is fundamentally different about memory in the MPI model compared with OpenMP, and — given that difference — how does one process obtain a value that another process computed?

34.2 What do mpi_comm_rank and mpi_comm_size each return, and why does an SPMD program (one program text, run by every process) need both to divide up its work?

34.3 † You run the example-01-hello.f90 program with mpirun -np 3. Write down exactly what lines appear. Which part of each line is process-count-dependent, and what about the output is not guaranteed from run to run?

34.4 Why does call mpi_init() fail to compile against the use mpi interface, while it does compile against use mpi_f08? Give the corrected use mpi call.

Part B — Point-to-Point ⭐⭐

34.5 † From memory, list the seven arguments of mpi_send in order, with a word on each. Then state the two ways mpi_recv's argument list differs from it.

34.6 Predict the output of this two-process program (launched with -np 2), then say which rank prints.

if (rank == 0) then
  x = 7.0_dp
  call mpi_send(x, 1, MPI_DOUBLE_PRECISION, 1, 0, MPI_COMM_WORLD, ierr)
else if (rank == 1) then
  call mpi_recv(x, 1, MPI_DOUBLE_PRECISION, 0, 0, MPI_COMM_WORLD, status, ierr)
  print '(a,f4.1)', 'got ', 2.0_dp*x
end if

34.7 † Suppose you send an array of real(dp) but pass MPI_INTEGER as the datatype (everything else correct). Does the program crash? What does the receiver end up with, and why is this class of bug so dangerous? State the rule that prevents it.

34.8 (Type, compile, and run.) Modify example-02-send-recv.f90 so that rank 0 sends the integer array [1, 2, 3, 4] to rank 1, which prints its sum. What datatype must you use, and what does it print? Confirm against code/exercise-solutions.f90.

Part C — Collectives ⭐⭐

34.9 † Run example-03-collectives.f90 with mpirun -np 8. Working from the formulas, what do the "sum" and "max" lines print? Give the general result for -np N.

34.10 In one sentence each, state when you would choose mpi_reduce and when you would choose mpi_allreduce, and name a use for each in the heat solver.

34.11 † A colleague's program hangs. Inspecting it, you find mpi_bcast is called by every process except inside a branch that only rank 0 takes. Explain precisely why the program hangs, and give the rule about collectives that it violates.

34.12 Match each communication need to the single best collective (mpi_bcast, mpi_scatter, mpi_gather, mpi_reduce, mpi_allreduce): (a) rank 0 has read a configuration value that all processes need; (b) each process has computed a local error and every process must know the global maximum to decide whether to stop; (c) each process holds one strip of the final field and rank 0 must assemble the whole plate to write it; (d) rank 0 holds the full initial field and must hand each process its strip.

Part D — Find the Bug ⭐⭐

34.13 † This exchange between two neighbouring ranks passes a small test and then hangs on a big run. Name the bug, explain why the message size matters, and rewrite it as a single deadlock-free call.

call mpi_send(mine,  n, MPI_DOUBLE_PRECISION, other, 0, MPI_COMM_WORLD, ierr)
call mpi_recv(yours, n, MPI_DOUBLE_PRECISION, other, 0, MPI_COMM_WORLD, status, ierr)

34.14 A program is meant to sum a value across all ranks onto rank 0, but rank 0 prints garbage. The reduce call is call mpi_reduce(mine, total, 1, MPI_DOUBLE_PRECISION, MPI_SUM, MPI_COMM_WORLD, ierr). Compare it to the signature in §34.3 and identify what is missing.

34.15 † In a halo exchange, one rank sends its edge row with count = nx but the receiving rank posts its receive with count = nx - 2 (it forgot the two boundary columns). Against the argument table in §34.2, what is wrong, and what is the symptom — a crash, a hang, or silently wrong data?

34.16 A process updates its strip before calling exchange_halos instead of after. The program runs, produces no error, and gives slightly wrong temperatures near the strip boundaries that get worse each step. Explain what the process is computing its edge rows from, and why the error compounds.

Part E — Domain Decomposition and the Solver ⭐⭐⭐

34.17 † Define ghost cell and halo exchange in your own words, and explain the key payoff: why, once the halo is filled, a process can run the unmodified serial Chapter 24 stencil update over its owned rows.

34.18 † (Design it — solver.) Rewrite the project's exchange_halos to use non-blocking communication (mpi_irecv, mpi_isend, mpi_waitall) instead of mpi_sendrecv. Post the receives first, then the sends, then wait. What independent work could a real solver do between the posts and the wait, and what must you not touch before mpi_waitall? Confirm your version compiles against code/exercise-solutions.f90.

34.19 (Design it — solver.) The project uses a 1D decomposition (horizontal strips). Sketch — in prose and a small diagram, no full code — what changes for a 2D decomposition (a grid of rectangular tiles): how many neighbours does an interior tile have, which halos must it exchange, and what new complication appears at the tile corners?

34.20 † Explain why the solver stores each strip as u(nx, 0:nloc+1) — the full width on the first index, the decomposed direction on the second — rather than the other way around. Tie your answer to column-major order (Chapter 5) and say what would go wrong (for the halo message) if you split the first index instead.

Part F — Back of the Envelope ⭐⭐⭐

34.21 † A square plate of $N \times N$ cells is split into $P$ horizontal strips (a 1D decomposition), one per process. (a) How many real numbers does one interior process send per step (both halos)? (b) How many does it compute per step (its interior update)? (c) Write the ratio of communication to computation, and say what happens to it as $P$ grows with $N$ fixed.

34.22 Using your ratio from 34.21, and taking $N = 1000$: at roughly what number of strips $P$ does each process's communication (edge rows) become comparable to, say, 1% of its computation? What does this tell you about how thin you can profitably slice a fixed plate — and which Chapter 31 diagnostic would reveal it on real timings?

34.23 † Compare the total halo data exchanged per step for a $1000 \times 1000$ plate on $P = 64$ processes under (a) a 1D strip decomposition and (b) a 2D tile decomposition ($8 \times 8$ tiles). Which moves less data across the network, and by roughly what factor? State the general reason 2D wins at scale.

Part G — Interleaved (Chapters 32, 33; also 5, 24, 31) ⭐⭐

34.24 † (Ch. 32.) Coarrays (Chapter 32) and MPI decompose the plate identically and exchange the same halo, yet the mechanism differs. In one sentence each, describe how a process obtains its neighbour's edge row under coarrays versus under MPI, and state which of the two is part of the Fortran standard itself.

34.25 (Ch. 33.) Why did OpenMP (Chapter 33) parallelise the update loop with a single directive and no change to the array, while MPI forced you to split the field and add ghost cells? Name the one property of shared versus distributed memory responsible for the difference.

34.26 † (Ch. 31.) Chapter 31's Amdahl estimate warned that the solver's serial I/O (gathering to rank 0 to write) caps its speedup. Explain how MPI-IO (via parallel HDF5) attacks that cap, and which of Amdahl's or Gustafson's laws it is trying to improve on.

34.27 (Ch. 24.) The distributed solver's inner update is byte-for-byte the Chapter 24 stencil. Explain why keeping that kernel unchanged across the coarray, OpenMP, and MPI versions is a feature, not a coincidence — and what design decision from Chapter 24 made it possible.

34.28 † (Ch. 5, Port it.) Here is an mpi4py fragment that finds the global maximum of a per-process value and gives it to every process:

import numpy as np
from mpi4py import MPI
comm = MPI.COMM_WORLD
mine = np.array(comm.Get_rank() + 1, dtype='float64')
gmax = np.zeros(1)
comm.Allreduce(mine, gmax, op=MPI.MAX)

Translate it to Fortran + MPI. What is gmax on every rank when launched with -np 5? Confirm against code/exercise-solutions.f90.

Solutions to the daggered and odd-numbered problems are in appendices/answers-to-selected.md; the code ones are worked as compilable, hand-checked programs in code/exercise-solutions.f90. The design problems (34.18, 34.19) have model answers plus room for your own reasoning — if your halo exchange posts receives before sends, cannot deadlock, and leaves the physics kernel untouched, you are on the right track.