Part VIII: Parallel Programming

"The free lunch is over." — Herb Sutter, on the end of automatic speedups from rising clock rates

Around 2005, processor clock speeds stopped climbing. The transistors kept coming, but they went into more cores rather than faster ones, and from that moment the only way to make a computation meaningfully faster was to make it run on many processors at once. This is why a supercomputer is not one enormous CPU but hundreds of thousands of ordinary ones wired together — and why parallel programming stopped being an exotic specialty and became the central skill of high-performance computing. Part VIII is where your Fortran leaves the single core behind.

Fortran is unusually well equipped for this. It has a native parallel model — coarrays, built into the language itself — that most languages can only envy. It works seamlessly with OpenMP for the multiple cores of one machine and with MPI for the many machines of a cluster, the two workhorses of real HPC. And it can offload to GPUs. We take them in turn, always with the same running example: the heat solver you have built, profiled, and optimized now learns to decompose its plate across images, threads, processes, and finally a graphics processor. By the end, you will have written code that scales — the thing this entire book has been building toward.

What You Will Learn

Chapter 31 — Why Parallel? The end of clock scaling, Amdahl's and Gustafson's laws, the taxonomy of parallelism, and how to think about parallelizing before you write any parallel code.

Chapter 32 — Coarrays. Fortran's built-in parallelism: images, codimensions, synchronization, and the 2018 collectives — SPMD parallelism with no external library.

Chapter 33 — OpenMP. Shared-memory parallelism by directive: parallel regions, work sharing, data scoping, reduction, and scheduling.

Chapter 34 — MPI. Distributed-memory parallelism for clusters: ranks and communicators, point-to- point and collective communication, and domain decomposition with halo exchange.

Chapter 35 — GPU Computing. OpenACC and CUDA Fortran: offloading data-parallel work to graphics processors, and knowing when it pays off.

How This Part Fits

Part VIII assumes performance (Part VII) — you parallelize code you have already made fast — and it leans on arrays (Chapter 5) and modules (Chapter 8). It is the destination of the HPC track and the climax of the running project; the capstone in Chapter 38 assembles the parallel solver into a finished piece of computational science.

Time Investment

Chapter Title Difficulty Est. hours
31 Why Parallel? Intermediate 4
32 Coarrays Advanced 6
33 OpenMP Advanced 6
34 MPI Advanced 6
35 GPU Computing Advanced 5
Part VIII total ~27 hours

Begin with Chapter 31 and the hardware reality — Amdahl's Law — that governs every parallel program ever written.

Chapters in This Part