Appendix G: Parallel Programming Reference

A side-by-side quick reference to Fortran's four parallel programming models — coarrays, OpenMP, MPI, and GPU offload (OpenACC and CUDA Fortran). It is the companion card to Part VIII: where Chapters 31–35 teach each model in depth, this appendix collects the syntax, the argument orders, and the compile-and-run commands in one place so you can look them up without re-reading a chapter. For the underlying toolchain (installing OpenMPI, OpenCoarrays, and the compiler flags themselves) see Appendix C.

Two things are worth saying once, up front. First, all four models parallelize the same computation in this book — the five-point heat-equation stencil frozen in Chapter 24 and carried behind the stable step interface. Coarrays (Chapter 32), OpenMP (Chapter 33), MPI (Chapter 34), and a GPU (Chapter 35) are four deliveries of one physics; the answer is identical, only the machine and the notation change. Section G.6 tabulates that correspondence directly. Second, an honesty note carried over from Chapter 35: the GPU examples were written to be exactly correct but, lacking a GPU and nvfortran, were not machine-checked, and a few coarray features (notably teams) are only partially supported by current gfortran — those caveats are flagged where they occur.

G.1 The four models at a glance

The one-line build-and-run for each (details in the per-model sections below):

$ gfortran -fcoarray=single -std=f2018 -Wall prog.f90 -o prog && ./prog   # coarrays, 1 image (test)
$ caf -std=f2018 prog.f90 -o prog && cafrun -n 4 ./prog                    # coarrays, 4 images
$ gfortran -std=f2018 -fopenmp -Wall prog.f90 -o prog && OMP_NUM_THREADS=4 ./prog   # OpenMP
$ mpif90 -std=f2018 -Wall -O2 prog.f90 -o prog && mpirun -np 4 ./prog      # MPI, 4 processes
$ nvfortran -acc -Minfo=accel prog.f90 -o prog && ./prog                   # OpenACC on the GPU

G.2 Coarrays (Fortran 2008/2018)

Coarrays add parallelism to the language itself: declare a variable with a codimension [*] and it exists as one independent copy per image, any of which can reach any other's copy with a coindexed reference a[q].

Declaration and image queries

The () bounds are the local shape (identical on every image); the [] bounds are the coshape (the arrangement of images). The trailing * means "as many images as the program was launched with."

Remote (coindexed) access

x   = a[q]         ! GET: read image q's copy of a
a[q] = x           ! PUT: write into image q's copy
col = u(:,1)[q]    ! remote array section (one contiguous column from image q)

An unbracketed a is always this image's local copy (fast); a bracketed a[q] may be a network transfer — so communicate rarely and in bulk (whole columns, not element-by-element).

Synchronization

The correctness rule: a remote read must be ordered against remote writes by a synchronization — the pattern write → sync all → read is race-free; without the barrier the program is undefined.

use, intrinsic :: iso_fortran_env, only: lock_type
type(lock_type) :: lck[*]
lock(lck)
  ! ... exclusive access to a shared resource ...
unlock(lck)

Collective subroutines (Fortran 2018)

Every image must call a collective, with matching arguments; the result is written back to the argument on every image (or, with result_image, on just one).

co_sum(x, result_image=1) leaves the sum only on image 1 (the analogue of MPI's mpi_reduce); co_sum(x) leaves it on all (the analogue of mpi_allreduce).

Teams (brief)

Teams partition the images into named subsets that each behave, inside a change team block, as a self-contained image set (this_image()/num_images() become team-relative).

use, intrinsic :: iso_fortran_env, only: team_type
type(team_type) :: t
form team(team_number, t)          ! each image chooses a team number
change team(t)
  ! this_image() / num_images() are now RELATIVE to this image's team
end team

⚠️ gfortran caveat — teams are the least-supported corner of coarrays. Teams (form team / change team) are fully standard (Fortran 2018) but incomplete or absent in current gfortran + OpenCoarrays — a program using them may fail to compile or link. Treat teams as a model to recognize, not one to depend on with gfortran today; Intel's ifx/ifort has led on 2018 coarray features. The collectives (co_sum and friends) are well supported with OpenCoarrays. Verify on your exact toolchain.

Compile and run

$ gfortran -fcoarray=single -std=f2018 -Wall prog.f90 -o prog && ./prog   # 1 image, for testing logic
$ caf -std=f2018 -Wall prog.f90 -o prog                                    # OpenCoarrays wrapper
$ cafrun -n 4 ./prog                                                        # run on 4 images

gfortran alone runs coarray code only as a single degenerate image (-fcoarray=single, where num_images() is 1 and every sync all is a no-op). Multi-image execution on gfortran requires the external OpenCoarrays runtime: caf wraps gfortran -fcoarray=lib + the library, and cafrun -n N launches N images (wrapping mpirun). Intel builds coarrays in (ifx -coarray, image count via the FOR_COARRAY_NUM_IMAGES environment variable) with no separate launcher.

G.3 OpenMP (shared-memory threads)

OpenMP parallelizes a serial loop by annotating it with a directive (a specially formatted comment, !$omp …). Without -fopenmp the directives are ignored and the same source builds a correct serial program; with it, the loop's iterations are split across a team of threads that all share one memory.

Directives

Data-sharing clauses

Reduction operators: +, *, -, max, min, .and., .or., .eqv., .neqv., iand, ior, ieor. The inner loop index and every scratch temporary are your responsibility to scope — only the !$omp do loop variable is made private automatically.

Synchronization and scheduling

For combining per-thread results prefer, in order: reduction → atomic → critical.

Runtime library and environment (use omp_lib)

Compile and run

$ gfortran -std=f2018 -fopenmp -Wall prog.f90 -o prog
$ OMP_NUM_THREADS=4 ./prog

The -fopenmp flag both enables the directives and links the runtime that provides omp_lib. Omit it and your "parallel" program is silently serial.

G.4 MPI (distributed-memory processes)

MPI runs many processes, each with private memory, coordinating by explicit messages. It is a library (Open MPI, MPICH), not part of Fortran; a program uses it with use mpi and mpi_* calls. Every mpi_* call takes ierr (an integer status) as its last argument.

The skeleton

use mpi                                         ! or: use mpi_f08 (modern — see below)
implicit none
integer :: ierr, rank, nprocs
integer :: status(MPI_STATUS_SIZE)

call mpi_init(ierr)                             ! start the runtime
call mpi_comm_rank(MPI_COMM_WORLD, rank,   ierr) ! my rank, 0 .. nprocs-1
call mpi_comm_size(MPI_COMM_WORLD, nprocs, ierr) ! how many processes
! ... work ...
call mpi_finalize(ierr)                          ! shut the runtime down

Point-to-point communication (mind the argument order)

call mpi_send(buf, count, datatype, dest,   tag, comm, ierr)
!              buf  count  datatype  dest    tag  comm  ierr
call mpi_recv(buf, count, datatype, source, tag, comm, status, ierr)
!              buf  count  datatype  source  tag  comm  status  ierr

call mpi_sendrecv(sendbuf, sendcount, sendtype, dest,   sendtag,  &  ! send half
                  recvbuf, recvcount, recvtype, source, recvtag,  &  ! recv half
                  comm, status, ierr)

call mpi_isend(buf, count, datatype, dest,   tag, comm, request, ierr)  ! non-blocking
call mpi_irecv(buf, count, datatype, source, tag, comm, request, ierr)  ! non-blocking
call mpi_wait(request, status, ierr)                                    ! complete one
call mpi_waitall(count, requests, statuses, ierr)                       ! complete several

Six things must line up for delivery: buffer, count, datatype, dest/source, tag, communicator. A paired send-then-receive on both ranks can deadlock on large messages (it "works" for small ones that MPI buffers) — use mpi_sendrecv (one call, cannot deadlock) or order the calls so one side receives first. With mpi_isend/mpi_irecv, do not touch the buffer until the matching mpi_wait returns.

Collective communication

call mpi_bcast(buffer, count, datatype, root, comm, ierr)                 ! one -> all
call mpi_scatter(sendbuf, sendcount, sendtype,  &                         ! one -> all (pieces)
                 recvbuf, recvcount, recvtype, root, comm, ierr)
call mpi_gather (sendbuf, sendcount, sendtype,  &                         ! all -> one
                 recvbuf, recvcount, recvtype, root, comm, ierr)
call mpi_reduce(sendbuf, recvbuf, count, datatype, op, root, comm, ierr)  ! combine -> root
call mpi_allreduce(sendbuf, recvbuf, count, datatype, op, comm, ierr)     ! combine -> all
call mpi_barrier(comm, ierr)                                              ! all wait

Every process in the communicator must make the matching call, or the collective hangs. In mpi_gather and mpi_scatter, recvcount/sendcount is the count per process, not the total. mpi_reduce has a root; mpi_allreduce does not (everyone gets the result).

Datatypes and reduction operations

Reduction ops: MPI_SUM, MPI_PROD, MPI_MAX, MPI_MIN, MPI_LAND, MPI_LOR, MPI_MAXLOC, MPI_MINLOC. Useful constants: MPI_COMM_WORLD, MPI_PROC_NULL (a no-op rank, for grid edges), MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_STATUS_SIZE, MPI_SUCCESS.

🔧 use mpi vs mpi_f08. Three interfaces exist, in ascending modernity: the legacy include 'mpif.h' (no compile-time checking — avoid in new code); use mpi (this book's default — handles are plain integers, ierr mandatory); and use mpi_f08 (from MPI-3, preferred for new code — handles become derived types type(MPI_Comm), type(MPI_Datatype), type(MPI_Status), type(MPI_Request), and ierr becomes a genuinely optional final argument, so call mpi_init() is valid). We teach use mpi because it is what most existing code uses; reach for mpi_f08 in fresh code.

Compile and run

$ mpif90 -std=f2018 -Wall -O2 prog.f90 -o prog     # mpif90 (or mpifort) = gfortran + MPI flags
$ mpirun -np 4 ./prog                              # launch 4 processes (mpiexec is equivalent)

mpif90 is not a different compiler — it is gfortran with MPI's include/link paperwork filled in, so all the optimization flags of Appendix C still apply. -np N may exceed the physical core count (processes time-share) for testing a decomposition.

G.5 GPU: OpenACC and CUDA Fortran

Two roads onto the graphics processor: OpenACC directives (portable, incremental — the easy path) and CUDA Fortran explicit kernels (NVIDIA-only, more control). The governing fact for both is that the device has its own memory: the dominant cost is moving data across the host↔device bridge, so the rule is move data once, compute on it many times.

OpenACC directives

OpenACC data clauses (on !$acc data` or `!$acc parallel loop)

The canonical pattern — one data region around a whole time loop, present on each step's kernel — is what makes an offload win instead of embarrass:

!$acc data copy(u) create(u_new)          ! field RESIDENT across ALL steps: one copy in, one out
do step = 1, nsteps
  !$acc parallel loop collapse(2) present(u, u_new)   ! no per-step transfer
  do j = 2, ny-1
    do i = 2, nx-1
      u_new(i,j) = u(i,j) + r*( u(i-1,j) + u(i+1,j) + u(i,j-1) + u(i,j+1) - 4.0_dp*u(i,j) )
    end do
  end do
  ! ... commit u_new into u ...
end do
!$acc end data

CUDA Fortran (use cudafor)

attributes(global) subroutine saxpy_kernel(a, x, y, n)
  real(dp), value :: a
  integer,  value :: n
  real(dp) :: x(n), y(n)                           ! dummy arrays live in device memory
  integer :: i
  i = (blockIdx%x - 1)*blockDim%x + threadIdx%x    ! ONE-based global index
  if (i <= n) y(i) = a*x(i) + y(i)                 ! guard the tail
end subroutine saxpy_kernel

⚠️ CUDA Fortran thread indices are ONE-based. In CUDA C the index is blockIdx.x*blockDim.x + threadIdx.x (0-based); in CUDA Fortran threadIdx%x and blockIdx%x count from 1, so the correct global index is (blockIdx%x - 1)*blockDim%x + threadIdx%x. Copy the C formula verbatim and you are off by one at every block boundary. (blockDim/gridDim are counts, not indices — they are not shifted.)

Compile and run

$ nvfortran -acc -Minfo=accel prog.f90 -o prog     # OpenACC on the GPU (-Minfo=accel reports what offloaded)
$ nvfortran -acc=multicore prog.f90 -o prog        # OpenACC targeting CPU cores instead
$ gfortran  -fopenacc prog.f90 -o prog             # OpenACC via gfortran (offload maturity trails nvfortran)
$ nvfortran -cuda prog.f90 -o prog                 # CUDA Fortran (NVIDIA only; a *.cuf file enables it too)

A note on these GPU examples. As stated in Chapter 35, this book has no GPU and no nvfortran to compile against, so the OpenACC and CUDA Fortran syntax above is written to be exactly correct but was not machine-checked, and any performance figure is an illustrative order of magnitude, never a promise. CUDA Fortran has no gfortran equivalent — it is nvfortran-only.

G.6 The same idea in four dialects

The deepest payoff of Part VIII is seeing that the four models are four spellings of one small set of ideas. Learn to read across a row and the next model stops being a fresh mountain.

The row that recurs three times over — co_sum, reduction(+:s), mpi_allreduce(...,MPI_SUM,...) (and OpenACC's reduction(+:s)) — is one operation in four dialects: combine a value across all the parallel workers. Recognizing it is much of what makes the later models feel like variations.

G.7 Which model when?

Portability, honestly weighed. OpenMP is the most universal for shared memory — built into gfortran with just -fopenmp. Coarray syntax is standard and portable, but multi-image execution on gfortran depends on the external OpenCoarrays install (Intel's ifx builds it in), and teams are weakly supported on gfortran today (G.2). MPI is a library you install (Open MPI / MPICH) — ubiquitous on clusters, not part of the language. OpenACC is most mature on nvfortran (gfortran's -fopenacc is catching up); CUDA Fortran is NVIDIA-only. When in doubt for a single node, OpenMP is the smallest, most portable step; for a cluster, MPI is the safe default; coarrays are the elegant native alternative that spans both.

All four solve the same problem. Every model in this appendix parallelizes the identical five-point heat stencil of Chapter 24, behind the same frozen step interface, and each chapter's Project Checkpoint reaches the same deterministic field — only the schedule and the machine differ. Coarrays and MPI use the same 1D domain decomposition and the same halo (coindexed read vs. explicit mpi_sendrecv); OpenMP and OpenACC parallelize the same interior sweep (threads vs. GPU cores). The physics is invariant; the parallelism is the wardrobe. If a parallel run ever disagrees with the serial answer, you have a bug — a scoping mistake, a missing synchronization, or an un-exchanged halo — not a faster result.