Case Study 1: Auditing a Coarray Reduction

"It passed on my laptop, and on the cluster it gives a different answer every time."

Executive Summary

A colleague hands you a short coarray routine that computes the squared 2-norm of a vector distributed across images — the sum of the squares of all its elements. On their laptop, built with -fcoarray=single, it prints the right answer every time. On the cluster, on 4 images, it prints a different answer on almost every run, always too small. They are convinced it is a compiler or network bug. It is not: it is two classic coarray mistakes hiding behind a test that could never expose them. This study reads their code, diagnoses both faults precisely in the language of segments and coindexed access, and repairs it two ways — once by hand with per-image slots and a barrier, and once by deleting the hand-rolled communication in favour of the co_sum collective. Along the way you will see why the single-image test passed — the most important lesson, because it is why the bug shipped. Nothing here requires a cluster; the reasoning is the tool, and one careful read is worth more than a hundred reruns.

Skills applied: the SPMD model and private per-image data (§32.1); coindexed access a[q] (§32.2); segments and why a remote access must be synchronized (§32.3); mutual exclusion vs the wrong single-slot pattern (§32.3); the collective co_sum and why it is the right tool (§32.4); single-image testing with -fcoarray=single and its blind spot (§32.5).

Background

The task is a distributed reduction: a vector's elements are spread across the images (each image holds a contiguous chunk), and we want $\sum_i x_i^2$, summed over the whole vector. To make every number checkable by hand, take the vector to be the integers $1$ through $12$, so the answer is fixed and known:

$$ \sum_{i=1}^{12} i^2 = \frac{12 \cdot 13 \cdot 25}{6} = 650. $$

A correct reduction must return 650 no matter how many images the run uses — 1, 2, 3, 4, 6, or 12 — because the mathematics does not depend on how we chopped up the work. That image-count independence is the acceptance test, and it is exactly the test the buggy code fails.

Phase 1 — The Symptom

Here is the routine as delivered. Each image sums the squares over its own chunk, then adds that partial into a single coarray accumulator held on image 1; image 1 reads the total:

program buggy_norm
  implicit none
  integer, parameter :: L = 12
  integer :: me, ni, chunk, lo, hi, i, part
  integer :: acc[*]                          ! a SINGLE accumulator slot, on image 1
  me = this_image();  ni = num_images()
  chunk = L / ni;  lo = (me-1)*chunk + 1;  hi = me*chunk

  part = 0
  do i = lo, hi
    part = part + i*i                        ! this image's partial sum of squares
  end do

  acc[1] = acc[1] + part                     ! (BUG) every image reads-modifies-writes acc[1]...
  sync all                                   ! ...and the sync is AFTER the damage is done

  if (me == 1) print '(a,i0)', 'sum of squares = ', acc[1]
end program buggy_norm

Built single-image, it is flawless:

$ gfortran -fcoarray=single -std=f2018 -Wall buggy_norm.f90 -o bn && ./bn
sum of squares = 650

Run on 4 images, it misbehaves — and inconsistently. A typical session:

$ cafrun -n 4 ./bn
sum of squares = 650
$ cafrun -n 4 ./bn
sum of squares = 573
$ cafrun -n 4 ./bn
sum of squares = 456

The number is different each time and never exceeds the correct 650. A value that changes from run to run is the unmistakable fingerprint of a race condition. Two distinct faults produce it.

Phase 2 — The Diagnosis

Fault 1: acc[1] is never initialized, and the accumulate races. Look at acc[1] = acc[1] + part executed on four images at once. Each image must read acc[1], add its part, and write acc[1] back. Nothing orders these read-modify-write sequences against each other. Two images can both read the same old acc[1], both add their partials to it, and both write back — and the second write overwrites the first, so one image's partial is silently lost. That is a lost update, and it is why the total is always too small: some partials vanish. Worse, in the language of §32.3, the images are defining the coarray acc in segments that are not ordered with respect to one another, which the standard declares simply undefined — the program has no guaranteed meaning at all. (And acc is never set to 0, so even the first read is of an undefined value.)

Fault 2: the sync all is in the wrong place to help. The barrier sits after the racy accumulate, so it does nothing to order the competing writes — they have already collided by the time any image reaches it. A barrier can only order accesses that straddle it; here both the writes and image 1's read are on the same side of the wrong side. Placing the sync correctly would fix the read (Fault-2 half), but the accumulate would still race (Fault 1). Both must be addressed.

Why the single-image test passed — the crucial lesson. With -fcoarray=single there is exactly one image. There is no second image to race against, so the read-modify-write of acc[1] cannot be interleaved, and sync all is a no-op that is trivially in the "right" place. The one-image build exercises the code path but cannot exercise the concurrency, so it hides every synchronization bug the program has. This is the blind spot of -fcoarray=single: it is perfect for checking logic and worthless for checking races. A green single-image test is necessary, never sufficient.

Phase 3 — Fix A: Per-Image Slots and a Barrier

The disciplined hand-rolled fix removes the shared write entirely. Give every image its own slot to write — so no two images ever touch the same location — synchronize once, then let image 1 read and sum all the slots:

program norm_fix_a
  implicit none
  integer, parameter :: L = 12
  integer :: me, ni, chunk, lo, hi, i, total, q
  integer :: part[*]                         ! one slot PER IMAGE -- no shared write
  me = this_image();  ni = num_images()
  chunk = L / ni;  lo = (me-1)*chunk + 1;  hi = me*chunk

  part = 0
  do i = lo, hi
    part = part + i*i                        ! each image writes ONLY its own part
  end do
  sync all                                   ! all partials written before any are read

  if (me == 1) then
    total = 0
    do q = 1, ni
      total = total + part[q]                ! coindexed READ of each image's own slot
    end do
    print '(a,i0)', 'sum of squares = ', total
  end if
end program norm_fix_a

$ caf -std=f2018 -Wall norm_fix_a.f90 -o na && cafrun -n 4 ./na
sum of squares = 650

Now every image writes only part (its private copy), so there is no contention; the single sync all cleanly separates all the writes (before) from image 1's reads (after); and image 1 sums the four slots $14 + 77 + 194 + 365 = 650$. Run it on 1, 2, 3, or 6 images and it prints 650 every time — the acceptance test passes. The pattern — write your own, synchronize, one image gathers — is the same one the chapter's gather examples use, and it is always correct because no location is ever written by more than one image.

Phase 4 — Fix B: Delete the Communication, Call co_sum

Fix A is correct but it is still code you can get wrong — the slot indexing, the barrier placement, the gather loop. The reduction across images is such a common operation that Fortran 2018 gives you an intrinsic for it, and the right fix is to throw the hand-rolled version away:

program norm_fix_b
  implicit none
  integer, parameter :: L = 12
  integer :: me, ni, chunk, lo, hi, i, ssq
  me = this_image();  ni = num_images()
  chunk = L / ni;  lo = (me-1)*chunk + 1;  hi = me*chunk

  ssq = 0
  do i = lo, hi
    ssq = ssq + i*i                          ! local partial
  end do
  call co_sum(ssq)                           ! combine across ALL images, correctly, in one line

  if (me == 1) print '(a,i0)', 'sum of squares = ', ssq
end program norm_fix_b

$ caf -std=f2018 -Wall norm_fix_b.f90 -o nb && cafrun -n 4 ./nb
sum of squares = 650

co_sum(ssq) combines every image's ssq and leaves the total on every image — no slots, no barrier to place, no gather loop, no race possible, because the collective carries its own synchronization and every image simply calls it. It is shorter, it is clearer, and it is correct by construction. This is the whole argument for the collectives of §32.4: the patterns worth having a library for are the ones easiest to get subtly wrong by hand.

Phase 5 — Verdict and Sanity Check

• The delivered code had two faults: an unsynchronized, single-slot accumulate (a lost-update race and, formally, undefined behaviour) and a barrier placed where it could not help. The symptom — a run-to-run varying total, always below the true value — is the textbook signature.
• The single-image test could never have caught it. -fcoarray=single removes the concurrency, so it validates logic and hides races. Treat a passing one-image build as a check on what the program computes, never on whether it synchronizes.
• Prefer the collective. Fix A (per-image slots + barrier) is a fine hand-rolled reduction and worth understanding, but Fix B (co_sum) is the code to ship: it cannot race and it cannot be indexed wrong.
• Sanity check, always the same number. The acceptance test is image-count independence: $\sum_{1}^{12} i^2 = 650$ on 1, 2, 3, 4, 6, or 12 images. Any correct reduction returns it; the buggy one returned it only on 1 image, which is exactly how it fooled everyone.

Discussion Questions

1. The buggy program prints 650 on one image and varying wrong values on four. Which is more dangerous — a program that is always wrong, or one that is wrong only sometimes, only on some machines? Why do concurrency bugs like this survive so long in real codes?
2. Fix A uses part[*] with one slot per image and reads them on image 1. Could you instead have every image write into a rank-2 coarray indexed by image number and reduce with maxval/sum? What does the collective co_sum save you over any such hand-rolled scheme?
3. The sync all in the buggy code was not missing — it was misplaced. Explain, in terms of segments, why the position of a synchronization is as important as its presence, and give the general rule for where a barrier must sit relative to the writes and reads it protects.

Your Turn: Extensions

• Option A. Take the buggy program and fix it in place using a critical block around the acc[1] = acc[1] + part accumulate (plus a correctly placed sync all and an initialization of acc[1] on image 1). Confirm it now prints 650 on any image count. Then argue why this "fixed hand-roll" is still inferior to co_sum.
• Option B. Extend Fix B to compute the actual 2-norm $\sqrt{\sum x_i^2}$ of the vector $x_i = i,\ i = 1..12$, printing it as a real. (Answer: $\sqrt{650} \approx 25.4951$.) Which line changes, and does the reduction itself change at all?
• Option C. Instrument Fix A to also print, from image 1, each image's partial part[q] before summing, so you can see the decomposition ($14, 77, 194, 365$ on 4 images). Then run on 2 and 3 images and confirm the partials differ but the total does not — the acceptance test, made visible.

Key Takeaways

• A run-to-run varying result is a race. Almost always it is a coarray written by several images without synchronization, or read across an unordered segment boundary — undefined behaviour, not a compiler bug.
• -fcoarray=single validates logic, not synchronization. One image cannot race; a passing single-image test is necessary but never sufficient. Reason about segments, or run truly parallel, to catch races.
• Never let two images write the same location unguarded. Give each image its own slot (then gather), use critical/atomic for a genuine shared accumulator, or — best — call a collective.
• co_sum is the right reduction. It is shorter than any hand-rolled version, carries its own synchronization, and is correct by construction. The acceptance test for any reduction is image-count independence.