Case Study 1: The Offload That Made It Slower

"We put it on the GPU and it got slower. How is that even possible?"

Executive Summary

A colleague proudly reports that they have GPU-accelerated the heat solver with OpenACC — and that the "accelerated" version runs slower than the plain CPU code it replaced. This is not a rare mishap; it is the single most common way a first GPU port fails, and it fails for a reason you can diagnose with arithmetic alone, before touching a profiler. You will read their code, count how many times the temperature field crosses the host–device bus, convert that transfer volume into wall-clock time, and show numerically that the transfers — not the computation — are what sank the port. Then you will apply the one structural fix from §35.4 — a !$acc data region around the time loop — and show the same kernel, unchanged, turn a 10% slowdown into a 10× speedup. Nothing here requires a GPU: it is the transfer arithmetic that tells you, in five minutes, whether an offload is healthy or doomed.

Skills applied: the host/device model and separate memory (§35.1); OpenACC data clauses and the data region (§35.2); host–device transfer as the bottleneck and the per-step-transfer pitfall (§35.4); arithmetic intensity and "move once, compute many" (§35.4, ⚡ Performance Note); Amdahl's Law with transfer counted as serial (§35.5, and Chapter 31).

Background

The solver runs a $2000 \times 2000$ plate (real(dp), so $4 \times 10^6$ cells $\times\,8$ bytes $= 32$ MB) for $5000$ time steps. The machine's host–device bus delivers about $16$ GB/s ($1$ GB $= 10^9$ bytes). From the CPU baseline, one stencil sweep of the whole plate takes about $4$ ms; a measurement of the GPU kernel alone (kernel time only, no transfer) clocks a single sweep at about $0.4$ ms — ten times faster, exactly as hoped. So why is the whole program slower? Here is the offloaded time loop your colleague wrote (round numbers throughout are illustrative — Tier 3 — chosen so the arithmetic is clean):

! Your colleague's "accelerated" solver — the offload that made it slower.
do step = 1, nsteps
  !$acc parallel loop collapse(2) copy(u) create(u_new)   ! <-- data clause on the STEP kernel
  do j = 2, ny-1
    do i = 2, nx-1
      u_new(i,j) = u(i,j) + r*(u(i-1,j)+u(i+1,j)+u(i,j-1)+u(i,j+1) - 4.0_dp*u(i,j))
    end do
  end do
  u(2:nx-1,2:ny-1) = u_new(2:nx-1,2:ny-1)                 ! commit (on the host!)
end do

The kernel is correct — it computes the right physics — and it is genuinely fast on the device. The disease is entirely in where the data clause sits: copy(u) is on the per-step kernel, so the whole 32 MB field is shipped to the device and back every step. Worse, the commit u(...) = u_new(...) runs on the host, which means u_new (a create array) would have to come back too — but even ignoring that, the per-step copy(u) alone is fatal. Let us prove it with numbers.

Phase 1 — Count the Transfers

The field is 32 MB. A one-way copy over a 16 GB/s bus takes $32\times10^6 / 16\times10^9 = 2.0 \times 10^{-3}$ s $= 2.0$ ms; a round trip (in and out) is $4.0$ ms. The copy(u) clause does exactly that round trip on every one of the 5000 steps. So the transfer bill for the run is $5000 \times 4.0\ \text{ms} = 20{,}000$ ms $= 20$ s — and that is before a single flop of useful computation. The GPU compute, at $0.4$ ms per sweep, adds only $5000 \times 0.4 = 2000$ ms $= 2$ s. The transfers are ten times the computation. Your colleague built a Ferrari and drives it through a car wash between every block.

Phase 2 — Put It Beside the CPU

The comparison that matters is against the code they replaced. Here is a small audit program — plain Fortran, no GPU needed — that tallies the three run times: the CPU baseline, the broken per-step-transfer GPU version, and the fixed resident version we are about to write.

program transfer_audit
  use, intrinsic :: iso_fortran_env, only: dp => real64
  implicit none
  real(dp) :: bytes, bw, rt_ms, gpu_sweep, cpu_sweep
  real(dp) :: broken_ms, cpu_ms, fixed_ms
  integer  :: nsteps

  bytes     = 4.0e6_dp * 8.0_dp          ! 2000x2000 real(dp) = 32 MB
  bw        = 16.0e9_dp                   ! 16 GB/s, bytes/s
  rt_ms     = 2.0_dp * bytes / bw * 1.0e3_dp   ! round-trip transfer, ms
  nsteps    = 5000
  gpu_sweep = 0.4_dp                      ! GPU kernel time per sweep (ms)
  cpu_sweep = 4.0_dp                      ! CPU time per sweep (ms)

  broken_ms = real(nsteps, dp)*(rt_ms + gpu_sweep)   ! transfer EVERY step
  cpu_ms    = real(nsteps, dp)*cpu_sweep
  fixed_ms  = rt_ms + real(nsteps, dp)*gpu_sweep     ! transfer ONCE (residency)

  print '(a, f6.1, a)',            'round-trip transfer   = ', rt_ms, ' ms'
  print '(a, f9.1, a, f6.2, a)',   'CPU  (5000 sweeps)    = ', cpu_ms,    ' ms = ', cpu_ms/1000.0_dp,    ' s'
  print '(a, f9.1, a, f6.2, a)',   'GPU  broken per-step  = ', broken_ms, ' ms = ', broken_ms/1000.0_dp, ' s'
  print '(a, f9.1, a, f6.2, a)',   'GPU  fixed residency  = ', fixed_ms,  ' ms = ', fixed_ms/1000.0_dp,  ' s'
  print '(a, f6.2, a)',            'broken vs CPU speedup = ', cpu_ms/broken_ms, 'x  (a SLOWDOWN)'
  print '(a, f6.2, a)',            'fixed  vs CPU speedup = ', cpu_ms/fixed_ms,  'x'
end program transfer_audit

$ gfortran -std=f2018 -Wall transfer_audit.f90 -o audit && ./audit
round-trip transfer   =    4.0 ms
CPU  (5000 sweeps)    =   20000.0 ms =  20.00 s
GPU  broken per-step  =   22000.0 ms =  22.00 s
GPU  fixed residency  =    2004.0 ms =   2.00 s
broken vs CPU speedup =   0.91x  (a SLOWDOWN)
fixed  vs CPU speedup =   9.98x

There is the whole story in six lines. The broken GPU version takes 22 s against the CPU's 20 s — a genuine slowdown of 0.91×, exactly what your colleague observed. The transfers ($20$ s) dwarf the fast kernel ($2$ s) and drag the total below the CPU baseline.

Phase 3 — The Fix: One Data Region

The cure is structural and touches not one line of the kernel. Hoist the data movement out of the loop into a !$acc data region that keeps the field resident on the device for the whole run, and mark the per-step kernels present. Do the commit on the device too, so nothing bounces to the host mid-run:

! Fixed: the field is resident on the device for the whole run.
!$acc data copy(u) create(u_new)
do step = 1, nsteps
  !$acc parallel loop collapse(2) present(u, u_new)     ! stencil, on resident data
  do j = 2, ny-1
    do i = 2, nx-1
      u_new(i,j) = u(i,j) + r*(u(i-1,j)+u(i+1,j)+u(i,j-1)+u(i,j+1) - 4.0_dp*u(i,j))
    end do
  end do
  !$acc parallel loop collapse(2) present(u, u_new)     ! commit, also on the device
  do j = 2, ny-1
    do i = 2, nx-1
      u(i,j) = u_new(i,j)
    end do
  end do
end do
!$acc end data                                          ! field copied back ONCE, here

The transfer count drops from $2 \times 5000 = 10{,}000$ one-way trips to exactly $2$: one in at !$acc data, one out at !$acc end data`. The audit's `fixed_ms` row is the result: $4$ ms of transfer plus $2000$ ms of compute is about $2$ s, a **10× speedup** over the CPU — from the very same kernel that was a slowdown a moment ago. Confirm it withnvfortran -acc -Minfo=accel: the feedback should now showuandu_new` copied only at the data-region boundary, and nothing transferred inside the step loop.

Phase 4 — Why the Numbers Fall This Way

The deep reason is arithmetic intensity (§35.4). The stencil does about 6 flops on each cell but the cell is only 8 bytes; even counting just one read-miss and one write, that is roughly $6/16 \approx 0.4$ flops per byte — a memory-bound kernel that does almost no arithmetic per byte it touches. A kernel like that can never outrun its data movement on a single pass; the only way it wins is to reuse the data on the device across many passes, so the one-time transfer is amortized. That is precisely what the data region buys: the 32 MB crosses the bridge once and is then swept 5000 times for free (no further transfer). The broken version amortized nothing — it re-paid the full transfer on every sweep — so the low intensity was fatal. Residency converts a transfer-bound kernel into a compute-bound run by giving it reuse.

Phase 5 — The General Lesson, as a Checklist

Before you accept any offload as "accelerated," run this three-line audit in your head:

  1. How large is the data, and how fast is the bus? Bytes ÷ bandwidth is your one-way transfer time.
  2. How many times does the data cross the bridge? Once (residency) or every step (the trap)? Multiply by the transfer time.
  3. Is the transfer bill larger than the compute bill? If yes — as it almost always is for a memory-bound kernel with per-step transfers — the offload is transfer-bound, and no faster kernel will save it. Fix the data movement first.

Your colleague's kernel was never the problem. The kernel was 10× faster the whole time. The commute killed it, and the commute is a property of where your data clauses sit, not how clever your arithmetic is.

Discussion Questions

  1. The broken version's kernel is genuinely 10× faster than the CPU, yet the program is slower. Explain to a skeptic how both statements are true at once, using the transfer and compute bills.
  2. Suppose the stencil did far more arithmetic per cell (say a 50-flop chemistry update instead of a 6-flop diffusion step). Would the per-step-transfer version still be a slowdown? At what arithmetic intensity does the transfer stop dominating?
  3. The fix moved the commit loop onto the device. What would happen to the transfer count if the commit had been left as a host array assignment u(...) = u_new(...) inside the region?

Your Turn: Extensions

  • Option A. Change nsteps in the audit program to 100 and rerun the arithmetic by hand. At only 100 steps, is the fixed version still worth it over the CPU? What does that say about small runs on a GPU (§35.5)?
  • Option B. Add a row to the audit for a version that writes a VTK frame every 500 steps with !$acc update self(u) — 10 extra one-way transfers of 32 MB over the whole run. How much does periodic output cost, and does it change the verdict?
  • Option C. The bus in this study was 16 GB/s (an older PCIe generation). Rerun the numbers for a 64 GB/s interconnect. Does a faster bus fix the broken per-step version, or does residency still win — and by how much? What does that tell you about relying on hardware to rescue a bad data structure?

Key Takeaways

  • A correct, fast kernel is not a fast program: host–device transfer can dominate and turn a 10× kernel into a 10% slowdown. Audit the transfers before you celebrate.
  • The per-step data clause (copy on the kernel inside the time loop) is the classic killer; the structural fix is a !$acc data region around the whole loop, so the field is resident and crosses the bridge once, not every step.
  • Arithmetic intensity predicts the outcome: a memory-bound kernel (low flops per byte) only wins on the GPU if it reuses its data on the device across many passes — which residency provides and per-step transfers destroy.
  • The five-minute transfer audit — bytes ÷ bandwidth × crossings, compared to the compute bill — tells you whether an offload is healthy without a profiler and without a GPU.