Case Study 1: Refereeing a Result — Reading a Heat-Solver Manuscript Like a Reviewer

"The first principle is that you must not fool yourself — and you are the easiest person to fool." — Richard P. Feynman

Executive Summary

A short manuscript lands in your inbox with a request to referee it: "A Fast Parallel Solver for the 2D Diffusion Equation." It reports a Fortran heat solver, some heat-map figures, and a headline claim — "$20\times$ faster than the baseline." Your job is not to rewrite it; it is to decide, as a reviewer, whether to believe it, and to say precisely what would make it trustworthy. This case study walks that review. You will read the claims, discover that the crucial verification is missing and reconstruct it yourself (recovering second-order accuracy against the analytical solution), diagnose a reported instability from its symptoms alone, audit the performance claim for its missing baseline, and write the referee report. Where Case Study 2 builds a presentation, this one judges one — the skill you apply to every paper you read for the rest of your career, including your own.

(The manuscript here is a constructed teaching example — a composite of the mistakes real first drafts make, not a real paper.)

Skills applied: distinguishing verification from validation (§38.3); reconstructing a convergence study against an analytical solution (§38.3, and Chapter 22); diagnosing CFL instability from symptoms (Chapter 24); auditing a performance claim with Amdahl and a baseline (§38.4, Chapter 31); the structure of what reviewers check (§38.6).

Background

The manuscript's method section is competent: the 2D heat equation, a five-point stencil, an explicit FTCS step, OpenMP on the update loop. The results are four heat maps of a plate warming, and a table showing wall times on 1 and 16 cores. The abstract closes: "Our solver is $20\times$ faster than the baseline and produces physically correct diffusion." Read that last clause carefully. "Physically correct" is asserted, not shown — and asserting correctness is exactly what a reviewer must not accept on faith. We take the paper apart along the reviewer's checklist from §38.6.

Phase 1 — Read the Claims and Separate Them

A manuscript makes several claims; a reviewer sorts them by what evidence each would need. This one makes three:

Immediately, the most important gap is visible: the correctness claim is supported by heat maps, and a heat map proves nothing — a buggy solver produces plausible-looking heat maps too. The manuscript needs a convergence study and does not have one. That is the review's central finding, and everything else is secondary.

The reviewer's instinct. When a computational paper claims correctness, your eyes go straight to the V&V. No convergence study, no comparison to a known solution, no order-of-accuracy number? Then the correctness claim is unsupported, regardless of how good the figures look. This is the first thing to check and the most common thing to be missing.

Phase 2 — Reconstruct the Missing Verification

Rather than just demand a convergence study, we do what a generous reviewer does: check whether the method could pass one. The manuscript's stencil is the standard five-point Laplacian, so its discrete eigenvalue for the fundamental mode should approach the continuous $-2\pi^2$ at second order. We reconstruct that check — the study the authors should have included — on two grids:

! cs01-verify.f90 -- reconstruct the convergence study the manuscript omitted.
! The five-point stencil's discrete eigenvalue for the sin(pi x)sin(pi y) mode is
! lambda_h = -(8/h^2) sin^2(pi h/2); it should approach -2 pi^2 at order 2.
! Compile: gfortran -std=f2018 -Wall cs01-verify.f90 -o verify && ./verify
program cs01_verify
  use, intrinsic :: iso_fortran_env, only: dp => real64
  implicit none
  real(dp), parameter :: pi = 3.141592653589793_dp
  real(dp), parameter :: exact = -2.0_dp*pi**2
  real(dp) :: h, e1, e2
  h  = 0.125_dp                                        ! 1/8
  e1 = abs(-8.0_dp*sin(pi*h/2)**2/h**2 - exact)
  h  = 0.0625_dp                                       ! 1/16
  e2 = abs(-8.0_dp*sin(pi*h/2)**2/h**2 - exact)
  print '(a,f10.6)', 'error at h=1/8  : ', e1
  print '(a,f10.6)', 'error at h=1/16 : ', e2
  print '(a,f7.3)',  'ratio           : ', e1/e2
  print '(a,f7.3)',  'observed order  : ', log(e1/e2)/log(2.0_dp)
end program cs01_verify

$ gfortran -std=f2018 -Wall cs01-verify.f90 -o verify && ./verify
error at h=1/8  :   0.252369
error at h=1/16 :   0.063336
ratio           :   3.985
observed order  :   1.994

The observed order is $1.994 \approx 2$: the discretization is sound. So the manuscript's method is correct even though the manuscript did not demonstrate it. The referee report will say exactly that — "add this study; your method passes it, but the reader cannot know that from what you wrote." Verification is not a formality the authors happened to skip; it is the evidence that turns "trust me" into "here is why."

Sanity check. The ratio approaches 4 (order 2), matching the five-point stencil's theoretical $O(h^2)$ truncation error. Had we measured a ratio near 2 (order 1), that would have been the story — a real bug in the manuscript's stencil — and the review would read very differently.

Phase 3 — Diagnose the Reported Instability

Buried in the manuscript's discussion is a telling sentence: "At the highest resolution, some runs produced NaN values and were discarded." A reviewer does not let that pass — discarded runs are a symptom, and the symptom has a diagnosis. The authors refined the grid but (the methods section reveals) used a fixed timestep across all resolutions. That is the classic CFL blunder from Chapter 24.

Work the numbers. With $\alpha$ and $\Delta t$ fixed, the diffusion number is $r = \alpha\Delta t/h^2$. Halving $h$ quadruples $r$. If their coarse grid ran at a comfortable $r = 0.1$, then:

The highest-resolution runs crossed the CFL cliff and detonated into NaN; the authors discarded them without recognizing why. The fix is one line and belongs in the methods: derive the timestep from the grid, $\Delta t = 0.9\,h^2/(4\alpha)$, so refinement automatically stays stable. The referee report flags this not as a fatal flaw but as a correctness-and-honesty issue: discarding runs that "misbehave" without diagnosing them is exactly how a hidden bug survives into a published result.

Phase 4 — Audit the "20× Faster" Claim

Now the headline. "$20\times$ faster than the baseline" — a reviewer asks the two questions of §38.4 immediately: faster than what baseline, and built how? The manuscript never says. That single omission makes the number unfalsifiable, and a good reviewer is allergic to it.

We can bound what is plausible. The manuscript reports OpenMP on 16 cores. By Amdahl's Law, even a perfectly parallel-looking kernel with a $98\%$ parallel fraction is capped at $S(16) = 1/(0.02 + 0.98/16) \approx 12.3\times$ on 16 cores — *below* the claimed $20\times$. So a $20\times$ speedup on 16 cores from parallelism alone is not possible under Amdahl; it can only be real if the "baseline" was a slow, unoptimized serial build (-O0, say) and the fast version added compiler optimization on top of threading. That is a legitimate speedup to report — but it is a speedup over a bad baseline, and honesty demands saying so. A $20\times$ number measured against -O0 is a very different claim from $20\times$ against a tuned serial build, and the reader must be told which.

Phase 5 — Write the Referee Report

The review writes itself once the analysis is done. A good report is specific, actionable, and separates the fatal from the fixable:

Recommendation: major revision. The method is sound (the stencil converges at second order — see the convergence study I reconstructed), but the manuscript does not demonstrate its correctness and overstates its performance.

1. Add a verification study (required). Compare to the analytical solution $\sin(\pi x)\sin(\pi y)e^{-2\alpha\pi^2 t}$ and report the observed order of accuracy. Your method passes this; show it. Correctness is not assertable from heat maps.
2. Fix and disclose the instability (required). The discarded high-resolution NaN runs are a CFL violation from a fixed timestep under grid refinement. Derive $\Delta t$ from $h$ and report it; do not silently discard runs.
3. Qualify the speedup (required). State the baseline (tuned serial or -O0?) and the full build configuration and problem size. As written, $20\times$ on 16 cores exceeds the Amdahl ceiling and is only plausible against an unoptimized baseline.
4. Scope the claim (minor). "Physically correct" overstates it: you have (or will have) verification, not validation — you have not compared to experiment. Say "verified," not "physically correct."

Four items, three of them about trust rather than cleverness — exactly the ratio §38.6 warned of. The method is fine; the argument was missing, and building that argument is the revision.

Discussion Questions

1. The method turned out to be correct, yet the paper was rejected-pending-revision anyway. Why is a correct method with no verification study still not publishable — what, precisely, is the reader unable to do?
2. The authors discarded the NaN runs. Describe a plausible worse outcome in which that habit hides a genuine bug (not a CFL violation) and a wrong result gets published. How does a convergence study guard against it?
3. A $20\times$ speedup against an -O0 baseline is "real" in the sense that the numbers are true. Why is reporting it without the baseline nonetheless a form of fooling the reader — and yourself?

Your Turn: Extensions

• Option A. Extend cs01-verify.f90 to a third grid ($h = 1/32$) and confirm the order stays near 2. Then break the stencil (drop the $1/h^2$) and watch the "order" collapse — see a convergence study catch a bug.
• Option B. Write the two-sentence methods addition that fixes the CFL bug: the stable_dt formula and the sentence disclosing the safety factor. This is the difference between a reproducible method and a fragile one.
• Option C. Rewrite the manuscript's abstract honestly: replace "physically correct" and "$20\times$ faster than the baseline" with claims the (revised) evidence supports. Keep it to five sentences. Notice how much stronger the honest abstract reads.

Key Takeaways

• A reviewer reads a computational paper for trust first: is there a convergence study, is the comparison to a genuine reference, does the order of accuracy come out right? Heat maps prove nothing.
• Reconstruct the missing check when you can — a method may be correct even when the paper fails to show it, and the most useful review says exactly what evidence to add.
• Symptoms have diagnoses. Discarded NaN runs at high resolution are a CFL violation from a fixed timestep; the fix (derive $\Delta t$ from $h$) is one line and belongs in the methods.
• A speedup with no baseline is unfalsifiable. Amdahl bounds what is plausible; a number above the ceiling reveals an unoptimized baseline the authors did not disclose.
• Verification is not validation. "Verified" (matches the analytical solution) is honest; "physically correct" (matches reality) requires experiment the paper does not have. Scope every claim to its evidence.