Self-Assessment Quiz: Testing, Documentation, and Software Engineering

Twenty questions to confirm you can test numerical code against the right oracles, choose between bit-for-bit and tolerance comparisons, wire up CI, document with FORD, and reason about reproducibility. Aim for 16 or more. Answers and a topic map are at the end — try the whole quiz first.

Question 1

A unit test is best described as: - A. A test that runs the entire program on realistic input - B. A small, automated check of one procedure (or a few) in isolation against a known answer - C. A manual comparison of two output files - D. A benchmark of how fast a routine runs

Question 2

Testing numerical code is harder than testing a sort routine mainly because: - A. Fortran has no test frameworks - B. Finding the oracle — the known-correct answer — is hard when the code exists precisely because the answer is hard to compute - C. Floating-point code cannot be tested at all - D. Numerical code has no bugs

Question 3

A floating-point assertion should compare: - A. got == want - B. got /= want - C. abs(got - want) <= tol for a chosen tolerance - D. the two values' string representations

Question 4

Which is not one of the four oracle sources for numerical tests in §37.1? - A. Exact solutions on special inputs (e.g. a quadratic) - B. Invariants and conservation laws (e.g. the maximum principle) - C. The wall-clock runtime of the routine - D. Convergence order under grid refinement

Question 5

laplacian(x^2 + y^2) is a good unit-test input because the true Laplacian is: - A. Different at every point, so it stresses the code - B. Exactly $4$ everywhere, and the five-point stencil is exact for quadratics - C. Impossible to compute by hand - D. Zero everywhere

Question 6

A regression test: - A. Proves the code is mathematically correct - B. Pins output against a known-good result and catches anything that changes what should not - C. Measures performance regressions only - D. Is the same thing as a unit test

Question 7

Comparing two results bit-for-bit means: - A. They agree to a stated tolerance - B. They are identical to the last bit (a == b is exactly true) - C. They have the same order of magnitude - D. They round to the same integer

Question 8

Which change can make a bit-for-bit regression test fail on correct, unchanged source? - A. Renaming a local variable - B. Adding a comment - C. Building with -ffast-math, which lets the compiler reassociate arithmetic - D. Running the program twice

Question 9

The safest default comparison for a portable numerical regression test is: - A. Bit-for-bit equality - B. A physically meaningful tolerance - C. Comparing only the first digit - D. No comparison; just check the program ran

Question 10

Verification (as distinct from validation) means confirming that: - A. The equations describe physical reality - B. The code correctly solves the equations it claims to solve (e.g. vs an analytical solution) - C. The code compiles without warnings - D. The output looks plausible

Question 11

The linear temperature field is an ideal analytical test case for this solver because: - A. It is hard to compute - B. The five-point stencil is exact for linear fields, so it is a discrete fixed point of step with zero error - C. It never reaches steady state - D. It violates the boundary conditions

Question 12

Continuous integration is the practice of: - A. Compiling the code once before release - B. Automatically building and testing the project on every change, on a clean machine - C. Integrating the equations of motion continuously in time - D. Running the code on a schedule only

Question 13

In the GitHub Actions workflow, matrix: gcc: [11, 12, 13] causes: - A. The code to be compiled with three optimization levels - B. The build-and-test job to run three times in parallel, once per gfortran version - C. Three copies of the output to be saved - D. The tests to run three times on one compiler

Question 14

A test suite communicates failure to fpm test and CI through: - A. A printed "FAIL" message - B. A nonzero process exit code (e.g. from error stop 1) - C. A log file - D. An email

Question 15

FORD is: - A. A Fortran compiler - B. A build system - C. A documentation generator that turns !>/!! doc comments into a browsable site - D. A unit-test framework

Question 16

FORD-annotated source still compiles with gfortran because: - A. FORD rewrites the source first - B. !> and !! are ordinary Fortran comments, which the compiler ignores - C. gfortran has built-in FORD support - D. The annotations are removed by the linker

Question 17

For a reproducible scientific run you should record all of the following EXCEPT: - A. The compiler and its version - B. The compiler flags - C. The random seed (for a stochastic run) - D. The wall-clock time the run happened to take

Question 18

A fixed random seed guarantees reproducibility on the same compiler but not across compilers because: - A. Seeds are random - B. The Fortran standard does not specify which generator random_number uses - C. Different compilers use different seed sizes only - D. random_seed is not portable

Question 19

"It works on my machine," said of a scientific result, is best understood as: - A. A guarantee of correctness - B. A confession that the result depends on undocumented features of one environment, so it is not yet reproducible - C. A compiler error - D. A normal and acceptable state of affairs

Question 20

(What does this code print?) Assume the harness of §37.1.

call assert_close('a', 0.1_dp + 0.2_dp, 0.3_dp, 1.0e-12_dp)
call assert_close('b', 0.1_dp + 0.2_dp, 0.3_dp, 0.0_dp)
  • A. PASS a then PASS b
  • B. PASS a then FAIL b
  • C. FAIL a then FAIL b
  • D. PASS a then PASS a

Answer Key

Q Ans Why
1 B A unit test checks one unit in isolation against a known answer, automatically.
2 B The oracle problem: the "right answer" is what the code approximates, so it is hard to know in advance.
3 C Floating-point rounds; compare within a tolerance, never with ==.
4 C Runtime is not a correctness oracle; the four are exact cases, invariants, symmetry, convergence order.
5 B The stencil is exact for quadratics, so $\nabla^2(x^2+y^2)=4$ everywhere is an exact, hand-checkable oracle.
6 B A regression test pins a known-good result and catches unintended change.
7 B Bit-for-bit means identical to the last bit — exact equality of the IEEE patterns.
8 C -ffast-math reassociates non-associative FP arithmetic, changing the last bits.
9 B A tolerance is robust across compilers/flags/parallelism; bit-for-bit is fragile.
10 B Verification = solving the equations right; validation = right equations.
11 B Exact for linear fields, so it is a zero-error discrete fixed point — an oracle with no tolerance ambiguity.
12 B CI builds and tests automatically on every change, on a clean machine.
13 B The matrix runs the job once per listed compiler version, in parallel.
14 B Automation reads the exit code; error stop 1 makes failure visible.
15 C FORD generates documentation from !>/!! doc comments in the source.
16 B Doc comments are ordinary comments; the compiler ignores them, so the source still builds.
17 D The time it happened to run is irrelevant to regenerating the result.
18 B The standard leaves the generator unspecified, so the same seed can draw differently per compiler.
19 B It signals dependence on an undocumented environment — not yet reproducible, not yet a result.
20 B 0.1 + 0.2 differs from 0.3 by ~$10^{-17}$: within 1e-12 (PASS a) but not within 0.0 (FAIL b).

Topics to review by question

  • Q1–5 → §37.1 (unit tests, the oracle problem, the four oracle sources).
  • Q6–9 → §37.2 (regression tests, bit-for-bit vs tolerance).
  • Q10–11 → §37.2 (verification against analytical solutions).
  • Q12–14 → §37.3 (continuous integration, the compiler matrix, exit codes).
  • Q15–16 → §37.4 (FORD documentation).
  • Q17–19 → §37.5 (reproducibility, seeds, "works on my machine").
  • Q20 → §37.1–§37.2 (why a tolerance, not ==) and Chapter 20.

Scored below 16? Reread the flagged sections. Testing and reproducibility are what turn the Chapter 36 package into the Chapter 38 capstone's trustworthy artifact — a result a reviewer can believe.