Chapter 25 Exercises: Scientific Data Formats

These problems build the skill this chapter is really about: turning a wall of numbers into a self-describing, portable, reproducible scientific record. Because NetCDF and HDF5 need their libraries installed and linked (and we never run code in this book), you will reason about file structure as much as you write code — predicting a ncdump/h5dump header, tracing a round trip, or diagnosing a call that compiled but produced an empty file. Where a problem is pure Fortran arithmetic (the back-of-envelope set), you can and should compile and run it.

Difficulty: ⭐ warm-up (minutes) · ⭐⭐ standard (requires real thought or code) · ⭐⭐⭐ challenge (design, or a multi-part investigation).

Solutions. Problems marked † and all odd-numbered problems have full solutions in appendices/answers-to-selected.md. Three carry worked, compilable code in code/exercise-solutions.f90 (25.12, 25.25, 25.29). Compile the library-linked ones with `nf-config --fflags --flibs` (NetCDF) or h5fc (HDF5), as shown in the chapter — but do not expect to run them without the libraries present. When a problem asks for output, hand-compute it; state your reasoning, not just the number.

Part A — Concepts and Warm-up (⭐)

25.1 † In one sentence each, give the four reasons §25.1 said text files do not scale for scientific data. For each, say whether raw Fortran unformatted I/O (Chapter 7, §7.5) fixes it, and explain the one it makes worse.

25.2 Define self-describing format in your own words without using the phrase "explains itself", then name the two specific pieces of information a raw stream file omits that make it unreadable on a different machine or after the writing code is lost.

25.3 † Match each term to its data model — write NetCDF, HDF5, or both beside each, and add a three-word reason: dimension, dataset, group, variable, attribute, coordinate variable, chunk.

25.4 A colleague says "NetCDF and HDF5 are competitors; pick one and never look back." Correct the statement in two sentences, using the specific fact from §25.2/§25.3 about how NetCDF-4 is stored. Then name one job where you would still deliberately choose plain HDF5 over NetCDF-4.

25.5 † For each scenario, choose NetCDF or HDF5 and justify in one line: (a) monthly sea-surface temperature grids you will publish for other climate scientists; (b) a particle-physics run that writes a tree of thousands of nested datasets you alone will analyze; (c) output you want ParaView to open with axes already in metres and no configuration; (d) a checkpoint file your own solver writes and reads back on the same cluster, once, to resume after a crash.

25.6 Why does HDF5 require you to call h5open_f and h5close_f, and to close every dataspace and dataset by hand, when native Fortran allocatable arrays need none of that? Answer in terms of what an HDF5 identifier is versus what an allocatable is, and name the Chapter 9 guarantee the library cannot give you.

Part B — Read a Header and Predict (⭐⭐)

25.7 † Here is a ncdump -h header. Answer the questions below it.

netcdf ocean {
dimensions:
        lon = 360 ;
        lat = 180 ;
        depth = 40 ;
variables:
        float salinity(depth, lat, lon) ;
                salinity:units = "psu" ;
                salinity:_FillValue = -1.e+30f ;
        double lon(lon) ;
                lon:units = "degrees_east" ;
// global attributes:
                :Conventions = "CF-1.8" ;
}

(a) How many values does salinity hold? (b) In what Fortran array shape and index order would you declare it to read it, and why is that not literally (depth, lat, lon)? (c) lon is listed as both a dimension and a variable — what is lon(lon) called and what does it store? (d) What does _FillValue mark, and why is it declared as a float (-1.e+30f) rather than a double?

25.8 Predict the exact ncdump -h header that example-02-netcdf-write.f90 produces — the dimensions, the variable line (with its axis order), the variable attribute, and the global attribute. Do it without looking at §25.2, then check. Which line demonstrates the row-major/column-major display convention?

25.9 † A program calls, in order and checking each status: nf90_create, nf90_def_dim (twice), nf90_def_var, nf90_put_var, nf90_enddef, nf90_close. It fails. Which call returns the error, what is the rule it violates, and what one-line reordering repairs it? Why would omitting the status check turn this loud failure into a silent one?

25.10 Given the h5dump -H header below, read off five facts: the dataset's element type, its byte order, its shape, its storage layout, and whether (and how) it is compressed. Then state one thing this header tells a reader that a raw .bin file of the same numbers could not.

DATASET "pressure" {
   DATATYPE  H5T_IEEE_F64LE
   DATASPACE  SIMPLE { ( 512, 512 ) / ( 512, 512 ) }
   STORAGE_LAYOUT { CHUNKED ( 64, 64 ) }
   FILTERS { COMPRESSION DEFLATE { LEVEL 4 } }
}

Part C — NetCDF: Write and Read (⭐⭐)

25.11 † Write a complete Fortran program that creates grid.nc holding one NF90_INT variable mask(nx, ny) with nx = 8, ny = 6, a variable attribute mask:long_name = "land-sea mask", and a global title. Include the check wrapper and show the compile command. Which call would you add, and where, to also record a global Conventions attribute?

25.12 † (code) Write a reader that opens heat.nc from example-02, discovers its grid size from the file (do not hard-code 4 and 3), reads temperature, and prints the grid shape and the sum. State the sum you expect and show the arithmetic. Why is it safe to state the sum exactly even though you cannot run the program? (Worked as solve_read_netcdf in code/exercise-solutions.f90.)

25.13 Modify the reader of 25.12 to also read and print the units attribute of temperature. Which nf90_* inquiry function retrieves an attribute's value, how does reading an attribute differ from reading a variable, and why must you blank the receiving character variable and trim it afterward?

25.14 nf90_put_var(ncid, varid, temperature) wrote the whole array in one call. Rewrite the write so it stores the array one row at a time in a loop, using the start= and count= optional arguments. Give the start and count for writing row i of an nx × ny variable, and name one situation where writing a slice at a time is genuinely necessary rather than merely possible.

25.15 † A program opens a file with nf90_open(..., NF90_NOWRITE, ncid) and then calls nf90_put_var. It fails at run time even though it compiled cleanly. Why? Give the change to the open mode that fixes it, say what that mode allows that NF90_NOWRITE does not, and explain why this is a run-time error and not a compile error.

Part D — HDF5: Groups, Chunking, Compression (⭐⭐/⭐⭐⭐)

25.16 List, in order, the HDF5 calls to create a file sim.h5, place a $100 \times 100$ double dataset named u inside a group /step0, write it, and close everything cleanly. You need not fill in every argument — name each routine, and for each h5*create_f/h5*open_f name the matching close and the handle it manages. How many handles are open at the peak?

25.17 † Explain why compression requires chunking in HDF5 — what unit does the deflate filter act on? Then: for a $1000 \times 1000$ field you will almost always read back whole, is a chunk shape of $(1000, 1)$ (one column per chunk) a good choice? Argue both from the whole-read case and from an occasional square region-of-interest read, and give a better shape.

25.18 A loop writes one HDF5 snapshot per timestep. It runs correctly for a while, then the library begins returning errors from h5screate_simple_f, though the dataset writes that do happen look correct. Diagnose the most likely cause and give the fix. Which chapter's automatic mechanism would have prevented this bug if HDF5 objects were native Fortran objects?

25.19 † ⭐⭐⭐ Write the property-list portion of an HDF5 writer that stores a $2000 \times 2000$ double field with $256 \times 256$ chunks and deflate level 5, and also enables the shuffle filter (h5pset_shuffle_f) before deflate. Explain in two sentences what shuffle does to the bytes and why it is applied before deflate to improve the ratio on floating-point data. How large is one chunk in bytes?

25.20 HDF5 stores string attributes differently from numeric ones (you must build a string datatype with h5tcopy_f/h5tset_size_f). Without writing the full code, describe the extra steps a string attribute needs versus the numeric h5acreate_f/h5awrite_f pair, and contrast this with how trivially NetCDF attaches nf90_put_att(ncid, varid, 'units', 'K'). Which format is friendlier for text metadata, and why does that matter for CF (§25.5)?

Part E — CF Conventions and Metadata (⭐⭐)

25.21 † The file below is valid NetCDF but not usefully reproducible. List every CF-convention improvement you would make (there are at least six), then write the nf90_put_att calls for three of them: a variable units, a global Conventions, and a global history.

netcdf run {
dimensions:
        x = 200 ;
        y = 200 ;
variables:
        double temp(y, x) ;
}

25.22 What is a coordinate variable, precisely (state the naming rule), and what two things does it let a visualization tool do that a bare dimension does not? Give the two nf90_def_var/nf90_put_att lines that add an x coordinate variable in metres to the file above, plus the nf90_put_var that fills it for spacing dx.

25.23 † A student adds temp:standard_name = "plate_temperature" to be "extra CF-compliant". Explain why this may be worse than adding nothing at all, what CF property standard_name must satisfy, and which attribute they should use instead for a quantity with no matching entry in the CF standard-name table.

25.24 The history global attribute is conventionally append-only. Explain what "append-only" means here, what it buys you for reproducibility (§25.5, and the tie to Chapter 37), and write an example two-line history value for a file that was created by a solver and later regridded by a post-processing tool.

Part F — Design It: Extend the Solver (⭐⭐⭐)

25.25 † (code) Extend the project's output to a time series in one file: instead of one .nc per snapshot, write a single series.nc with variable temperature(x, y, time) where time is an NF90_UNLIMITED dimension, appending one record per checkpoint with start=/count=. Sketch the writer, give the start/count for appending record k, and say why the unlimited dimension must be the slowest-varying (the last index in the Fortran dimension list). (Worked as solve_timeseries in code/exercise-solutions.f90.)

25.26 Add a time coordinate variable to the series of 25.25 with units = "seconds since 2000-01-01 00:00:00", and write the simulated time of each record. Why does CF encode time as "units since an epoch" rather than a plain number, and what does that let a tool do that a bare step index cannot?

25.27 † Upgrade write_field_netcdf (the Project Checkpoint) so the field variable is chunked and compressed: pass chunksizes= and deflate_level= to nf90_def_var (NetCDF-4 only, which is why the checkpoint uses NF90_NETCDF4). Choose a chunk shape for a $1000 \times 1000$ field and justify it against a whole-field read and a region-of-interest read. Flag honestly: which exact optional-argument spellings would you verify against your installed netcdf-fortran, and what standalone routines are the fallback?

25.28 ⭐⭐⭐ Design a heat_output module that offers both the text write_field (Chapter 7) and the NetCDF write_field_netcdf (this chapter) behind a single write_snapshot(field, basename, step, format) subroutine, where format is an enumeration-like integer or string selecting the backend. Sketch the public interface and the internal dispatch, and give one reason a production code keeps both backends rather than deleting the text one.

Part G — Back of the Envelope (⭐⭐)

25.29 † (code) A $1000 \times 1000$ real(dp) field is saved for 1,000 snapshots. Compute bytes per snapshot, total raw GB (decimal, $10^9$ bytes/GB), and the compressed size at 4×. Watch for integer overflow — which integer kind must the byte total use, and why does default integer fail here (give the approximate limit it blows through)? (Worked as solve_storage in code/exercise-solutions.f90.)

25.30 A run outputs a $2048 \times 2048$ real(dp) field every step for 5,000 steps, saving every 10th. (a) How many snapshots? (b) Bytes per snapshot? (c) Total raw GB? (d) At a deflate ratio of 3×, GB on disk? Show each step, and state which of these numbers you would put in a proposal to justify buying more storage.

25.31 Writing a 24 MB text snapshot takes, illustratively, ~0.8 s (dominated by decimal conversion); the same field as 8 MB unformatted binary takes ~0.02 s. Over the 500-snapshot schedule of 25.30, estimate the wall-clock time each output path costs, and state the one-line lesson. Label your rates: are they Tier-1 measured, or Tier-2 illustrative?

Part H — Port It, Find the Bug, and Interleave (⭐⭐/⭐⭐⭐)

25.32 † (Port it.) A collaborator hands you this Python h5py writer. Translate it to a Fortran HDF5 writer (h5*_f), matching the dataset name, shape, type, and compression. Where does the Fortran version need calls the Python has hidden from you (library init, dataspace, property list, handle closing)?

import h5py, numpy as np
u = np.zeros((100, 100), dtype='float64')
with h5py.File('field.h5', 'w') as f:
    f.create_dataset('temperature', data=u, chunks=(10, 10), compression='gzip', compression_opts=5)

25.33 (Find the bug.) This HDF5 writer compiles and runs but produces a file whose temperature dataset is uncompressed, despite the property-list calls. What is missing?

call h5pcreate_f(H5P_DATASET_CREATE_F, dcpl, hdferr)
call h5pset_chunk_f(dcpl, 2, chunk_dims, hdferr)
call h5pset_deflate_f(dcpl, 6, hdferr)
call h5dcreate_f(file_id, 'temperature', H5T_NATIVE_DOUBLE, space_id, dset_id, hdferr)  ! (!)
call h5dwrite_f(dset_id, H5T_NATIVE_DOUBLE, u, dims, hdferr)

25.34 † (Modernize / wrap legacy — Chapters 7 + 25.) A legacy routine writes the field as raw unformatted bytes: open(newunit=u, file=f, form='unformatted'); write(u) field; close(u). It is fast and exact but non-portable and mute. Without changing the solver, describe how you would wrap or replace this output with a NetCDF writer that preserves the same data, and list the three pieces of metadata the raw version discards that your NetCDF version should record.

25.35 (Interleave — Chapters 7 + 9 + 25.) The solver reads its parameters from a namelist (Chapter 7) into a field_t (Chapter 9), then writes the field to NetCDF (this chapter). Sketch the data flow as four steps, naming the exact mechanism at each: config file → parameters → field object → output file. Which single one of these four artifacts is self-describing, and which relies on you remembering what the numbers mean?

25.36 (Interleave — Chapter 7 vs 25.) Both Chapter 7's write_field and this chapter's write_field_netcdf have the "one bundled argument + filename" shape. Write the two if branches of a driver that picks the writer from a logical :: use_netcdf flag, and give one concrete reason a production run would keep both writers available rather than deleting the text one.