Chapter 25 — Key Takeaways (Scientific Data Formats)

The one-page reference for writing and reading NetCDF and HDF5 from Fortran, and labelling data so it outlives the code that wrote it.

The core idea

Self-describing = the file stores its own shape, type, byte order, and metadata, so any program on any machine can interpret it. Raw text and raw binary are not: text is bulky, slow, lossy, and mute; raw binary fixes the first three but is non-portable (endianness, record markers) and still mute. NetCDF and HDF5 give you binary's speed and size plus portability, metadata, and transparent compression.

The two formats at a glance

NetCDF HDF5
Model dimensions + variables + attributes (flat) groups + datasets + attributes (a tree)
Fortran module use netcdf use hdf5
Call style functions returning a status subroutines with a trailing hdferr
Library init none h5open_f / h5close_f (required)
Hierarchy flat (NetCDF-4 adds groups) native groups
Compression NetCDF-4 opts on nf90_def_var property list (h5pset_chunk_f+h5pset_deflate_f)
Compile helper `nf-config --fflags --flibs` h5fc
Home turf climate/weather/ocean, shareable gridded data large simulation output, complex hierarchy
Relationship NetCDF-4 is HDF5 underneath the storage layer beneath NetCDF-4

NetCDF write — the canonical sequence

use netcdf
call check( nf90_create('f.nc', NF90_CLOBBER, ncid) )          ! or NF90_NETCDF4
call check( nf90_def_dim(ncid, 'x', nx, x_dimid) )             ! define mode
call check( nf90_def_var(ncid, 'temp', NF90_DOUBLE, [x_dimid, y_dimid], varid) )
call check( nf90_put_att(ncid, varid, 'units', 'K') )          ! NF90_GLOBAL for file-level
call check( nf90_enddef(ncid) )                                ! --> data mode
call check( nf90_put_var(ncid, varid, temperature) )
call check( nf90_close(ncid) )

Read: nf90_open(..., NF90_NOWRITE, ncid) → nf90_inq_dimid/nf90_inquire_dimension (learn shape) → nf90_inq_varid → nf90_get_var → nf90_close. nf90_get_att reads an attribute.

HDF5 write — chunked and compressed

use hdf5
call h5open_f(hdferr)
call h5fcreate_f('f.h5', H5F_ACC_TRUNC_F, file_id, hdferr)
call h5screate_simple_f(2, dims, space_id, hdferr)             ! dims: integer(hsize_t)
call h5pcreate_f(H5P_DATASET_CREATE_F, dcpl, hdferr)
call h5pset_chunk_f(dcpl, 2, chunk_dims, hdferr)
call h5pset_deflate_f(dcpl, 5, hdferr)                         ! gzip level 1..9
call h5dcreate_f(file_id, 'temp', H5T_NATIVE_DOUBLE, space_id, dset_id, hdferr, dcpl)
call h5dwrite_f(dset_id, H5T_NATIVE_DOUBLE, u, dims, hdferr)
call h5pclose_f(dcpl,hdferr); call h5dclose_f(dset_id,hdferr)  ! close EVERY handle
call h5sclose_f(space_id,hdferr); call h5fclose_f(file_id,hdferr)
call h5close_f(hdferr)

Read: h5fopen_f(..., H5F_ACC_RDONLY_F, ...) → h5dopen_f → (h5dget_space_f + h5sget_simple_extent_dims_f to size the buffer) → h5dread_f → close each handle → h5close_f. Groups: h5gcreate_f/h5gopen_f/h5gclose_f; the path /fields/temp addresses a dataset in a group.

The calls introduced

NetCDF (nf90_*) HDF5 (h5*_f) HDF5 types/constants
nf90_create, nf90_open, nf90_close h5open_f, h5close_f hid_t, hsize_t (kinds)
nf90_def_dim, nf90_def_var h5fcreate_f, h5fopen_f, h5fclose_f H5F_ACC_TRUNC_F, H5F_ACC_RDONLY_F
nf90_put_att, nf90_get_att h5screate_simple_f, h5sclose_f H5T_NATIVE_DOUBLE, H5T_IEEE_F64LE
nf90_enddef h5dcreate_f, h5dopen_f, h5dclose_f H5P_DATASET_CREATE_F
nf90_put_var, nf90_get_var h5dwrite_f, h5dread_f H5S_UNLIMITED_F (extendible)
nf90_inq_varid, nf90_inq_dimid h5pcreate_f, h5pset_chunk_f, h5pset_deflate_f, h5pset_shuffle_f
nf90_inquire_dimension, nf90_strerror h5gcreate_f, h5gopen_f, h5gclose_f, h5gn_members_f

Which format — decision aid

If you need… Reach for
To publish gridded data other scientists will use NetCDF + CF conventions
ParaView/xarray/cdo to read it with no configuration NetCDF (CF)
A hierarchy of groups, nested cases, mesh+solution HDF5
Maximum control over chunking/compression HDF5
Both metadata conventions and HDF5's speed NetCDF-4 (NF90_NETCDF4)
A private scratch file, same machine, read back once Ch. 7 unformatted/stream is fine

CF conventions checklist (for reproducible NetCDF)

  • units on every variable, from UDUNITS ("K", "m", "s", "seconds since 2000-01-01").
  • Coordinate variables: a 1D variable named like its dimension (x(x)) holding physical positions; add axis = "X"/"Y"/"Z"/"T".
  • long_name always; standard_name only if a value exists in the CF table — never invent one.
  • _FillValue to mark missing data.
  • Global: title, institution, source, history (append-only provenance log), Conventions = "CF-1.x".

Pitfalls

Pitfall Fix
Ignoring a nf90_* status / an hdferr Wrap every call: check(status) / test hdferr. Silent corruption otherwise.
nf90_put_var before nf90_enddef Define, end define mode, then write. Data only in data mode.
ncdump shows var(y, x) — panic Not transposed: ncdump is row-major, Fortran column-major; the library bridges them.
Leaking HDF5 handles in a loop Close every dataspace/dataset/plist you open — HDF5 objects are not allocatable.
Raw binary across machines Endianness scrambles it; self-describing formats record byte order.
Inventing a standard_name Use long_name; standard_name is a controlled vocabulary.

Numbers and rules worth memorizing

  • Text ≈ 3× binary in size (~24 chars vs 8 bytes per real(dp)), plus a conversion cost each way.
  • Compression needs chunking; shuffle before deflate helps smooth floating-point fields.
  • Chunk = the shape of your typical read (tens of kB–few MB); square tiles balance whole and region reads.
  • Deflate 4–6 is the smooth-field sweet spot; 9 buys little extra ratio for much CPU.
  • NetCDF-4 = HDF5 underneath — you rarely have to choose against yourself.
  • Overflow watch: a large byte count needs integer(int64), not default integer.

Compile commands introduced

$ gfortran -std=f2018 -Wall prog.f90 `nf-config --fflags --flibs` -o prog   # NetCDF
$ h5fc     -std=f2018 -Wall prog.f90 -o prog                                # HDF5

Heat-solver piece added this chapter

heat_io gains write_field_netcdf(field, filename, step, time) — takes the Ch. 9 field_t, writes a self-describing, CF-labelled, NF90_NETCDF4 snapshot. The scalable alternative to Ch. 7's text write_field; interchangeable at the call site (one bundled arg + filename). Because it is NetCDF-4 (HDF5 underneath), adding chunksizes=/deflate_level= later buys compression for free. This is the output the Ch. 38 capstone writes and the Ch. 26 visualization reads.