Case Study 1: The Fragile Parser

"The data is never as clean as the code that reads it assumes."

Executive Summary

You have inherited a routine that reads an instrument log — one line per reading, three fields per line: a sensor id, a temperature, and a status word. It works on the test file and then, on real data from the field, it starts returning garbage: temperatures that are off by a factor of ten, ids with stray characters, statuses that read as blank. The cause is the oldest string mistake in scientific computing: the routine parses by fixed columns, assuming every field sits at hard-coded character positions, and real data does not oblige. We will read the inherited code, pinpoint exactly why it breaks, and rewrite it to parse by content rather than by column — tokenizing on whitespace with scan and verify, and converting the numeric field with an internal read. The rewrite is shorter than the original and it does not care how the fields are spaced.

Skills applied: fixed- vs deferred-length strings and the trailing-blank trap (§12.1, §12.3); scan/verify to find field boundaries (§12.2); a reusable tokenizer (§12.5); internal reads for text→number conversion (§12.4); reading old string code and modernizing it (§12.6).

Background

The log lines look, in the test file, like this — every field neatly aligned:

S12   23.5  OK
S07   24.1  OK

and the inherited routine was written to match, pulling each field from the columns it "always" occupies. But the field data that arrives from the instruments is not column-aligned; it is whitespace-separated, and the spacing varies with the magnitude of the numbers and the length of the ids:

S12   23.5  OK
S3 100.25 FAIL

The second line has a shorter id, a wider temperature, and single spaces — and it is this kind of line that turns the parser's output to nonsense. Our job is to make the reader robust to any spacing, because that is what real data has.

Phase 1 — Read the Inherited Routine

Here is the parsing core you inherited, faithfully reproduced. Read it before the diagnosis:

! Inherited (fragile): each field is read from HARD-CODED columns.
character(len=6)  :: id
character(len=2)  :: status
real(dp)          :: temp
id     = line(1:6)          ! "the id is always in columns 1-6"
read(line(9:14), *) temp    ! "the temperature is always in columns 9-14"
status = line(17:18)        ! "the status is always in columns 17-18"

The routine encodes three assumptions in its three lines: that the id occupies columns 1–6, the temperature columns 9–14, and the status columns 17–18. On the aligned test line S12 23.5 OK those happen to hold — line(1:6) is 'S12 ', line(9:14) is '23.5 ', line(17:18)… already we are in trouble, because that test line is only 14 characters long, so column 17 does not even exist. The code "works" only on lines padded to a precise, undocumented layout.

Phase 2 — Find Where It Breaks

Feed it the realistic line S3 100.25 FAIL and trace the three assumptions against the actual columns (1:S 2:3 3:_ 4:1 5:0 6:0 7:. 8:2 9:5 10:_ 11:F 12:A 13:I 14:L):

Every field is wrong. The id has swallowed three characters of the temperature; the temperature field has captured the tail of the number and the start of the status; the status read runs off the end of the string entirely. And note the two string-specific hazards from the chapter hiding in here: the fixed-length id variable will carry trailing blanks wherever it is used next (§12.3), and the whole approach depends on a rigid column layout that no one wrote down. The routine is not merely buggy; it is built on a model of the data — "fields live at fixed columns" — that is simply false for whitespace-separated text.

The reasoning that matters: the fix is not to patch the column numbers. Any column numbers are wrong, because the data is not column-based. The fix is to change the model: parse the line as a sequence of whitespace-separated tokens, letting the content decide where each field starts and ends. That is exactly the job §12.5's tokenizer was built for.

Phase 3 — Parse by Content, Not by Column

Replace the column arithmetic with a small, reusable next_token that skips leading blanks (verify), takes characters up to the next blank (scan), and hands back a deferred-length token of exactly the right size — no trailing blanks, no assumptions:

subroutine next_token(line, pos, tok)
  character(len=*),          intent(in)    :: line
  integer,                   intent(inout) :: pos   ! advances past the token
  character(:), allocatable, intent(out)   :: tok
  integer :: first, last
  first = verify(line(pos:), ' ')          ! first non-blank in the remainder
  if (first == 0) then                      ! nothing but blanks left
    tok = ''
    pos = len(line) + 1
    return
  end if
  first = pos + first - 1                   ! absolute start of the token
  last  = scan(line(first:), ' ')           ! next blank after the token
  if (last == 0) then
    last = len(line)                        ! token runs to the end
  else
    last = first + last - 2                 ! absolute end of the token
  end if
  tok = line(first:last)
  pos = last + 1
end subroutine next_token

Each call returns the next field and advances pos, so pulling three fields is three calls — and it does not matter whether they are separated by one space or seven.

Phase 4 — Convert the Numeric Field Safely

The id and status are text, so we keep them as trimmed deferred-length strings. The temperature is a number encoded as text, so we convert it with an internal read (§12.4) — the same read(..., *) you would use on a file, aimed at the token string:

subroutine parse_reading(line, id, temp, status)
  use, intrinsic :: iso_fortran_env, only: dp => real64
  character(len=*),          intent(in)  :: line
  character(:), allocatable, intent(out) :: id, status
  real(dp),                  intent(out) :: temp
  character(:), allocatable :: ttext
  integer :: pos
  pos = 1
  call next_token(line, pos, id)        ! field 1: sensor id (text)
  call next_token(line, pos, ttext)     ! field 2: temperature (text -> number)
  call next_token(line, pos, status)    ! field 3: status word (text)
  read(ttext, *) temp                    ! internal read converts the text
end subroutine parse_reading

Phase 5 — Verify on Differently-Spaced Lines

Here is the whole thing as a running program, exercised on both the aligned test line and the realistic field line, to prove it no longer cares about spacing:

program robust_log
  use, intrinsic :: iso_fortran_env, only: dp => real64
  implicit none
  call show('S12   23.5  OK')      ! aligned, wide spacing
  call show('S3 100.25 FAIL')      ! short id, wide number, single spaces

contains

  subroutine show(line)
    character(len=*), intent(in) :: line
    character(:), allocatable    :: id, status
    real(dp) :: temp
    call parse_reading(line, id, temp, status)
    print '(a, a, a, f0.2, a, a)', 'id=', id, '  temp=', temp, '  status=', status
  end subroutine show

  ! parse_reading and next_token exactly as in Phases 3-4
  subroutine parse_reading(line, id, temp, status)
    character(len=*),          intent(in)  :: line
    character(:), allocatable, intent(out) :: id, status
    real(dp),                  intent(out) :: temp
    character(:), allocatable :: ttext
    integer :: pos
    pos = 1
    call next_token(line, pos, id)
    call next_token(line, pos, ttext)
    call next_token(line, pos, status)
    read(ttext, *) temp
  end subroutine parse_reading

  subroutine next_token(line, pos, tok)
    character(len=*),          intent(in)    :: line
    integer,                   intent(inout) :: pos
    character(:), allocatable, intent(out)   :: tok
    integer :: first, last
    first = verify(line(pos:), ' ')
    if (first == 0) then
      tok = ''
      pos = len(line) + 1
      return
    end if
    first = pos + first - 1
    last  = scan(line(first:), ' ')
    if (last == 0) then
      last = len(line)
    else
      last = first + last - 2
    end if
    tok = line(first:last)
    pos = last + 1
  end subroutine next_token

end program robust_log

$ gfortran -std=f2018 -Wall robust_log.f90 -o log && ./log
id=S12  temp=23.50  status=OK
id=S3  temp=100.25  status=FAIL

Sanity check. On the aligned line the tokenizer finds S12 (columns 1–3), then skips to 23.5 (columns 7–10), then OK (columns 13–14): id S12, temperature 23.50, status OK — correct. On the realistic line it finds S3 (1–2), 100.25 (4–9), FAIL (11–14): id S3, temperature 100.25, status FAIL — also correct, from a line the fixed-column parser mangled completely. Same code, two very different layouts, both right. The deferred-length tokens print with no trailing blanks because they were sized to their contents, and the internal read turned '100.25' into the number 100.25 without a single hard-coded column.

Discussion Questions

1. The inherited code's assumptions were reasonable if every producer of the log padded its fields to fixed columns. Whose responsibility is that contract, and why is parsing by content the more robust choice even when a fixed layout is promised?
2. The rewrite keeps id and status as text but converts temp to a number. What breaks if a line has a non-numeric temperature like 'N/A', and where in the chapter is the tool (iostat on the internal read) that would let you handle it gracefully?
3. next_token returns a deferred-length string. What concrete bug from Phase 2 does that eliminate compared with the fixed-length character(len=6) :: id?

Your Turn: Extensions

• Option A. Extend parse_reading to accept comma-or-space separated fields by changing the single blank delimiter to a two-character set (' ,') in next_token. Verify it parses S3,100.25,FAIL unchanged otherwise.
• Option B. Add an integer "reading number" as a fourth field and parse it with a second internal read. Confirm the tokenizer needs no structural change — only one more next_token call.
• Option C. Make parse_reading count the tokens first and return an error flag if a line does not have exactly three fields, previewing the defensive validation of Chapter 13.

Key Takeaways

• Parsing text by fixed columns encodes a false model of whitespace-separated data; the first differently-spaced line breaks every field at once.
• Parse by content: verify to find where a token starts, scan to find where it ends, a deferred-length string to hold it, and an internal read to convert the numeric ones.
• A reusable next_token turns "parse three fields" into three calls that are indifferent to spacing — shorter than the fragile original and correct on real data.
• Deferred-length tokens carry no trailing blanks, so the fields you extract are clean the moment you have them — no (1:n) slicing, no companion length counter.