Chapter 25 — Key Takeaways (Monitoring and Observability)
The page to work through once, and then to hand to whoever asks for an observability budget.
The reframing
Monitoring a data pipeline is not monitoring a service. A service is up or down. A pipeline can be perfectly healthy and produce the wrong answer, or perfectly dead and produce no signal at all. Every measurement here exists because one of those is true.
"Is the pipeline up?" is the wrong question. The right one has three parts:
- Is the data current? — not "did the job run"
- Is the data right? — Chapter 23's assertions, as signals
- Will it still be current tomorrow? ← almost nobody has this, and it is where every incident in Chapters 21, 22, and 24 was visible months ahead
Four signals, not the golden four
| Freshness | how old is the newest data | | Volume | how many rows arrived | | Distribution | does the data look like itself | | Schema / lineage | did the shape or the graph change |
📐 A test and a metric are different instruments. A volume floor at 3,000 correctly passes a 9,000-row day; a 49% drop is still a signal. The test protects the pipeline; the metric informs a person. Set the test loose enough to survive a bad Tuesday, then chart the same measurement tightly.
Telemetry
Metrics for trends · structured logs for one run · traces when a latency question spans three systems and you cannot answer it.
⚠️ Cardinality. A metric labelled order_id is 2.4 million time series a year. Labels: DAG,
task, table, source, status. Never an ID, timestamp, path, or error message.
Duration, and the number nobody publishes
Alert on the ratio to a trailing 14-run median, not a threshold.
$$r = \frac{\text{this run}}{\text{median(trailing 14)}}\qquad \text{alert } r > 2.0 \text{ \textbf{and} } r < 0.5$$
Scale-free · median absorbs outliers · r < 0.5 is usually not good news · fires on the day of
the change. Chapter 21's job went 4.2× slower and stayed inside its window for eleven days.
🔎 Publish four numbers per run:
duration_s peak_rss_mb rows_out finished_at
Four derived signals fall out: duration ratio · memory over limit · rows ratio · the margin. This needs a five-column table and a job that writes one row per run — not a platform.
The margin
⚠️ A pass/fail SLA cannot see erosion. It is 1 at four hours of margin and 1 at four minutes, so the entire process of running out of room is invisible and the first observable event is the failure.
Publish the margin and two things become possible: extrapolate (Kestrel: −10.4 min/month, from 41 minutes that is under four months to a date) and attribute (the slope decomposes into steps, and each step is a decision).
Put the SLA on the DAG's completion, not its tasks — no task-level measurement sees a DAG that started late.
Measure the quantity you are protecting, not the boolean you are reporting.
Memory and cost
Peak RSS ÷ container limit, alert at 0.60. Not 0.90 — the gap is the time you have to act, and it is where a busy Monday lives.
💸 Attribute cost per job per run and alert on the ratio. A job going from $0.06 to $0.18 is the same defect as $600 to $1,800, and only one of them appears on a bill. Part IV found $2,845/month, $6,307/year, $10,232/year and $31,361/year — none of it by a cost report.
One dashboard
Today and trend side by side · every line backed by a failure that happened · the decaying states visible (mutes, quarantine, red squares) · fits on a screen, because a dashboard that scrolls has an unread bottom half and the trends live there.
Alerts
An alert is a request that a person do something. If nothing should be done, it is a metric.
Actionable · says what to do (runbook, likely cause, owner) · routed by who fixes it · rate-limited.
🏭 Review quarterly with two questions: did anyone act, and should someone have? Kestrel's first review: 412 alerts, 89% no action, 36% never acknowledged — and two of those were real. After: 61 alerts, 44 acted on. 274 became dashboard rows.
Measure acknowledgment and action, not delivery. A rule at 0% acknowledgment provides the assurance of a rule that does not exist, even when every firing is correct.
"We need an alert for this" is a ratchet. Pair it: add one, and name one to remove or downgrade, or say why the total should grow.
Absences
Every alerting system is built on events; an absence is not one. Three failures share the shape — a dead scheduler (Ch. 24 CS2) · a broken alert route (Ch. 19 CS2) · a check never invoked (Ch. 19 §19.8).
A heartbeat converts absence into event, and the monitor must be outside the thing it monitors. A heartbeat proves only its own chain — Kestrel's trivial canary passed while the worker pool could not scale, so there are now two.
Retention
📏 Aggregates and run records: forever. 72,000 runs/year × 20 bytes = 1.44 MB/year, and they cannot be recreated. Detail: the length of a realistic investigation — a fortnight to a month. Downsample rather than delete.
Most teams have this exactly backwards: a year of verbose logs nobody queries past week two, and no run history at all.
Consumers
💸 Monitor who reads the output. Kestrel: 17 of 90 models read by nobody. Nine were dead — deleting them saved $214/month and bought back three months of margin. Five were read quarterly, and a 90-day window would have deleted them.
A usage report is a list of questions, not a list of deletions, and the window must be longer than your slowest legitimate consumer — 400 days, not 90.
The two case studies
The outage with a date on it. 100% compliance for eight quarters over a margin falling 4h41m → 41m. The metric had been collected since 2024 and never queried — what was missing was the question. The slope decomposed into six steps, five of which are incidents from earlier chapters, each found for an unrelated reason and fixed locally.
Ask of every green indicator: what number was thresholded to produce this? Then plot it. It needs no new data — the number is already there, or there would be no threshold.
📐 "The trend is bad" fails as a funding argument. "On the second of December the CEO opens an empty dashboard" worked immediately — a date and a named consequence. Then publish the forecast monthly, including when it moves out, or it stops being a measurement.
🏭 A 90-second addition is unarguable; forty of them are an outage. Put the margin impact in the pull request. It has never blocked a change and has made four authors find a cheaper approach.