Chapter 25: Monitoring, Alerting, and Observability

"Six checks, all green, and the number was wrong for thirty-one days. The checks were not broken. They were answering a question nobody had asked."

Overview

Chapter 24 ended with two failures: one where every task was green and the data was wrong, and one where nothing ran at all and nothing failed. Neither is unusual, and neither is detectable by anything most teams have.

This chapter is about what to measure instead. It is deliberately not a survey of tools — the tools change every two years and the questions do not — and it is organized around a single reframing that does most of the work:

Monitoring a data pipeline is not monitoring a service. A service is up or down and a request succeeds or fails. A pipeline can be perfectly healthy and produce the wrong answer, or be perfectly dead and produce no signal at all. Every measurement in this chapter exists because one of those two is true.

Chapter 21 Case Study 2 asked for alerting on duration change; that is §25.4. Chapter 22 Case Study 1 asked for memory published next to duration; that is §25.6. Chapter 24 §24.8 asked for an SLA on a DAG's completion rather than its tasks; that is §25.5.

25.1 The Question

"Is the pipeline up?" is the wrong question, and it is the one every default dashboard answers.

The right question has three parts, and a monitoring system is complete when it can answer all three without a human querying a warehouse:

1. Is the data current? Not "did the job run" — is the data in the table as fresh as its consumers believe.

2. Is the data right? Chapter 23's assertions, surfaced as signals rather than as build failures.

3. Will it still be current tomorrow? The trend questions: is duration creeping, is memory creeping, is the margin to the deadline shrinking. This is the part almost nobody has, and it is where every incident in Chapters 21, 22, and 24 was visible months in advance.

Two vocabulary notes, because the words are used loosely and the distinction matters here.

Monitoring is checking known things: you decided in advance what to measure and what is bad. Observability is the property that lets you answer questions you did not anticipate — which in practice means having enough retained detail that a new query is possible.

Data teams need both, and are usually short of the second. A team can tell you every job's status for the last year and cannot tell you why Tuesday's run produced 4% fewer rows, because nothing recorded anything that would answer it.

🎓 Interview Angle — "how would you know if a pipeline was silently wrong?"

A common question, and most answers list tools. The strong answer is a taxonomy, because the question is really asking whether you understand that there are several distinct failures wearing one name.

"Silently wrong splits four ways, and each needs a different instrument:

1. Wrong values, right shape — a sign flip, a unit change, a redefined column. Detected by distribution monitoring and by reconciliation against an independent system; no per-row test finds it.

2. Missing rows — a watermark with no lookback, a filter that over-matches. Detected by a volume floor and an anti-join, and by nothing else — every standard test passes on an empty table.

3. Right data, wrong place — a rerun writing today's rows into a historical partition. Detected by asserting a record's content against its location, which almost nobody does.

4. Nothing ran at all — a dead scheduler, a check never invoked. Detected by a heartbeat, because no event-based system can see an absence.

And then the closer: "the reason all four are hard is that a pipeline can be completely healthy and produce the wrong answer. Every one of those detections measures the data, or its absence, rather than the job."

Each of the four is a documented incident in this book — Chapter 20's watermark, Chapter 19's missing ref(), Chapter 24's rerun, and Chapter 24's dead scheduler. Being able to name the taxonomy is worth more than any tool name, because it demonstrates the thing the tools are for, and because the follow-up question is always "which would you build first?"

The honest answer to that follow-up is the volume floor and the heartbeat, in that order: they are the two cheapest, they cover the two failure classes that produce no signal whatsoever, and between them they would have caught three of the four.

25.2 Four Signals, and Why They Are Not the Usual Four

Operations has "the four golden signals": latency, traffic, errors, saturation. They are correct for a service and they miss the failures in this book, because all four are properties of request handling and a batch pipeline handles no requests.

The four for data:

These are the same four as Chapter 23's assertions, surfaced continuously rather than as a pass/fail at build time. That is not redundancy — a test tells you a build should stop; a metric tells you a trend is moving — and the second catches things the first cannot, because a test has to be wrong today to fire.

📐 Design Decision — a test and a metric are not the same instrument

Kestrel's volume floor (Chapter 19 Case Study 1) is set at 3,000 against a daily average of 17,753, because Chapter 23 §23.7 says to set thresholds for catastrophe rather than anomaly.

So a day that lands 9,000 rows passes the test. That is correct — 9,000 could be a public holiday, and a test that fires on it gets disabled.

And 9,000 rows is a 49% drop that somebody should look at, which is a metric question:

text daily order lines, trailing 28 days 17,7xx ──────────────────────────╮ ╰── 9,014 ← passes the test at 3,000 and is plainly a signal

The test protects the pipeline. The metric informs a person. They want different thresholds, different routing, and different urgency — the test blocks a build and pages nobody; the metric pages nobody and appears on a page somebody reads each morning.

The mistake teams make is picking one. Set the test loose enough to survive a bad Tuesday, then put the same measurement on a chart with a much tighter band, and let a human do what humans are better at than thresholds.

25.3 Metrics, Logs, and Traces

Three telemetry types, three jobs, and using the wrong one is expensive in a specific way.

Metrics are numeric time series with labels. Cheap, aggregatable, retained for years. Use them for anything you want a trend of — and they are the only one of the three you can afford to keep long enough to answer "has this been creeping?"

⚠️ Cardinality is the trap. A metric labelled with order_id creates one time series per order — 6,575 a day, 2.4 million a year — and it will take down your metrics backend. Labels must be low-cardinality: DAG, task, table, source, status. Never an ID, a timestamp, a path, or an error message.

Logs are timestamped events with context. Expensive to store, expensive to query, and the only thing that tells you what happened in one specific run.

Structured logs are the whole difference:

# ❌ Greppable by a person, aggregatable by nobody.
log.info(f"Loaded {n} rows for {dt} in {elapsed:.1f}s")

# ✅ Queryable. "Show me every load under 5,000 rows this quarter" is a
#    filter rather than a regular expression.
log.info("load_complete", extra={
    "event": "load_complete", "table": "bronze.orders", "dt": str(dt),
    "rows": n, "elapsed_s": round(elapsed, 1), "run_id": run_id,
})

Traces follow one unit of work across systems. Underused in data engineering and genuinely useful for the question "where did the six hours go?" across an extract, a load, a transform, and an export that live in four different tools.

The pragmatic position, and it is worth stating because the observability literature will push you further: metrics and structured logs cover almost everything in this chapter. Adopt tracing when you have a latency question spanning three or more systems and cannot answer it — not before.

25.4 Duration, and Why the Threshold Is Wrong

Duration is the most under-monitored property of a data pipeline, and the reason is that everyone monitors it in the way that does not work.

The usual approach: alert if a job takes more than N minutes. The problem: N has to be set above the worst legitimate run, so it is far above the normal one, and the whole band between normal and N is invisible.

Chapter 21 Case Study 2's job went from 9.1 minutes to 38.3 — a 4.2× regression — and ran inside its window for eleven days, reporting nothing, because the threshold was set where a threshold has to be set.

Alert on the ratio to a trailing median instead:

$$r = \frac{\text{this run}}{\text{median of the trailing 14 runs}} \qquad \text{alert when } r > 2.0$$

Four properties that make this work where a threshold does not:

It is scale-free. The same rule covers a 9-second job and a 90-minute one, so there is no per-job tuning and therefore no per-job neglect.

The median absorbs outliers. One bad night does not move the baseline, so the rule does not gradually accept a regression the way a mean would.

It catches improvements too. r < 0.5 is usually not good news — it means the job processed less than it should, which is Chapter 19 Case Study 1's missing day.

It fires on the day of the change, which is when the cause is knowable.

🔎 Read the Plan — the four numbers to publish per run

For every scheduled job, publish these four alongside its status. Together they are about twenty bytes and they answer every trend question in this chapter.

text run_id duration_s peak_rss_mb rows_out finished_at

From those four, derived:

• duration / median(14) — §25.4. Alert above 2.0 and below 0.5.
• peak_rss / container_limit — §25.6. Alert above 0.60.
• rows_out / median(14) — the volume signal, as a trend rather than a floor.
• deadline - finished_at — the margin, §25.5, which is the one nobody has.

Four raw numbers, four derived signals, and between them they would have caught the duration regression in Chapter 21, the memory growth in Chapter 22, the missing day in Chapter 19, and the SLA erosion in Chapter 24.

None of this needs an observability platform. It needs a table with five columns and a job that writes one row per run, and a team that has that table can build the rest in an afternoon.

25.5 The Margin

A deadline is met or missed. The margin is how close you came, and it is the difference between finding out in a quarter and finding out on the night.

Kestrel's 6am SLA (Chapter 1 §1.7) is met when gold.daily_revenue is current by 06:00 America/New_York. For two years it was met every day, and the trend looked like this:

margin to 06:00, monthly median
2024-09   4h 41m  ████████████████████████
2025-03   4h 12m  █████████████████████
2025-09   3h 26m  █████████████████
2026-03   2h 39m  █████████████
2026-07   1h 17m  ██████          ← still met, every day
2026-08   0h 41m  ███             ← still met, every day

Every one of those months is a success by the only measurement anyone had. And the chart is a scheduled outage with a date on it.

⚠️ Failure Mode — a pass/fail SLA cannot see erosion

This is the single most valuable idea in this chapter, and it costs one column.

An SLA measured as compliance produces a series of 1s. It is 1 at four hours of margin and 1 at four minutes, so the entire process of running out of room is invisible, and the first observable event is the failure.

Publish the margin. Then two things become possible:

• Extrapolate. Kestrel's margin fell 240 minutes over 23 months — 10.4 a month. From August's 41 minutes that is just under four months to zero: a date, computable, on a chart, and arriving before the next planning cycle.
• Attribute. The margin's drops are steps, not a smooth slope, and each step lines up with a change: a model added, a source grown, a coalesce(1) deployed. A smooth-looking trend decomposes into decisions once you can see it.

And the corollary for Chapter 24 §24.8: put the SLA on the DAG's completion time, not on its tasks. Airflow's task-level SLA miss did not fire during the backfill starvation, because no individual task was slow — the DAG was slow because it started late, and no task-level measurement can see that.

The general form: measure the quantity you are protecting, not the boolean you are reporting.

25.6 Memory and Cost

Both are monitored the same way, both are under-monitored for the same reason — they are not in the scheduler's default UI — and both produce failures that arrive without warning.

Memory. Chapter 22 Case Study 1: a job read 34 columns to use 8, and the data grew 3% until 147 MB of Parquet became 4,145 MB of RSS against a 4,096 MB limit. The failure was exit code 137 with no traceback, and three days went into the platform.

$$\text{publish } \frac{\text{peak RSS}}{\text{container limit}}, \quad \text{alert above } 0.60$$

Not 0.90. The gap between 60% and 90% is the time you have to act, and it is where a busy Monday lives.

Cost. Every chapter in Part IV found money in a place nobody was looking: $2,845/month in a view chain, $6,307/year in a Spark job that should not have been one, $10,232/year in a coalesce(1), $31,361/year in a skewed aggregation.

None of those was found by a cost report. All four were found by an unrelated investigation, which is the argument for the metric:

💸 Cost Check — attribute cost per job, not per warehouse

A monthly cloud bill is a number nobody can act on. Cost per job per run is a number with an owner, and getting it is easier than it sounds:

• Snowflake — QUERY_HISTORY gives credits per query; tag queries with the DAG and task.
• BigQuery — INFORMATION_SCHEMA.JOBS gives bytes billed; label jobs.
• Spark / EMR / Databricks — executor-seconds × the instance rate. Chapter 3's basis.

Then publish cost_per_run next to duration_s, and alert on the same ratio rule as §25.4 — above 2.0 against a trailing median.

Kestrel's version of this found $180/month in its first week, in a model whose cost had tripled when someone removed a WHERE clause during a refactor. The refactor was correct and the cost was not, and nothing else would have connected them — the query was faster, the tests passed, and the bill was one line item among four hundred.

The threshold that matters is not a dollar amount. It is the ratio, because a job that costs $0.06 and starts costing $0.18 is the same defect as one that goes from $600 to $1,800, and only one of them shows up on a bill.

25.7 The One Dashboard

Most data dashboards are built and then not looked at, and the reason is that they are built around what is easy to plot rather than around a decision.

Build one. It answers §25.1's three questions for the tables that matter, and it fits on a screen.

┌─ KESTREL DATA HEALTH ─────────────────── 2026-09-01 07:14 ──┐
│                                                              │
│  TODAY                                                       │
│  6am SLA          ✅ met, margin 41m        ← §25.5          │
│  daily_revenue    fresh 05:19  (target 06:00)                │
│  fct_order_item   17,904 rows   (median 17,753, +0.9%)       │
│  tests            313/313                                    │
│  unknown-member   6 (threshold 50)          ← Ch. 19 CS2     │
│                                                              │
│  TRENDS (28d)                                                │
│  margin           1h17m → 41m   ▼ 36m       ← the alarm      │
│  DAG duration     2h31m → 3h09m ▲ 25%                        │
│  peak memory      58% of limit  ▲ from 51%  ← Ch. 22 CS1     │
│  cost / night     $19.40        ▲ 11%                        │
│                                                              │
│  OPEN                                                        │
│  muted checks     1  (2 days, data-eng)     ← Ch. 23 CS2     │
│  quarantine       18 rows, owner: data-eng  ← Ch. 23 CS1     │
│  red task states  0                         ← Ch. 24 CS1     │
└──────────────────────────────────────────────────────────────┘

Four properties that make this the one that gets read:

Today and trend, side by side. Today's numbers say whether to act now; the trends say whether to plan. Most dashboards have only the first.

Every line has a chapter behind it. Nothing is there because it was easy to plot — each row is a failure that has actually happened.

The OPEN section is the states that decay into noise — mutes, quarantine, red squares — which is Chapter 23 Case Study 2's and Chapter 24 Case Study 1's shared lesson, made visible.

It fits on a screen. A dashboard that requires scrolling is a dashboard whose bottom half is not read, and the bottom half is where the trends usually go.

25.8 Alerts That Get Acted On

An alert is a request that a person do something. If nothing should be done, it is a metric.

Four requirements, and most alerts fail the second:

Actionable. Something to do, now. "CPU is 78%" is not an alert.

Says what to do. The message names the runbook, the likely cause, and the owner.

❌  fct_order_item: dbt test failed
✅  fct_order_item grain violated: 1,204 duplicate (order_id, line_number).
    Likely: int_order_items_deduped lost its tiebreaker (Ch. 18 §18.7).
    Runbook: docs/runbooks/grain-violation.md   Owner: #data-eng

Routed by who fixes it. Chapter 24 §24.12: platform health and pipeline failures are different channels, different owners, different urgency. Chapter 23 §23.12: a source freshness failure belongs to the producing team.

Rate-limited. Chapter 24 Case Study 2: forty-one correct alerts, all ignored. Alert on the rate, not the event, and never make a person do that arithmetic at 02:00.

🏭 From the Pipeline — the alert review, and the two questions

Once a quarter, list every alert that fired and ask two things:

1. Did anyone do anything? If not, it is a metric. Move it to the dashboard. 2. Should someone have done something and did not? That is a routing or wording failure, not a diligence failure.

Kestrel's first review covered 412 alerts over three months:

acted on	47	11%
acknowledged, no action	218	53%
not acknowledged at all	147	36%

Eighty-nine percent produced no action, and the 36% nobody acknowledged is the number that matters — those are the alerts that had already stopped being read.

After: 61 alerts in the following quarter, 44 acted on. The rest became dashboard rows.

The uncomfortable finding: two of the 147 unacknowledged alerts were real and were resolved later by other means. The alerts were correct and the system had already trained everyone to ignore them — which is Chapter 23 §23.7's mechanism, measured.

25.9 Absences

Chapter 24 Case Study 2's lesson, generalized: every alerting system is built on events, and an absence is not one.

The three absence-shaped failures in this book, all of which produce zero signal:

A heartbeat converts an absence into an event: something must arrive, so its non-arrival is detectable. And each heartbeat proves a specific chain, which is the part to be deliberate about.

The rule: the monitor must be outside the thing it monitors. An alert Airflow sends when Airflow is down is not an alert, and this is the most common way the canary is built wrong.

And the second rule, which is less obvious: a heartbeat proves only its own chain. Kestrel's canary proved the scheduler was alive during an incident in which the worker pool could not scale — because the canary's own task was tiny and got the one available slot. They added a second canary that requests a realistic resource envelope, and the two together are diagnostic where either alone is ambiguous.

25.10 Lineage as an Instrument

Chapter 22 Case Study 2 took three days to diagnose because the alert named where the defect was visible, not where it was.

Lineage turns that search into a list. dbt's manifest already has it (Chapter 19 §19.9), and the useful application is not a diagram — it is two things in text:

In the alert. When fct_order_item fails, the message lists its six sources and their freshness. A three-day search becomes a six-item checklist, and it is a manifest lookup.

In the impact assessment. Before changing a model, dbt ls --select model+ and the exposures downstream of it. Chapter 19 §19.12.

What lineage does not do is worth saying, because lineage tooling is sold as if it does: it shows you where a defect could have come from, not where it did. It narrows a search; it does not perform one, and a team that buys a lineage product expecting the second will be disappointed by a tool that is doing its job.

25.11 Retention: How Long to Keep What

Every signal in this chapter has a retention decision attached, and getting it wrong is expensive in two opposite directions: keep too much and the bill is real, keep too little and the trend questions in §25.1 become unanswerable exactly when you need them.

The asymmetry is what makes this worth a section. Chapter 21 Case Study 2's duration regression needed fourteen days of history to detect and eleven days to explain. Chapter 22 Case Study 1's memory growth needed two years. §25.5's margin chart needs three years before its slope is worth extrapolating.

And the three have wildly different storage costs, which is the resolution:

📏 Scale Note — the five-column table is free, and it is the one you cannot reconstruct

Kestrel runs about 72,000 job executions a year — eleven DAGs, several hourly. At twenty bytes per run record:

$$72{,}000 \times 20\ \text{bytes} = 1.44\ \text{MB a year}$$

A decade of it costs about a penny. And it is the only artifact in the table above that cannot be recreated: logs can be re-derived by a rerun, metrics can be re-aggregated, but the duration of a run in March 2025 exists in exactly one place or in none.

So the rule is not "retain everything":

• Aggregates and run records: forever. Tiny, and irreplaceable.
• Detail: the length of a realistic investigation — for a data platform, a fortnight to a month, not a year.
• Downsample rather than delete. One-minute resolution for fifteen days and hourly thereafter answers both the incident question and the trend question, at a fraction of the storage of either extreme.

Almost every team has this exactly backwards: paying to keep verbose logs for a year that nobody queries past week two, while retaining no run history at all — which is what every trend question in this chapter needs. The correction is cheaper than the status quo, which is unusual enough to be worth saying twice.

25.12 Monitoring the Consumers

Everything so far measures the pipeline. The other half is whether anyone reads what it produces, and it is the half that is almost never instrumented.

This is not idle curiosity. Chapter 18 Case Study 1 found a ranking column serving a dashboard decommissioned eight months earlier — one of four window specifications forcing a sort over 1.9 million rows, for nobody. The cost of unread output is paid every night, forever, and nothing surfaces it.

Three signals, all cheap, all from things you already have:

Query logs, by table. Snowflake's ACCESS_HISTORY, BigQuery's INFORMATION_SCHEMA.JOBS, Postgres' pg_stat_statements. The question: which tables were read in the last N days, by whom, and how often?

-- Snowflake. The list nobody has, in one query.
SELECT value:objectName::string AS table_name,
       COUNT(*)                 AS reads,
       COUNT(DISTINCT user_name) AS readers,
       MAX(query_start_time)    AS last_read
  FROM snowflake.account_usage.access_history,
       LATERAL FLATTEN(base_objects_accessed)
 WHERE query_start_time > dateadd(day, -400, current_timestamp)
 GROUP BY 1 ORDER BY reads;

Dashboard usage, from your BI tool. Most expose it; almost nobody exports it.

Exposures (Chapter 19 §19.9) — the declared consumers. The gap between the declared list and the observed list is the finding, and it runs both ways: a table read by nobody, and a table read by people nobody knew about.

💸 Cost Check — the seventeen models with no readers

Kestrel ran the query above and cross-referenced it against dbt ls:

text 90 models read in the last 90 days 54 read only by the pipeline that builds them 19 ← intermediate; fine read by nobody at all 17 ← ?

Seventeen models, built nightly, read by no human and no downstream model. Investigating them found a mix, and the mix is the entire point:

genuinely dead — a decommissioned dashboard, two abandoned experiments, a mart for a team that reorganized	9, deleted
read quarterly — which a 90-day window is exactly the wrong length to see	5, kept
read by a service account nobody could identify	2, kept pending investigation
a regulatory archive that is supposed to be unread	1, kept and documented, which it had not been

Deleting the nine saved $214 a month and removed 31 minutes from the nightly DAG — which, per §25.5, bought back three months of margin.

The lesson is not the $214. It is that a 90-day window would have deleted five models that are read quarterly, and the two nobody could identify were the most valuable thing the exercise produced. A usage report is a list of questions, not a list of deletions — and the window has to be longer than your slowest legitimate consumer, which for anything touching a quarterly close means 400 days rather than 90.

25.13 Instrumenting Your Own Pipelines

Start with a decorator and a table. Not a platform.

def instrumented(job_name):
    """Emit the four numbers from §25.4's callout, per run."""
    def wrap(fn):
        @functools.wraps(fn)
        def inner(*a, **kw):
            t0, peak = time.perf_counter(), PeakRSS()     # Ch. 22's sampler
            status, rows = "success", None
            try:
                with peak:
                    rows = fn(*a, **kw)
                return rows
            except Exception:
                status = "failure"
                raise
            finally:
                emit_run_record(
                    job=job_name, run_id=RUN_ID, status=status,
                    duration_s=round(time.perf_counter() - t0, 2),
                    peak_rss_mb=round(peak.peak_mb),
                    rows_out=rows if isinstance(rows, int) else None,
                    finished_at=datetime.now(timezone.utc),
                )
        return inner
    return wrap

emit_run_record appends one row to a table. Everything in §25.4, §25.5, and §25.6 is a query against that table.

Then, and only if you need it, adopt a standard. OpenTelemetry is where the industry is converging and is the right target if you are already emitting telemetry from services. StatsD is simpler and Airflow emits to it natively, which makes it the cheapest way to get DAG-level metrics with no code at all.

Airflow's built-in metrics are worth enabling on day one, whatever else you do:

[metrics]
statsd_on = True
statsd_host = statsd
statsd_prefix = airflow

That gives you dag.<id>.duration, ti_failures, scheduler.heartbeat, and pool occupancy — which covers §25.4 and half of §25.9 for the cost of four lines of configuration.

🔁 Idempotency Check — your monitoring must survive a replay too

A backfill of thirty intervals emits thirty run records, thirty freshness measurements, and potentially thirty alerts — and monitoring that treats a backfill as thirty incidents is monitoring that gets muted.

text what a 30-day backfill does to naive monitoring ───────────────────────────────────────────────────────────────────── duration alerts 30 runs at once; the trailing median is destroyed for the next 14 runs (§25.4) freshness the table's max(event_date) jumps backwards as old intervals land -- which reads as a failure volume assertions each interval is one day's volume; fine. But the TABLE's daily row count spikes 30x on the day the backfill ran. cost alerts a genuine 30x spike, correctly reported, and nobody wants to be paged for it SLA / margin the backfill's runs have no deadline, and mixing them into the margin calculation is wrong

The fix is one column, everywhere: run_type.

text run_id | dag_id | run_type | interval | started | ended | status ───────┼────────┼───────────┼─────────────┼─────────┼───────┼─────── 4182 | daily | scheduled | 2026-11-27 | ... 4183 | daily | backfill | 2024-03-14 | ... <- excluded from medians, margins, and cost alerts

Then every trailing median, every margin computation, and every duration ratio filters run_type = 'scheduled'. It is one predicate and it removes the entire class.

And the freshness measurement needs the same care in the other direction. Freshness should be measured against the scheduled interval, not against max(event_date) — a backfill legitimately lands old data, and a freshness check that reads the table's maximum will report a regression that is the backfill working.

The general principle: monitoring is a consumer of your pipeline's metadata, and it needs the same idempotency thinking as any other consumer. A signal that cannot distinguish a replay from an incident will be muted after the second backfill, and it will be muted permanently (Exercise 23.16).

🧪 Try It — compute your margin, and extrapolate it

Twenty minutes, and it produces the most important number in this chapter.

sql -- 1. the margin, per night, for 90 days SELECT interval_date, deadline_at, finished_at, date_diff('minute', finished_at, deadline_at) AS margin_minutes FROM sla_runs WHERE dag_id = 'kestrel_daily' AND run_type = 'scheduled' AND interval_date > current_date - 90 ORDER BY 1;

```python

2. fit a line and extrapolate to zero

import numpy as np days = np.arange(len(margins)) slope, intercept = np.polyfit(days, margins, 1) days_to_zero = -intercept / slope if slope < 0 else float("inf") print("slope: %.3f min/day -> margin hits zero in %.0f days" % (slope, days_to_zero)) ```

Three things to do with the answer.

Fit the last 30 days as well as the last 90, and compare the slopes. If the recent slope is steeper, the trend is accelerating and the 90-day extrapolation is optimistic. At Kestrel the 90-day fit gave 401 days and the 30-day fit gave about 120, and the second one is the number to plan against.

Plot it, and put the plot on the one dashboard (§25.7). A single line with a zero crossing is the most legible thing a data platform can show a manager.

And if the date is inside two quarters, say so first, in every conversation about the platform for the next month. It is the only number on the dashboard that is about the future, and everything else competes for attention with something that has already happened.

Then do the harder half: attribute the slope. Exercise 25.24's retrospective application of the margin-impact line to the last ten merged changes. The total is always larger than anyone would have guessed, because each individual change was unarguable.

🔐 Privacy & Governance — an alert is a message to a channel you do not control

Monitoring's whole job is to send information out of the system, which makes every alert a small data export.

text what routinely ends up in an alert payload ───────────────────────────────────────────────────────────────────── "412 rows failed not_null(email)" a count. Fine. "...for customer_id 8841, 9002, 9104" identifiers, in Slack, retained by Slack, searchable by everyone in the channel a query result attached as a snippet rows, in a chat message a link to a dashboard filtered to the failing rows fine -- the data stays put a stack trace containing a rendered SQL statement with literals identifiers, unintentionally

Row 2 is the common one and it is usually well-intentioned — the alert is more actionable with examples in it. The examples are personal data in a third-party system with its own retention and a much wider audience than the warehouse.

Three rules that keep alerts actionable without exporting anything:

Send counts and links, never rows. "412 rows failed; see " is more actionable than five example ids, because the recipient gets all 412 rather than a sample.

Redact rendered SQL in error messages. A failing statement with a literal WHERE email = '...' in it will be pasted into a chat message by somebody during an incident. Parameterise, and log the parameter names rather than the values.

And treat the alert channel as a data destination in the access review (Chapter 30). Who is in #data-alerts? Almost always more people than have warehouse access, and nobody has ever reviewed it because it is a chat channel rather than a system.

The uncomfortable trade worth naming: the most useful alert and the safest alert are not the same alert. An alert with five example rows in it resolves faster. The resolution is the link — it is both, and it costs one extra click — and saying that out loud is how a team accepts the rule rather than routing around it.

🧭 Version Note — "data observability" became a product category, and the argument did not change

Between about 2019 and now, a set of tools appeared that promise to tell you when your data is wrong. They are genuinely useful and they answer a narrower question than their name suggests.

text what these products do well what they cannot do ───────────────────────────────────────────────────────────────────────── freshness, automatically, across tell you a number is WRONG. They can hundreds of tables tell you it is UNUSUAL. volume anomalies against a know your business rules. A gift card learned baseline excluded from revenue is not an anomaly; it is a definition. schema-change detection reconcile against an external system column-level lineage from query distinguish a backfill from an logs -- genuinely hard, and incident, unless you tell them genuinely valuable

The right-hand column is Chapter 23, and it is not a criticism of the products — it is the distinction between §23.3's two questions, arriving as a purchasing decision.

Anomaly detection has a specific limitation worth internalising (§25.8): it learns from history, so a defect that has always been present is the baseline. Chapter 20's Case Study 2 lost rows for eight months; an anomaly detector trained on those eight months would report the fix as the anomaly.

What to buy them for, honestly: freshness and volume across a table count too large to instrument by hand, and lineage. What not to expect: correctness, reconciliation, or business rules — all three of which are assertions somebody has to write, and none of which any product can infer.

The evaluation question from Chapter 5 §5.8, applied here: what would this tell us that we do not already know? For a platform with forty tables and a quality register, the honest answer is often "lineage, and not much else" — and lineage may well be worth it on its own.

🏭 From the Pipeline — the dashboard that was deleted, and the one that was kept

Kestrel had eleven operational dashboards. A review deleted eight of them, and the exercise is worth describing because the objections were instructive.

text dashboard kept? why ───────────────────────────────────────────────────────────────────────── "Platform Health" NO 42 tiles, all green, nobody had looked at it in 90 days "Airflow DAG Status" NO duplicated the orchestrator's UI "Warehouse Credits by Day" NO no per-job attribution, so it could not answer a question "Data Freshness (all tables)" NO 211 rows; a wall of green with three reds that were muted "Pipeline Runtimes" NO absolute durations, no baseline ... three more NO ───────────────────────────────────────────────────────────────────────── "Today and the Trend" YES the one from §25.7 "Cost by Job" YES built during the review "The Margin" YES built during the review

The three that survived were two that did not exist.

The objections, and what each turned out to mean:

"I use that one." — Two people said this, about two different dashboards. Both, when asked when they had last opened it, said "during an incident." Neither could name the incident. The query log showed one view in four months.

"It's the only place you can see X." — True for "Warehouse Credits by Day", and the answer was to build the thing that actually answers the question rather than to keep the thing that gestures at it.

"Deleting it will make us less observable." — This is the objection to take seriously and it is backwards. Eleven dashboards, of which one was read, is less observable than three that are read, because attention is the scarce resource and a wall of green consumes it without paying anything back.

The measurement that made the review possible was the BI tool's own view log — who opened what, how often, in ninety days. It took twenty minutes to extract and it ended every argument, which is the same shape as Exercise 25.18's read audit and Chapter 30's usage statistics.

Six months later, nobody has asked for any of the eight back. That is the outcome to expect and it is worth saying in advance, because the fear of deleting is the whole obstacle.

🔎 Read the Plan — three queries against your own run history

```sql -- 1. the margin, and its slope (§25.7) SELECT interval_date, date_diff('minute', finished_at, deadline_at) AS margin_min FROM sla_runs WHERE run_type = 'scheduled' ORDER BY 1;

-- 2. duration as a RATIO to a trailing median (§25.4) SELECT task_id, interval_date, duration_s, duration_s / median(duration_s) OVER ( PARTITION BY task_id ORDER BY interval_date ROWS BETWEEN 14 PRECEDING AND 1 PRECEDING) AS ratio FROM sla_runs QUALIFY ratio > 2.0 OR ratio < 0.5;

-- 3. rows_out, which almost nobody records and which answers -- "did it do less work, or was it just faster?" SELECT task_id, interval_date, rows_out, rows_out / median(rows_out) OVER (...) AS row_ratio FROM sla_runs QUALIFY row_ratio < 0.75; ```

Query 3 is the one worth building first, and it needs rows_out in the run record — which is why Exercise 25.23(a) says to do that before anything else.

A fast run with a normal row count is a genuine improvement. A fast run with three-quarters of the rows is Chapter 20's Case Study 2, eight months of silent loss, visible in one query on the first night. Duration alone cannot distinguish them, which is the argument for the third column.

And notice the frame in query 2: ROWS BETWEEN 14 PRECEDING AND 1 PRECEDING. The current row is excluded from its own baseline. Including it dilutes exactly the anomaly you are trying to detect — a small error every day becomes the median — and it is the single most common bug in a hand-written anomaly check.

25.14 The Kestrel Platform

🧱 Kestrel Platform — Increment 25: the health signal

text platform/observability/ run_records.sql ← the five-column table everything derives from instrument.py ← §25.11's decorator health.py ← duration ratio, margin, memory, cost, freshness alerts.yml ← routing, severity, rate limits, runbook links dashboard/health.sql ← the one dashboard, as queries dags/ canary.py ← trivial: proves the chain canary_sized.py ← §25.9: requests a realistic resource envelope

Seven things this increment must get right:

1. Every scheduled job writes a run record with the five columns. Nothing else in this increment works without it.
2. The margin to the 6am SLA is published daily and charted over 90 days. §25.5 — it is the one metric on this list that has a date attached.
3. Duration and cost alert on the ratio to a trailing 14-run median, above 2.0 and below 0.5.
4. Peak memory over container limit, alerting at 0.60.
5. Two canaries — trivial and resource-sized — with an external monitor. §25.9.
6. Every alert names a runbook and an owner, and the alert file is reviewed quarterly against §25.8's two questions.
7. One dashboard, fitting on a screen, with today and trend side by side. Delete the others.
8. Run records retained forever; verbose logs 21 days. §25.11 — and check which way round yours currently is.
9. A quarterly consumer report cross-referencing query logs against dbt ls and the exposures, over a 400-day window. §25.12. Treat the output as questions, not as a delete list.

The exercise that matters is 25.23(b): compute Kestrel's margin for the last 90 days and extrapolate it to zero. If the answer is a date inside the next two quarters, that is the most important number your platform produces, and nothing else on the dashboard competes with it.

25.15 Summary

"Is the pipeline up?" is the wrong question. The right one has three parts: is the data current · is it right · will it still be current tomorrow. The third is the one almost nobody has, and it is where every incident in Chapters 21, 22, and 24 was visible months ahead.

Four signals for data — freshness, volume, distribution, schema/lineage — not the operational four, because a batch pipeline handles no requests.

📐 A test and a metric are different instruments. The volume floor at 3,000 correctly passes a 9,000-row day; a 49% drop is still a signal. The test protects the pipeline; the metric informs a person. Set the test loose and chart the same measurement tightly.

Metrics for trends, structured logs for one run, traces when a latency question spans three systems. ⚠️ Never label a metric with an ID — one time series per order is 2.4 million a year.

Duration: alert on the ratio to a trailing median, not on a threshold. Scale-free, outlier-robust, fires on the day of the change, and r < 0.5 is usually not good news. Chapter 21's job went 4.2× slower and stayed inside its window for eleven days.

🔎 Publish four numbers per run — duration_s, peak_rss_mb, rows_out, finished_at — and four derived signals fall out. This needs a five-column table, not a platform.

⚠️ A pass/fail SLA cannot see erosion. Kestrel's margin went 4h41m → 41m over two years, every month a success, falling 10.4 minutes a month — which from 41 minutes is under four months to zero. And the steps decompose into decisions. Measure the quantity you are protecting, not the boolean you are reporting.

Put the SLA on the DAG's completion, because no task-level measurement sees a DAG that started late.

Memory: publish peak over limit, alert at 0.60, because the gap to 0.90 is the time you have to act. Cost: attribute per job per run and alert on the ratio, because a job going from $0.06 to $0.18 is the same defect as $600 to $1,800 and only one appears on a bill.

Build one dashboard. Today and trend side by side, every line backed by a failure that happened, the decaying states visible, and it fits on a screen.

🏭 An alert is a request that a person do something. Actionable · says what to do · routed by who fixes it · rate-limited. Review quarterly with two questions: did anyone act, and should they have? Kestrel's first review found 89% produced no action and 36% were never acknowledged — and two of the unacknowledged ones were real.

Every alerting system is built on events; an absence is not one. Heartbeats convert absence into event — and a heartbeat proves only its own chain, so a trivial canary can pass while the worker pool cannot scale.

Lineage narrows a search; it does not perform one. Put the upstream freshness in the alert and a three-day investigation becomes a six-item checklist.

📏 Retention: keep aggregates and run records forever — 1.44 MB a year, and they cannot be recreated. Keep detail for the length of a realistic investigation, a fortnight to a month, and downsample rather than delete. Most teams have this exactly backwards.

💸 Monitor the consumers, not only the producers. Kestrel found 17 of 90 models read by nobody; deleting the nine that were genuinely dead saved $214/month and bought back three months of margin. Five more were read quarterly, and a 90-day window would have deleted them — a usage report is a list of questions, not a list of deletions.

Chapter 26 takes everything here and asks the question it implies: when one of these fires at 04:00, who wakes up, what do they read, and what are they allowed to decide?

Key terms: monitoring · observability · metric · log · trace · structured logging · cardinality · freshness · volume · distribution · duration ratio · trailing median · margin · SLA · SLO · alert fatigue · actionability · routing · runbook · heartbeat · canary · lineage · OpenTelemetry · StatsD