Further Reading: Monitoring, Alerting, and Observability

Sources are tagged Tier 1 (confident it exists, recommended without reservation) or Tier 2 (real and worth seeking, but confirm the current edition, version, or URL yourself).

The best material on this topic was written for services, not for data, and that is not a limitation to work around — it is where the thinking is. The operations community solved alert fatigue, actionability, and the absence problem before data engineering had the vocabulary. What does not transfer is the assumption that a healthy system produces correct output, which is §25.1's whole premise, and which no SRE text has reason to question.

The foundations

Google's Site Reliability Engineering (O'Reilly, 2016), and The Site Reliability Workbook (2018). Free online, both. Read "Monitoring Distributed Systems" and "Being On-Call" from the first, and the SLO chapters from the second. The four golden signals are here; so is the argument that every page must be actionable, and so is the error-budget framing that §25.5's margin is a variant of. The single most useful chapter for this material is the workbook's on alerting, which works through the arithmetic of burn rates in a way §25.4's ratio rule is a simplification of. Tier 1.
Rob Ewaschuk, "My Philosophy on Alerting." An internal Google document, widely circulated, short, and the origin of most of §25.8. "Every page should be actionable" and "pages should be about symptoms, not causes" are both here, argued rather than asserted. Tier 2 — it circulates as a Google Doc and a dozen mirrors; find a version and check it is complete.
Charity Majors, Liz Fong-Jones, and George Miranda, Observability Engineering (O'Reilly, 2022). The book that made the monitoring/observability distinction precise, which §25.1 uses. Read it for the distinction and the high-cardinality argument; note that its core claim — that you should instrument with wide, high-cardinality events rather than pre-aggregated metrics — is in tension with §25.3's cardinality warning, and the tension is real. Their answer is a purpose-built column store; yours is probably Prometheus, where the warning stands. Tier 1, read with that caveat.
Cindy Sridharan, Distributed Systems Observability (O'Reilly, 2018). Short, free, and the clearest treatment of what metrics, logs, and traces are each for. §25.3 is a data-shaped compression of it. Tier 2 — a free O'Reilly report; confirm it is still available.

On the data-specific problems

Barr Moses, Lior Gavish, and Molly Vorwerck, Data Quality Fundamentals (O'Reilly, 2022). Recommended in Chapter 23 and again here for a different part: the "data downtime" metric in §25.11, and the freshness/volume/distribution/schema framing in §25.2, which is theirs. Same caveat as before — the authors sell an observability platform, and the sections arguing that automated detection is the primary control should be read against §25.2's test-versus-metric distinction. Tier 1.
The OpenLineage specification and Marquez. An open standard for emitting lineage events from pipelines, with Airflow and dbt integrations. §25.10's "put the upstream freshness in the alert" needs lineage in a machine-readable form, and this is the vendor-neutral one. Worth knowing exists even if you use dbt's manifest instead, which for a dbt-centric platform is simpler and enough. Tier 2 — an active project; check current adapter support.
The Airflow documentation on metrics (StatsD and OpenTelemetry). §25.13. Four lines of configuration gives you DAG duration, task failures, scheduler heartbeat, and pool occupancy — which covers §25.4 and half of §25.9 with no application code at all. Tier 1.

On alerting as a human problem

The clinical alarm-fatigue literature. Genuinely worth reading, and almost nobody in software does. Hospitals measured what Case Study 2 measured — the proportion of alarms that produce action, and what happens to the ones that do not — with far better data and much higher stakes. The findings transfer intact: the proportion ignored is a function of the base rate, not of the individual alarm's quality, and the fix is never to train people to be more careful. Tier 2 — search the medical literature for "alarm fatigue"; the review articles are the way in.
John Allspaw on incident analysis, and the resilience-engineering literature generally. Case Study 2's finding that two correct alerts were ignored because of their 411 neighbours is a systems-property finding, not a diligence finding, and this literature is where that distinction is made carefully. Tier 2 — talks and blog posts; scattered.
Anything on "the ratchet" in operational process. Case Study 2's central mechanism — every incident adds a control and nothing removes one — is a general property of postmortem culture and is under-discussed relative to how universal it is. The paired action item is not a standard practice, and if you find good writing on it, it is worth more than another alerting tool. Tier 2 — thin; this is a gap in the literature.

Tools, briefly

Prometheus + Grafana for metrics, Loki / Elasticsearch / a cloud log service for structured logs, Grafana Alerting or Alertmanager for routing and rate limiting. This stack does everything in this chapter, and the routing and grouping features in Alertmanager are the ones worth learning properly — they are what §25.8's rate-limiting requirement is asking for. Tier 1 for Prometheus and Grafana; the log layer is a choice.
dbt_artifacts and similar packages that load run_results.json into your warehouse. This is the cheapest possible route to §25.4's five-column table for a dbt project — the data already exists after every build and nobody keeps it. Tier 2 — a community package; check maintenance.
Data observability vendors. Chapter 23 §23.8's assessment applies unchanged: a net for the unknown, not a control for the known. Evaluate one after you have the four numbers per run and the six assertions, and evaluate it on coverage across tables you do not own, which is the problem it is genuinely good at.

Practice

code/health.py in this chapter. No dependencies. Feed it a five-column run history and it produces every derived signal in §25.4–§25.6, plus the margin forecast — including a date. --demo reproduces Case Study 1's erosion series.
Ask the Case Study 1 question of one green indicator today. What number was thresholded to produce this? Then plot that number for ninety days. It needs no new instrumentation, which is what makes it the highest-value hour in this chapter.
Count last month's alerts and how many were acted on. Exercise 25.16. If you cannot determine the second number, that is the answer, and instrumenting acknowledgment turns the next review from two engineer-days into ninety minutes.

A note on what to be skeptical of

Any observability tool sold on the number of signals it collects. Case Study 2's platform collected 412 alerts a quarter and produced 47 actions. Collection is not the constraint; attention is, and a tool that increases the first without addressing the second makes things worse in a way its dashboard will report as an improvement.

And "you can't manage what you don't measure," which is half true and misleading. Kestrel measured its SLA for two years and could not manage it, because it measured the boolean. The useful version is narrower: you cannot manage a quantity you have converted to a boolean before storing it — and that conversion is usually invisible, because it happens at the point where somebody decided what to record.