Chapter 26 — Key Takeaways (On-Call for Data)
The page to read before designing a rotation, and to hand to whoever is asked to join one.
The claim to argue with
Most data incidents should not wake anyone. A stale dashboard at 03:00 is not an outage. The discipline is deciding — deliberately, in advance, and in writing — which small set of things is worth a person's night, and then defending that list against the reflex to add to it.
Four differences
- The failure is rarely urgent; the decision is. An incident with a deadline, not a bleeding rate.
- Recovery is often waiting, not repair — so a rotation designed around repair trains the wrong behaviour.
- Blast radius is decisions, not requests. Usually unanswerable at 03:00, sometimes ever.
- The worst failures are silent — so a rotation that only answers pages is answering the least dangerous category.
📐 A four-person team cannot run 24/7 well. Six is the practical minimum. Kestrel pages for nothing between 22:00 and 05:00 except a source whose retention is shorter than the wait, and a privacy event — because with a 06:00 deadline a failure at 01:00 and one at 04:45 have identical consequences. They wrote down the cost and review it annually. A team that pretends otherwise is choosing to do it badly.
SLI · SLO · SLA
| SLI | the measurement | yours | | SLO | your internal target | yours | | SLA | an external commitment | most internal teams do not have one |
| Freshness | % of days current by the deadline | | Completeness | % of days reconciled within tolerance | | Correctness | % with no Sev-1/2 defect discovered later ← retrospective |
Correctness is the only SLI that measures the failures in this book, precisely because it is revised backwards.
⚠️ An SLO that cannot survive one bad day produces no decisions. 99.9% over 90 days is 0.09 days; once blown, one failure and nine are identical. Set it from observed attainment, then stretch. The test: has it been close in the last two quarters? Over 90 days one bad day is 1.11 points, so an objective between two achievable values is worth noticing.
Error budgets
$$99.5\% \text{ over } 90 \text{ days} \Rightarrow 0.45 \text{ days of budget}$$
It converts an argument into arithmetic. Gate risky changes · justify reliability work · price a decision (0.5% of $182.0M = $910,000/year of unexamined error).
Burn rate = recent failure rate ÷ the rate the SLO permits. Above ~6 pages; ~1.5 is a meeting. It must be computed over a SHORT window — over the full window it collapses to spent/budget and cannot exceed 1 without the budget already being blown.
What pages
The 03:00 test: is there something a person can usefully do before business hours, and is not doing it worth more than a night? Almost everything fails it — and the things that fail it are not unimportant, they are important tomorrow.
| Sev-1 | wrong data published and in use | page, any time | | Sev-2 | the SLA will be missed | page into the 05:00 window | | Sev-3 | a pipeline failed, SLA not yet at risk | ticket | | Sev-4 | a trend | the dashboard |
Sev-1 is about publication, not failure. A crashed job produced nothing — Sev-3. A successful job that wrote wrong numbers is Sev-1.
The runbook
Written for someone who is not you, at 04:00, who has never seen this failure.
Seven things, and the first two are what is usually missing: how to tell whether this is actually the problem · what to do if it is not · blast radius · mitigation (including "do nothing, and why that is safe") · the decision boundary · the recovery command, with a dry run · and the retention check.
⚠️ Before any recovery that re-reads a source, check three windows: the source, your time travel, your backups. Your retention must cover the age of the data you might reconstruct, not the time it takes you to notice. Chapter 24 Case Study 1 needed 141 days.
🏭 Communicate before recovering. The recovery may outlast the deadline, and "we do not know yet" is a complete message. The stakeholder's need is not the fix — it is knowing whether to trust the number in front of them.
Record the timeline as it happens. Reconstructing produces a story; recording produces evidence.
An incident ends when three things are stated: the data is correct or the degraded state is labelled · the stakeholders know · the follow-up is written, with an owner. Exhaustion is not one of them — hand off an open Sev-1.
Pre-authorization
📐 Decide in advance so 04:00 is not spent deciding whether one may act.
| MAY, alone | MUST escalate |
|---|---|
| leave a mart stale + post a notice | publish data a test failed on |
| re-run a task or DAG | change business logic to "unblock" |
| a documented backfill in its window | a backfill outside it |
| pause a DAG · roll back a deploy | delete or truncate anything · disable a check |
| declare a Sev-1 | resolve a Sev-1 |
Declaring is unilateral; resolving is not — the pressure at 05:50 is one-directional. Reverting is always allowed; fixing forward frequently is not.
"Stale over wrong" is pre-decided, by the business partner, with their name on it, so the engineer is executing a decision the business already made.
The postmortem
Blameless = what the system permitted. Six sections, and the fifth is required:
Data impact. Which tables, which range, how many rows, and the command that repairs them. If the answer is "none," say why.
That last clause is what does the work — it converts an omission into a claim, and Chapter 19 Case Study 2's five-week tail is what its absence looks like.
An action item that adds an alert must name one to remove or downgrade.
The rotation
A written handoff · compensation · toil on the backlog after the second manual repetition · a Sev-1 ends the shift, as a rule rather than a kindness.
📏 Measure the rotation: pages/shift < 2 · out-of-hours < 1 · ack < 10 min · actionable > 70% · quiet shifts > 50% · nights interrupted per person per quarter < 2.
Two pages in one night is one interrupted night — consolidating alerts improves the page count and does nothing for the person who was woken. The diagnostic pair: pages per shift against Sev-1s discovered by someone outside the data team.
Degraded modes
Stale and labelled · partial and labelled ← the dangerous one · empty and explained.
A revenue figure missing one channel looks like a revenue figure, so the label goes on the number, not in a footer. And the four-line status template does the whole job: what, why, what it means for you, when you will hear again — including "understated by roughly one day," which is the part stakeholders actually need.
The two case studies
The quarter that got worse after it ended. A retrospective measure punishes investigation, and quietly: not a refusal, a deprioritization. Scoring at discovery destroys it (maximized by finding nothing); freezing it destroys it differently (a record of what was believed). Publish both — a frozen report marked as known at, and a live number with a change log — and log revisions by investigator, so "caused three revisions this quarter" is a positive line.
A revisable measure is not a time series until it stabilizes. Report at a fixed lag derived from your own discovery distribution. And an objective set from an inflated attainment figure is inflated.
The runbook that only worked for its author. Written at the moment its author has the most context they will ever have, so every implicit reference is invisible to them and they cannot review their own. Only 19 of 35 blocking steps were writing problems — eleven were drift, five were missing access. Which is why it must be a drill, not a review.
The author is present and silent. Run both ways, the same two runbooks gave four questions and "mostly fine" with the author answering, and nine blocking steps in silence — because a question answered in two seconds does not feel like a defect, and at 05:20 there is nobody to ask.