Chapter 26: On-Call for Data

"Nothing is on fire. The dashboard is just wrong, and it will stay wrong until somebody decides whether that is worse than it being empty."

Overview

Chapters 23, 24, and 25 built the detection. This chapter is about the person who receives it.

The material is adapted from site reliability engineering, and the adaptation matters more than the borrowing. Data on-call is a different job from service on-call, in four specific ways that §26.1 sets out, and copying an SRE rotation into a data team without that adjustment is the most common way data on-call becomes something people leave a job over.

The chapter's most contested claim is stated up front so you can argue with it throughout:

Most data incidents should not wake anyone. A stale dashboard at 03:00 is not an outage. The discipline is deciding — deliberately, in advance, and in writing — which small set of things is genuinely worth a person's night, and then defending that list against the reflex to add to it.

Chapter 10 §10.6 asked for "check the retention window covers your detection time" as a runbook item; that is §26.5. Chapter 19 Case Study 2 asked for a required data-impact field in the incident template; that is §26.9.

26.1 Four Ways Data On-Call Is Different

One: the failure is usually not urgent, but the decision is.

A down checkout service loses money every second. A stale daily_revenue loses nothing until somebody makes a decision on it — which, at Kestrel, is 06:15. So the incident has a deadline rather than a bleeding rate, and that changes what "respond immediately" means.

Two: the recovery is often waiting.

A service is restarted. A pipeline frequently cannot be fixed at 03:00 at all, because the source system has not produced the data, or a vendor's API is down, or the fix is a backfill that takes four hours. The on-call engineer's job in those cases is assessment and communication, not repair, and a rotation designed around repair will train people to do the wrong thing.

Three: the blast radius is measured in decisions, not requests.

"How many users were affected" has no data analogue. The right question is which decisions were made on the wrong number, which is usually unanswerable at 03:00 and sometimes unanswerable ever (Chapter 23 Case Study 2's two pricing decisions).

Four: the worst failures are silent.

Every case study in Part IV. A service that is down pages you; a pipeline that is wrong does not, which means a data on-call rotation that only responds to pages is responding to the least dangerous category of problem.

📐 Design Decision — the honest arithmetic of a four-person rotation

Kestrel has four data engineers (Chapter 1 §1.5). A 24/7 rotation across four people is one week in four, permanently, for as long as they work there.

The literature's guidance is a minimum of six, and eight is comfortable. Four is below the floor, and no amount of good runbooks changes that.

So the question is not "how do we run a rotation with four?" It is "what actually needs one?" Kestrel's answer, arrived at deliberately:

	Coverage
Nothing pages between 22:00 and 05:00	except the two exceptions below
05:00–07:00, a two-hour window before the SLA	a rota of four, one morning in four
Business hours	whoever is on support duty that day
Exception 1: a source with a retention shorter than 12 hours	pages, any time
Exception 2: a security or privacy event	pages, any time

The 05:00 window is the insight. The 6am SLA is what actually matters; a failure at 01:00 and a failure at 04:45 have identical consequences, and there is no value in learning about the first one six hours early. Alerting between 22:00 and 05:00 buys nothing and costs a night.

The two exceptions are not arbitrary, and each has a reason that survives being challenged: retention — Chapter 20's "incremental by necessity" case, where waiting until morning makes the loss permanent — and privacy, where the clock is legal rather than operational (Chapter 31).

What this costs: a genuine catastrophe at 23:00 is discovered at 05:00. Kestrel accepted that, wrote down why, and reviews it annually. A four-person team that pretends it can do 24/7 is choosing to do it badly, and the choice is usually made by not making it.

26.2 SLA, SLO, SLI

Three words used interchangeably and worth separating, because only one of them is yours to set.

SLI — indicator. The measurement. "Percentage of days gold.daily_revenue was current for the prior day by 06:00 America/New_York."

SLO — objective. Your internal target. "99.5% of days over a rolling 90."

SLA — agreement. An external commitment, usually with consequences. Most internal data teams do not have one and should not pretend to.

Three SLIs are worth having for a data platform, and the third is the one nobody defines:

Correctness is retrospective, and that is what makes it useful and unpopular. A day is scored correct until something proves otherwise, so the metric is revised backwards when Chapter 20's eight-month watermark bug is found — and the revision is the point. It is the only SLI that measures the failures this book is about.

⚠️ Failure Mode — an SLO nobody can miss

99.5% over 90 days is 0.45 days of budget. Kestrel's first SLO was 99.9%, which is 0.09 days — meaning a single missed morning blows the quarter.

That sounds strict and is worse than useless, for a reason worth stating precisely: an objective that cannot survive one bad day produces no decisions. Once it is blown, there is no difference between one failure and nine, so the remaining eighty-nine days carry no signal and the number stops being consulted.

Set the SLO from the observed distribution, then tighten. Kestrel's actual attainment was 99.2%; setting the objective at 99.5% made it a small, achievable stretch with a real budget — 0.45 days, which is enough that spending one is a decision rather than a catastrophe.

And the opposite failure is just as common: an SLO set so loose it is never at risk, which produces exactly as few decisions for the opposite reason.

The test for a useful SLO: has it been close in the last two quarters? If it has never been close and has never been blown, it is not measuring anything about your operations.

26.3 Error Budgets, for Data

The error budget is 1 − SLO, spent rather than avoided.

$$\text{99.5\% over 90 days} \Rightarrow 0.45\ \text{days of budget}$$

Its whole value is that it converts an argument into arithmetic. "Can we ship the risky migration this week?" becomes "we have 0.38 days of budget left this quarter" — which is a fact both sides can look at.

Three specifically-data uses, and the third is the one that changes behaviour:

Gate risky changes. Budget exhausted, and a migration waits. Chapter 37's material.

Justify reliability work. Budget burning fast, and the reliability backlog gets priority over features. This is the standard use.

Price a decision. Chapter 20 Case Study 2's version: a 0.5% reconciliation tolerance on $182.0M is $910,000 a year of permanently unexamined error. That is a correctness budget stated in currency, and it is a much better conversation than a percentage.

Burn rate is what makes a budget actionable in the moment:

$$\text{burn rate} = \frac{\text{budget consumed}}{\text{budget that should have been consumed by now}}$$

A burn rate of 1.0 exhausts the budget exactly at the period's end. Above 6 or so, you will be out within days — which is a page. Around 1.5, it is a conversation in a meeting. Same measurement, two thresholds, two urgencies, and no per-incident judgment required.

26.4 What Pages, and What Does Not

The single most valuable list a data team writes, and it should be shorter than anyone expects.

The 03:00 test, for every alert: if this fires at three in the morning, is there something a person can usefully do before business hours, and is the cost of not doing it greater than a night?

Almost everything fails that test, and the ones that fail it are not unimportant — they are important tomorrow.

Two things about that table.

Sev-1 is about publication, not failure. A job that crashed produced nothing and is Sev-3. A job that succeeded and wrote wrong numbers into a dashboard someone is looking at is Sev-1, and the inversion is deliberate — it is Chapter 23 §23.1's argument encoded in a severity scale.

Sev-2 pages into a window, not immediately. §26.1's arithmetic: a 01:00 failure and a 04:45 failure have the same consequence.

🎓 Interview Angle — "walk me through your on-call rotation"

A weak answer describes a schedule. A strong answer describes what is excluded and why, because that is where the thinking is.

"We page for two things overnight: a source whose retention is shorter than the time until morning, and a privacy event. Everything else waits for a 05:00 window, because our SLA is 06:00 and a failure at 01:00 has the same consequence as one at 04:45 — so alerting on it buys nothing and costs a night."

Then the part that shows judgment: "we accepted that a catastrophe at 23:00 is found at 05:00. We wrote down why, and we review it annually."

The follow-up is always some version of "what if something really bad happens?" and the answer is not to add coverage — it is that a four-person team cannot run 24/7 well, so the choice is between doing it badly and doing something smaller properly. An interviewer who disagrees is telling you something useful about the team.

26.5 The Runbook

A runbook is written for someone who is not you, at 04:00, who has never seen this failure. That sentence is the whole specification, and most runbooks fail it in the same way: they were written by the person who fixed it, immediately after fixing it, when the context was obvious.

Seven things a runbook must contain, and the first two are the ones usually missing:

1. How to tell whether this is actually the problem. A runbook that assumes a correct diagnosis is a runbook that will be applied to the wrong incident.

2. What to do if it is not. Where to look next, and who to escalate to.

3. The blast radius. Which tables, which dashboards, which people.

4. The immediate mitigation — including "do nothing, and here is why that is safe."

5. The decision boundary. What the on-call engineer may do alone. §26.8.

6. The recovery, with the exact command, including a dry-run.

7. What to check before recovering, and this is the one Chapter 10 §10.6 asked for:

⚠️ Failure Mode — recovering past your retention window

Before any recovery that depends on re-reading a source, check that the source still has the data. It is one query and it is skipped, because at 04:00 you are trying to fix something.

sql -- Every runbook whose recovery reads a source starts here. SELECT MIN(updated_at) FROM source.orders; -- 18 months. Fine. SELECT MIN(event_ts) FROM kafka_topic_offsets; -- 7 days. ← check

Three retention windows that expire at different speeds, and a recovery may need all three:

• The source system. Kestrel's OLTP retains 18 months; the CDC stream retains 7 days; a webhook replay window is 72 hours.
• Your lake's time travel. Chapter 10 §10.6. Delta's default vacuum retention is 7 days and Kestrel's was 30 — and Chapter 24 Case Study 1 needed 141, because the damage was to data written five months earlier.
• Your own backups, which is the one nobody has tested restoring.

The general rule Chapter 10 asked for, stated properly: your retention must cover the age of the data you might need to reconstruct, not the time it takes you to notice. Those are different numbers and the second is usually much smaller, which is why the mistake is easy.

Put the check in the runbook as step one, with the query, so it happens before an hour of work discovers it was impossible.

A runbook that has never been followed by someone other than its author is a hypothesis. Chapter 19 Case Study 2's fire drill, applied to documentation: hand it to a colleague during a non-incident and watch them use it. Every runbook Kestrel tested this way had at least one step that only made sense to its author.

26.6 Kestrel's 6am Runbook

The one that matters, walked through, because a concrete example teaches more than the specification above.

# RUNBOOK: daily_revenue not current by 05:30

## 0. Is this the problem?
   SELECT MAX(order_date) FROM gold.daily_revenue;
   Expect: yesterday. If it IS yesterday, this is not the problem —
   the alert is wrong. Go to runbooks/false-freshness-alert.md.

## 1. What is the deadline?
   06:00 America/New_York. Check the CURRENT margin:
   python platform/observability/health.py --job kestrel_daily
   Under 30 minutes: skip to step 4 and decide. Over 60: you have time.

## 2. Where did it stop?
   Airflow → kestrel_daily → the first non-success task.
   No tasks at all?  → runbooks/scheduler-down.md   (Ch. 24 CS2)
   Tasks queued?     → runbooks/pool-exhausted.md   (Ch. 24 §24.8)

## 3. Which of the four is it?
   (a) a source has not delivered        → step 3a
   (b) a task failed                     → step 3b
   (c) a test failed                     → step 3c   ← read this one
   (d) it is just slow                   → step 3d

   3c. A TEST failed, so `dbt build` stopped and downstream models were
       not rebuilt. THE MART HOLDS YESTERDAY'S CORRECT DATA. This is the
       system working. Do not "fix" it by re-running with --no-tests.
       → assess the test, then step 4.

## 4. THE DECISION: stale, or possibly wrong?
   You may decide this alone. See §26.8.
   - Default: STALE. Leave the mart. Post the notice from step 6.
   - Publish anyway ONLY IF: the failing test is understood AND the
     defect does not affect the published measures AND you write down
     why in the incident channel.

## 5. Before recovering: check retention.
   SELECT MIN(updated_at) FROM source.orders;      -- need >= the window
   SELECT MIN(event_ts) FROM bronze.events;
   If the window is not covered, STOP and escalate. Recovering is not
   possible and pretending otherwise wastes the morning.

## 6. Communicate. (Do this before step 7, not after.)
   Template: runbooks/templates/stale-mart.md
   Post to #kestrel-data-status by 06:00, whatever the state.
   The CEO opens the dashboard at 06:15.

## 7. Recover.
   airflow dags backfill kestrel_daily -s <date> -e <date>
   Then: python platform/quality/coverage.py  (expect 24/24)

## 8. Data impact.
   Fill in the required section of the incident record. §26.9.
   If nothing was published, say "none, because the test blocked the
   build" — that IS the answer, and it must be written.

🏭 From the Pipeline — step 6 is before step 7, and that is not an accident

Communicate before recovering, and the ordering is the most-argued line in this runbook.

The argument against: you do not know what to say yet. The argument that won: the recovery takes forty minutes and the deadline is in twenty, so a person who recovers first communicates after the CEO has already opened an empty dashboard.

And "we do not know yet" is a complete message:

text 05:41 daily_revenue is not current for 2026-08-14. We are investigating and expect to know more by 06:15. The 06:00 figures will be stale; yesterday's numbers on the dashboard are correct and are labelled with yesterday's date.

Forty seconds to write, and it converts a surprise into an expectation — which is most of what incident communication does. The stakeholder's actual need is not the fix; it is knowing whether to trust the number in front of them.

The general rule: the first communication goes out before the diagnosis, not after. A team that waits until it has an answer reliably communicates after the person has already been affected, and then spends the incident explaining a silence as well as a failure.

26.7 Running an Incident

Three roles, which one person holds simultaneously on a small team — and naming them anyway is what makes it possible to hand one off:

Incident commander. Decides. Does not fix. On a four-person team the IC usually is fixing, and the discipline is to notice when those conflict — an IC who is deep in a query is not deciding.

Communications. Stakeholder updates, on a cadence, whether or not there is news. Silence is interpreted as either "fine" or "catastrophic," and never as "still working."

Operations. The person with their hands on it.

The timeline is written as it happens, not reconstructed:

05:12  alert: daily_revenue freshness
05:14  ack, investigating.               margin 46m
05:19  kestrel_daily task 7 of 12 failed: dbt test grain on fct_order_item
05:26  1,204 duplicate (order_id, line_number). Ch. 18 §18.7 tiebreaker?
05:31  DECISION: stale over wrong. Mart holds 08-13. Notice posted.
05:44  cause: int_order_items_deduped changed yesterday, tiebreaker dropped
06:02  fix merged; rebuild started
06:41  rebuild complete; 24/24 coverage; mart current for 08-14
06:45  stakeholder update: resolved, 45 minutes late

Reconstructing a timeline afterwards produces a story; recording it produces evidence, and the difference shows in the postmortem's quality more than any other single factor.

When to stop is the part of incident response that is least often written down, and on a small team it is the part that costs the most, because the person deciding is the person who is tired.

An incident ends when three things are true, and all three must be stated, not assumed:

The data is correct, or the degraded state is labelled and communicated (§26.11). Not "the pipeline is fixed" — Chapter 19 Case Study 2's five-week tail is exactly this conflation.

The stakeholders know. A resolution nobody was told about is an incident that is still running for everyone outside the channel.

The follow-up is written, with an owner. Not done — written. An incident that ends with "we should probably look at the watermark" ends with nothing.

And the thing that must not end it: exhaustion. A Sev-1 that is still open at 07:00 is handed over, which is what §26.7's roles are for even on a team where one person holds all three. Handing off an open incident is a normal operation, and a team that has never done it will do it badly the first time it is unavoidable.

26.8 What the On-Call Engineer May Decide Alone

The most under-specified thing in most data teams, and it is what makes 04:00 slow — not the technical problem, but the twenty minutes spent deciding whether one is allowed to act.

Pre-authorize, in writing:

Two asymmetries in that table are deliberate.

Declaring is unilateral; resolving is not. Anyone may raise the severity of an incident; lowering it requires a second person, because the pressure at 05:50 is always toward "it is probably fine."

Reverting is always allowed; fixing forward frequently is not. A rollback restores a known state. A fix at 04:00 by a tired person is a change made under the worst conditions available, and Chapter 24 Case Study 1 is what one of those looks like.

📐 Design Decision — "stale over wrong" is a pre-made decision, not a judgment call

Chapter 19 §19.7 says stale beats wrong. The version that works at 04:00 is that it is already decided, and the engineer's job is to apply it rather than to relitigate it.

Why it must be pre-made: at 05:50 the pressure is entirely one-directional. The CEO opens the dashboard at 06:15; nobody is arguing for staleness; and the person under that pressure is alone and tired. Asking them to make a value judgment then is asking them to make it badly.

How it was made, and this is the part that makes it stick: the finance partner was asked. "Would you rather have yesterday's correct number at 06:00, or today's possibly-wrong one?" The answer took four seconds and it is now the policy, with their name on it.

Which means the on-call engineer is not making a call the business might disagree with — they are executing one the business already made, and can say so in the channel.

The escape hatch is narrow and explicit (step 4 of §26.6): publish anyway only if the failing test is understood, the defect provably does not touch the published measures, and the reasoning is written down at the time. Three conditions, all of them checkable afterwards.

26.9 The Postmortem

Blameless, which means the question is what the system permitted rather than who did it. Chapter 24 Case Study 1's engineer cleared four red squares using the UI's most prominent button; Chapter 23 Case Study 2's engineer muted an alert correctly at 04:12. Neither was careless, and a postmortem that concludes otherwise has stopped early.

Six required sections, and the fifth is the one Chapter 19 Case Study 2 asked for:

1. Summary. Three sentences. 2. Timeline. Recorded, not reconstructed. 3. What made this possible. Not "root cause" — causes, plural, and the conditions. 4. Detection. How long, by what, and what would have been faster.

5. Data impact — required.

Which tables, which date range, how many rows, and the exact command that repairs them. If the answer is "none," say why.

"If the answer is none, say why" is the part that does the work. It converts an omission into a claim somebody has to make, and Chapter 19 Case Study 2's five-week tail is exactly what happens when this section is absent: the pipeline was fixed, everyone believed the incident was over, and 1,082 fact rows kept the wrong customer key permanently.

6. Action items, each with an owner and a date — and Chapter 25 Case Study 2's constraint: an action item that adds an alert must name one to remove or downgrade, or state why the total should grow.

26.10 Rotation Design

The honest numbers, because most writing on this assumes a team twice the size of yours.

Six is the practical minimum for 24/7, giving one week in six. Four gives one in four, which people leave over. Kestrel's answer was to shrink the coverage rather than pretend (§26.1).

Four things that make a small rotation survivable:

A real handoff, in writing. Fifteen minutes, covering what is unresolved, what is fragile this week, and what was deferred. The handoff is where a small team's continuity lives, and a rotation without one makes every engineer start from zero.

Compensation or time back. Being available is work whether or not anything fires, and a team that does not acknowledge that loses people rather than arguments.

Toil is on the backlog, not absorbed. If the on-call engineer does the same manual thing twice, it becomes a ticket. Chapter 23 Case Study 1's quarantine replay is the shape: a manual step performed on demand, forever, because nobody ever counted it.

A hard rule that a Sev-1 ends the shift. Whoever handles a 03:00 incident does not also work a full day, and this needs to be a rule rather than a kindness, because nobody applies it to themselves.

26.11 When You Cannot Fix It Tonight

Frequently the right answer, and the one runbooks least often cover.

Three degraded modes, in preference order:

Stale and labelled. The mart holds yesterday's correct data and the dashboard says so, visibly. A "data as of" line is four hours of work and it converts every future staleness incident from a surprise into an observation.

Partial and labelled. Some sources loaded. This is the dangerous one — a revenue figure missing one channel looks like a revenue figure — and it requires the label to be on the number, not in a footer.

Empty and explained. Rare, and right when partial data would be actively misleading.

And the communication that goes with it, which is a template rather than a composition:

STATUS: daily_revenue is stale (last current 2026-08-13).
CAUSE:  a data quality test failed; we are not publishing numbers we
        cannot stand behind.
IMPACT: the 06:00 dashboard shows 08-13, labelled. Month-to-date is
        understated by roughly one day.
NEXT:   update by 09:00. Contact: #kestrel-data-status.

Four lines, and it does the entire job: what, why, what it means for you, and when you will hear again. The "roughly one day" is what stakeholders actually need — not the cause, but the size and direction of the error.

26.12 Measuring the Rotation Itself

A rotation is a system with users, and nobody measures it. The consequence is that it degrades the way every uninstrumented thing in this book degrades: gradually, plausibly, and until somebody leaves.

Five numbers, collected per shift, and none needs a tool beyond a spreadsheet:

The last row is the one people argue with, and it is worth defending: a rotation that never fires is not over-provisioned — it is a platform that works. Treating quiet shifts as slack is how a team talks itself into lowering the paging bar.

📏 Scale Note — the shape of a rotation that is failing

Two failing rotations look completely different and both are common.

Too loud. Six pages a shift, most of them not actionable, and the engineer arrives at the fourth one already primed to dismiss it. This is Chapter 25 Case Study 2's 89% arriving at 03:00 instead of in a channel, and the failure is that a real page will be handled by someone who has spent the night learning that pages do not matter.

Too quiet, for the wrong reason. Zero pages for four months — and the platform had three undetected incidents, because the failures are silent (§26.1) and nothing was watching for silence. A quiet rotation over a platform with no correctness SLI is indistinguishable from a working one, and the two are told apart only by §26.2's retrospective measure.

Which is why correctness is scored backwards. A quarter with zero pages and a defect found the following quarter was not a good quarter, and the only way to know that is to revise it.

The diagnostic pair: pages per shift, and Sev-1s discovered by someone outside the data team. Low on both is a working platform. Low on the first and high on the second is the dangerous one, and it is the one that reads as success on every dashboard a team is likely to have.

And one measurement about the humans, which Kestrel added after their first year and describes as the most useful of the six:

Nights interrupted, per person, per quarter. Not pages — nights. Two pages in one night is one interrupted night, and it is the night that has the cost. Kestrel's ceiling is two per person per quarter, and exceeding it triggers a review of the paging list rather than a rota change.

The distinction matters because page counts can improve while the human cost does not: consolidating four alerts into one is progress on the page count and no progress at all for the person who was woken.

💸 Cost Check — what a rotation costs, and what an incident costs

On-call is a budget line that nobody writes down, and writing it down changes the conversation about how many alerts are acceptable.

```text A ROTATION, four engineers, one quarter pages 142 outside business hours 38 nights interrupted, per person 9.5 median handling time, per page 34 min engineer-hours, total ~80 h at a loaded cost of ~$85/h ~$6,800 / quarter ~$27,200 / year

plus the cost nobody invoices: next-day productivity after an interrupted night retention risk, which is the expensive one ```

text ONE SILENT INCIDENT, from this book's own history Ch. 20 CS2: a missing lookback, eight months orders lost $110,560.14 engineering to find and fix ~40 h the restatement conversation unmeasured

Two comparisons follow, and they point in opposite directions.

A year of on-call costs about a quarter of one silent incident, which is the argument for investing in the rotation — and it is the argument people expect.

And 86% of those pages were not actionable (Exercise 25.16's 14%), which means roughly $23,000 a year of the rotation's cost bought nothing. That is the argument for fixing the alerts rather than for staffing them, and it is the one that is usually missing.

The number to put in front of a manager is not the cost. It is 9.5 interrupted nights per person per quarter, of which 1.3 were useful — because the cost figure invites a debate about hourly rates and the nights figure does not.

And the intervention that pays for itself fastest is not more people. It is the actionable-rate audit (Exercise 25.16): count last month's alerts, count how many anyone acted on, and delete or downgrade the rest. It costs an afternoon and it is the only change on this page that improves both numbers at once.

🧪 Try It — run the drill, this week

One hour, one runbook, one colleague, one clock. It is the highest-yield hour in Part V and almost nobody does it.

```text SETUP pick a runbook for an incident that has actually happened the READER is someone who did not write it the AUTHOR is present and SILENT -- this is the hard part start a clock

RULES the reader works only from the runbook and the systems it names when the reader is stuck, they say so out loud and the clock keeps running; the author may unblock them ONLY after 3 minutes, and what they say is the finding nobody is being tested. The RUNBOOK is being tested.

RECORD, per blocking step the step, the time lost, and the CLASS: implicit context -- unambiguous to the author, ambiguous to anyone drift -- the system changed and the runbook did not missing access -- the reader cannot perform the step at all ```

What every drill finds, in rough order of frequency:

A step that only makes sense to its author. "Restart the connector" — there are four. "Check the usual dashboard." This is the most common class and the cheapest to fix: name the thing, paste the exact command.

A missing "Reader needs" block. Credentials, VPN, a role, a saved query. The author has all of it and cannot see the gap, which is why this class is invisible without a drill.

And a step that is simply wrong now, because a setting was renamed or a service was replaced.

Then the number that makes the case for doing it again: the author's estimate against the clock. Kestrel's first drill was 29 minutes against an estimate of 8, and the gap — not the individual findings — is what persuaded the team to drill every runbook.

Do it during working hours, with coffee, and do not treat it as an exam. A drill that feels like an assessment produces a reader who bluffs, and a reader who bluffs finds nothing.

🔐 Privacy & Governance — an incident is when controls get bypassed

Every access control in this book is designed for normal operation, and an incident is the moment somebody needs data they cannot normally reach, at 3 a.m., with a deadline.

text what happens during an incident, at almost every company ───────────────────────────────────────────────────────────────────── a temporary grant, "just to look" and it is never revoked a query against production from a laptop outside every masking policy a screenshot of rows, pasted into the incident channel permanent, wide audience a copy of a table into a scratch schema to compare, and it stays an export to CSV to "look at it in Excel" Chapter 22's 🔐, under pressure

None of that is misconduct. All of it is a competent engineer under time pressure, and a control regime that has not planned for it will be bypassed by exactly the people it was designed with.

The answer is not stricter controls. It is a designed break-glass path, and the design has four parts:

A pre-authorised elevated role, requested by anyone on-call, granted immediately. No approval gate, because an approval gate at 3 a.m. is a bypass generator.

A hard expiry. Four hours, automatic, non-renewable without a second request. The expiry is what makes the immediate grant safe, and it is the part that is usually missing.

Loud, non-punitive logging. The grant posts to a channel when it is used. Non-punitive matters: if using break-glass is embarrassing, people will find a quieter route and you will lose the log entirely.

And a review in the postmortem, as a standing question rather than an accusation: "what did we access, is any of it still lying around, and can we delete it now?"

The scratch copy is the one that survives. A table copied for comparison at 4 a.m. is still there in six months, unclassified, containing production data — and it is the single most common finding in a first catalog audit (Chapter 30). A scratch_ schema with a 7-day drop policy costs nothing and removes the whole class.

🔁 Idempotency Check — the fix you apply at 3 a.m. will be applied twice

Incident remediation is the least idempotent code anybody writes, because it is written under time pressure, run manually, and frequently run again by the next person on the call.

text what gets run at 3 a.m., and what happens on the second run ───────────────────────────────────────────────────────────────────────── "re-run the failed task" fine IF the task is idempotent. That is what Exercise 24.23(e) tests. a manual INSERT of the missing rows duplicates, if the first run had partially succeeded "just run the backfill for yesterday" fine, if it is delete-insert. Catastrophic, if it is append. a one-off UPDATE to correct a value applied twice = applied twice. `SET x = x * 1.1` is the classic. restarting a consumer from `earliest` a full replay, which the sink had better be idempotent about (ch 15)

Row four is the one that produces a second incident. SET net_cents = net_cents + 4200 is a correction; run twice, it is a new error, and the person who runs it twice is usually the person who is not sure whether the first one worked.

Three rules for remediation, and they belong in the runbook rather than in anybody's head:

Write corrections as absolute values, never as deltas. SET net_cents = 128400 is safe to repeat; + 4200 is not.

Scope every manual statement by an interval, and make it delete-insert. The same shape as the pipeline's own write — because the pipeline's shape was chosen for exactly this reason.

And record what you ran, where the next person will see it, before you run it. "Applying X to interval Y at 03:14" in the incident channel. The most common cause of a doubled correction is two people responding to the same page, and one line prevents it.

The runbook's job here is not to describe the fix. It is to make the fix repeatable — because the person applying it is tired, and because they will not be sure whether the previous attempt landed.

🔎 Read the Plan — the first five minutes of an incident, in order

The most expensive minutes of an incident are the first five, because that is when everybody is forming a hypothesis and nobody has a fact.

```text 1 IS IT ACTUALLY BROKEN? 30 s the freshness check, and the margin. A pipeline that finished at 05:52 against a 06:00 deadline is LATE and not BROKEN, and the response is different.

2 WHAT IS THE SCOPE? 60 s which datasets, which consumers, which deadline. This is what goes in the message (§26.9) and it is available before any diagnosis.

3 SEND THE MESSAGE. 60 s before step 4. Not after.

4 DID ANYTHING CHANGE? 90 s the last deploy, the last config change, the last upstream release. Most incidents have one, and it is faster to check than to reason about.

5 IS IT US OR THEM? 90 s the source's own health, the vendor's status page, the heartbeat (§14.11). "The source produced nothing" is a CORRECT pipeline and somebody else's incident. ```

Step 1 is the one people skip and it is the cheapest. A meaningful fraction of "the pipeline is broken" pages are a pipeline that is running normally against a deadline nobody has recalculated — and treating a late run as a failure produces a rushed intervention that makes it later.

Step 3 before step 4 is the ordering this chapter argues for, and it feels wrong every time. The message costs sixty seconds and it changes the cost of the entire incident to everyone outside the rotation.

Step 4 has the highest hit rate of anything on the page. "What changed?" answers most incidents and it is a git log plus a deploy history — two commands, no reasoning required.

And step 5 is the one that ends investigations early in the good way. Establishing that the source produced nothing is not a diagnosis of your system; it is evidence that your system is correct, and it should be reported as such rather than as an absence of progress.

🧭 Version Note — SRE practice arrived in data, unevenly

SLIs, SLOs, and error budgets came from service reliability engineering, and the translation to data is genuinely useful in three places and misleading in one.

text concept translates? the data version ───────────────────────────────────────────────────────────────────── SLI YES freshness, completeness, correctness SLO YES and the budget is in DAYS, not requests error budget YES, and it a day of stale data has a currency is better value (§26.14); a failed request stated in usually does not currency burn rate YES and it needs a short window (§26.5) "9s of availability" NO a pipeline is not up or down; it is fresh, stale, or WRONG -- and the third has no analogue in a service blameless postmortem YES unchanged, and it matters more here because correctness is scored RETROSPECTIVELY (Exercise 26.20)

The availability row is the one to resist. "Our pipeline is 99.9% available" is a sentence with no content: a pipeline that ran successfully and produced a wrong number was available, and that is the failure this book is about.

What genuinely transferred and is under-used is the error budget stated in currency (Exercise 26.14). A budget of 0.9 days per quarter at ~$6,150 a stale day is ~$5,535 — a number a business partner can weigh, and it makes 99.0% versus 99.5% a purchase rather than a preference.

And one thing that is harder here than in services: the incident timeline starts before anyone knew there was an incident. A service outage begins when it begins; a correctness incident began weeks earlier and was discovered today, so "time to detect" is the metric that matters and it is the one service-derived tooling does not measure.

26.13 The Kestrel Platform

🧱 Kestrel Platform — Increment 26: the operations layer

text docs/ slo.md ← three SLIs, three SLOs, and the reasoning oncall.md ← the rotation, the windows, the exceptions decisions.md ← §26.8's pre-authorization table runbooks/ daily-revenue-stale.md ← §26.6, in full scheduler-down.md ← Ch. 24 CS2 pool-exhausted.md ← Ch. 24 §24.8 grain-violation.md ← Ch. 18 §18.7 source-late.md templates/stale-mart.md ← §26.11's four lines templates/postmortem.md ← §26.9, with the required data-impact field platform/observability/slo.py ← attainment and burn rate

Eight things this increment must get right:

1. Three SLIs with SLOs set from observed attainment, not from ambition. §26.2 — and check the test: has it been close in the last two quarters?
2. The paging list is written down and is short, with the two overnight exceptions and the reasoning for each. Including what is deliberately not covered.
3. Every alert that pages names its runbook, and every runbook exists.
4. Every runbook starts with "is this actually the problem?" and includes the retention check. §26.5.
5. The pre-authorization table is agreed with the business partner, particularly "stale over wrong," with their name on it.
6. The postmortem template's data-impact section is required, and "none, because" is an acceptable answer that must be written.
7. slo.py computes attainment and burn rate, and burn rate above 6 pages while above 1.5 goes to a meeting.
8. A quarterly runbook drill: hand one to someone who did not write it, during a quiet afternoon, and watch.
9. Five numbers per shift, plus nights interrupted per person per quarter, ceiling two. §26.12 — and the ceiling triggers a review of the paging list, not a rota change.

The exercise that matters is 26.23(f): run the drill. Every runbook Kestrel tested this way had at least one step that only made sense to its author, and finding those costs an hour and a colleague rather than a night.

26.14 Summary

Data on-call differs in four ways: the failure is rarely urgent but the decision is · recovery is often waiting, not repair · blast radius is measured in decisions, not requests · and the worst failures are silent, so a rotation that only answers pages is answering the least dangerous category.

📐 A four-person team cannot run 24/7 well. Kestrel pages for nothing between 22:00 and 05:00 except a source with retention shorter than the wait, and a privacy event — because a failure at 01:00 and one at 04:45 have identical consequences when the deadline is 06:00. They wrote down what that costs and review it annually. A team that pretends otherwise is choosing to do it badly.

SLI is the measurement, SLO your target, SLA an external commitment most internal teams do not have. Three SLIs: freshness, completeness, and correctness — and correctness is retrospective, revised backwards when a defect is found, which is what makes it the only one measuring the failures in this book.

⚠️ An SLO that cannot survive one bad day produces no decisions. Set it from observed attainment and tighten. The test: has it been close in the last two quarters?

An error budget converts an argument into arithmetic, and burn rate gives one measurement two urgencies — above 6 pages, around 1.5 is a meeting.

The 03:00 test: is there something a person can usefully do before business hours, and is not doing it worth more than a night? Sev-1 is about publication, not failure — a job that crashed is Sev-3; a job that succeeded and published wrong numbers is Sev-1.

A runbook is written for someone who is not you, at 04:00, who has never seen this. It must say how to tell whether this is the problem, what to do if it is not, and — ⚠️ — check the retention window before recovering. Your retention must cover the age of the data you might reconstruct, not the time it takes you to notice; Chapter 24 Case Study 1 needed 141 days.

🏭 Communicate before recovering, and "we do not know yet" is a complete message. The stakeholder's need is not the fix — it is knowing whether to trust the number in front of them.

Record the timeline as it happens. Reconstructing produces a story; recording produces evidence.

📐 Pre-authorize the decisions, so 04:00 is not spent deciding whether one may act. Declaring a Sev-1 is unilateral; resolving it is not, because the pressure at 05:50 is one-directional. Reverting is always allowed; fixing forward frequently is not. And "stale over wrong" is already decided — by the finance partner, with their name on it.

The postmortem's data-impact section is required, and "if the answer is none, say why" is the part that does the work — it converts an omission into a claim.

Rotation: a real written handoff · compensation · toil on the backlog after the second manual repetition · and a Sev-1 ends the shift, as a rule rather than a kindness.

📏 Measure the rotation itself: pages per shift, pages out of hours, time to acknowledge, actionable rate, and nights interrupted per person — because two pages in one night is one interrupted night, and consolidating alerts improves the page count while doing nothing for the person who was woken. A quiet rotation is a working platform, not a wasted slot — but a quiet rotation over a platform with no correctness SLI is indistinguishable from a broken one, and the diagnostic pair is pages per shift against Sev-1s discovered by someone outside the data team.

When you cannot fix it: stale and labelled, partial and labelled, or empty and explained — and partial is the dangerous one, because a revenue figure missing one channel looks like a revenue figure.

Chapter 27 moves upstream of all of this: how a change reaches production in the first place, and what has to be true before it does.

Key terms: SLI · SLO · SLA · error budget · burn rate · severity · the 03:00 test · runbook · incident commander · timeline · blameless postmortem · data impact · pre-authorization · degraded mode · handoff · toil · rotation