Exercises: On-Call for Data

More than any other chapter, this one's exercises are arguments to have with colleagues rather than code to write. The paging list, the pre-authorization table, and "stale over wrong" are all decisions, and a decision made alone at 04:00 is the thing this chapter exists to prevent.

code/slo.py computes attainment, error budgets, burn rate, and the retrospective revision, with no dependencies.

Difficulty: ⭐ warm-up · ⭐⭐ standard · ⭐⭐⭐ deeper. Solutions: daggered (†) and odd-numbered problems are in appendices/answers-to-selected.md.

Part A — Warm-ups ⭐

26.1 † Give the four ways data on-call differs from service on-call, and one consequence of each.

26.2 Distinguish SLI, SLO, and SLA. Which is yours to set, and which do most internal data teams not have?

26.3 † Why is correctness scored retrospectively? What does that make it the only SLI able to measure?

26.4 Why is an SLO that cannot survive one bad day worse than useless? Give the test for a useful one.

26.5 † Define burn rate. Why does it need a short window, and what happens if you compute it over the full one?

26.6 State the 03:00 test. Why is a crashed job Sev-3 and a successful job Sev-1?

26.7 † Give the seven things a runbook must contain. Which two are usually missing?

26.8 What must be checked before any recovery that re-reads a source, and what is the general rule about retention?

26.9 † Why is communication step 1 rather than step 6? What is a complete message before you have a diagnosis?

26.10 Why is declaring a Sev-1 unilateral while resolving one is not?

Part B — Standard ⭐⭐

26.11 Run slo.py --demo and read both tables. Explain in one paragraph why correctness went from 100% to 88.89% without any new failure occurring.

26.12 † Define three SLIs for a platform you work on. For each, write the exact query that computes it. Then compute 90 days of history and report attainment.

26.13 Set an SLO for each from §26.2's method — observed attainment, then a small stretch. Apply the test: has it been close in the last two quarters? Report which of yours fails it.

26.14 † Take an SLO you set in 26.13 and compute the budget in days. Then state it in currency, the way Chapter 20 Case Study 2 does with the reconciliation tolerance.

26.15 Write your paging list. Be specific about what does not page overnight, and write down what that costs. Then show it to somebody who would be woken by it.

26.16 † Write the pre-authorization table from §26.8 for your team. Take the "stale over wrong" row to the business partner who owns the affected numbers and record their answer with their name on it.

26.17 Take an existing runbook and grep it for the definite article. Report how many "the"s are claims that the reader knows which one.

26.18 † Run a runbook drill. One runbook, one colleague who did not write it, the author present and silent, and a clock. Report the time, the blocking steps, and their classification into implicit context, drift, and missing access.

Part C — Deeper ⭐⭐⭐

26.19 §26.1 argues a four-person team should not attempt 24/7. Construct the strongest argument against — what does a team lose by accepting that a 23:00 catastrophe is found at 05:00? Price both sides for a system you know.

26.20 † Case Study 1 shows a retrospective measure creating an incentive not to investigate. Find another measure in your organization with that property — anything scored by a subsequent discovery — and design a counter-incentive for it.

26.21 Case Study 1 reports the SLO at a fixed six-month lag, derived from the discovery distribution. Compute your own distribution: for the last ten data defects, how long between occurrence and discovery? What lag does that imply?

26.22 † §26.11's three degraded modes call "partial and labelled" the dangerous one. Design the labelling for a revenue figure missing one channel, such that a reader scanning the number cannot miss it. Then test it on someone.

Part D — The Kestrel Platform ⭐⭐⭐

26.23 — Increment 26: the operations layer.

(a) Three SLIs, three SLOs set from observed attainment, and the reasoning in docs/slo.md.

(b) The paging list, short, with the two overnight exceptions and what is deliberately not covered.

(c) Every paging alert names a runbook; every runbook exists; check coverage against the paging list in both directions. Case Study 2 found one alert with no runbook and two runbooks for impossible incidents.

(d) Every runbook starts with "is this actually the problem?", carries a Reader needs block, puts communication first, and includes the retention check.

(e) The pre-authorization table, agreed with the business partner, with their name on it.

(f) Run the drill. One runbook, one colleague who did not write it, the author silent, a clock. Every runbook Kestrel tested this way had at least one step that only made sense to its author — and finding those costs an hour and a colleague rather than a night. This is the exercise that carries the chapter.

26.24 † Instrument the rotation: §26.12's five numbers plus nights interrupted per person per quarter. Run it for one quarter and report against the targets.

Expect the actionable rate to be the one that fails. Chapter 25 Case Study 2's 11% is typical, and a rotation is where that number is paid for in sleep rather than in attention.

Reflection

A. This chapter's two case studies are both about something that was correct and did not work — a measure that was correctly defined and punished investigation, and a runbook that was correct and unusable. What do they have in common? And where else in this book has "correct but does not work" appeared?

B. §26.1's claim is that most data incidents should not wake anyone. Argue against it for a system you know: what genuinely cannot wait until morning, and how many of those are there? If the answer is more than three, check whether each one is about the data or about a habit.