Affiliate disclosure

Book titles on this page link to Amazon. As an Amazon Associate, DataField.Dev earns from qualifying purchases — at no additional cost to you.

Further Reading: On-Call for Data

Sources are tagged Tier 1 (confident it exists, recommended without reservation) or Tier 2 (real and worth seeking, but confirm the current edition, version, or URL yourself).

Almost everything good on this subject was written for service reliability, and this chapter is largely an adaptation. Read the originals — they are better argued than any second-hand version, including this one — and read them holding §26.1's four differences in mind, because those are where the adaptation is load-bearing and where following the source material literally will produce a rotation that does not fit a data team.

The foundations

  • Google's Site Reliability Engineering (O'Reilly, 2016). Free online. Read Chapter 11, "Being On-Call," in full — it is short, it is the best thing written on the subject, and its material on operational load, the minimum viable rotation size, and the psychology of a paged engineer transfers intact. Chapters 4 (Service Level Objectives) and 6 (Monitoring) are the other two. §26.2's SLI/SLO/SLA distinction is theirs, and the argument for why an unachievable objective is worse than a loose one is made there more carefully than here. Tier 1.

  • Betsy Beyer et al., The Site Reliability Workbook (O'Reilly, 2018). Free online, and the practical companion. The SLO chapters and the alerting-on-SLOs chapter are the relevant ones, and the latter contains the multi-window burn-rate arithmetic that §26.3's single threshold is a simplification of. If you implement burn-rate alerting properly, implement theirs, not mine. Tier 1.

  • Sidney Dekker, The Field Guide to Understanding 'Human Error' (3rd ed., CRC Press, 2014). The book behind §26.9's "blameless means what the system permitted." Dekker's central move — that human error is a symptom of trouble deeper in the system rather than a cause — is what makes Chapter 24 Case Study 1's engineer (who clicked the UI's most prominent button) and Chapter 23 Case Study 2's engineer (who muted an alert correctly at 04:12) legible as system findings. Read it before writing your first postmortem, not after your first bad one. Tier 1.

  • Nathen Harvey and the PagerDuty incident response documentation. PagerDuty publishes its internal incident-response training openly, including the role definitions in §26.7 and the escalation material. It is written for services and is the clearest available description of running an incident with named roles, which is the part small data teams most often skip. Tier 2 — a maintained vendor site; check the current URL.

On runbooks

  • There is very little good writing on runbooks specifically, which is worth knowing so you do not spend an afternoon searching. The best available material is inside the SRE books above (on playbooks) and in the documentation literature generally.

  • Daniele Procida's "Diátaxis" framework for technical documentation. A runbook is what Diátaxis calls a how-to guide — goal-oriented, for someone who is trying to do something — and most bad runbooks are bad because they are secretly explanations or references. The framework's value here is diagnostic: if a step is telling you why, it belongs in a linked explanation, and Case Study 2's blocking steps are largely the reverse error. Tier 1 — diataxis.fr.

  • The checklist literature: Atul Gawande, The Checklist Manifesto (Metropolitan Books, 2009). Aviation and surgery, and directly applicable. The distinction between a DO-CONFIRM and a READ-DO checklist is the one worth taking: a runbook for an expert is DO-CONFIRM, a runbook for someone who has never seen the failure is READ-DO, and §26.5's specification is a READ-DO checklist — which is why Case Study 2's runbook, written as DO-CONFIRM by its author, failed for a reader. Tier 1.

On measurement and its incentives

  • Jerry Muller, The Tyranny of Metrics (Princeton, 2018). Case Study 1's problem — a measure that creates an incentive not to look — is the book's central subject, examined across medicine, education, policing, and business. Read it as a caution rather than a refutation: the answer is rarely to stop measuring, and Muller is better on the failure modes than on the remedies. Tier 1.

  • Goodhart's law, and Marilyn Strathern's formulation of it — "when a measure becomes a target, it ceases to be a good measure." Worth reading the actual sources rather than the slogan, because the mechanism matters: the measure degrades through the behaviour it induces, which is precisely what Case Study 1's credit-the-investigator change is designed to redirect rather than prevent. Tier 2 — the primary sources are short papers; the slogan is everywhere and the argument is not.

  • Anything careful on retrospective and revisable statistics. Economics has the best-developed practice here — GDP is revised for years after publication, and the profession has conventions for reporting revisions, vintages, and real-time data. Case Study 1's "publish both" and "report at a fixed lag" are standard practice in that field and are almost unknown in software. Search for "real-time data" and "data vintages" in the macroeconomic-forecasting literature; the transfer is direct and the terminology is worth borrowing. Tier 2 — an academic literature; you need the conventions, not the models.

On the human cost

  • The literature on shift work, sleep disruption, and cognitive performance. §26.12's "nights interrupted" metric exists because this research is unambiguous: a fragmented night degrades judgment more than a short one, and two pages in one night cost far more than two pages on separate nights. If you are asked to justify the metric, this is where the evidence is. Tier 2 — the occupational-health literature; the review articles are the way in.

  • Any honest account of on-call attrition. It is mostly in blog posts and conference talks rather than books, and the recurring finding is worth stating: people rarely leave because of one bad night. They leave because of a rotation with no end date, which is §26.1's arithmetic — one week in four, permanently — rather than any individual incident. Tier 2.

Practice

  • code/slo.py in this chapter. Attainment, error budget, burn rate over a short window, the retrospective revision, and §26.12's rotation metrics. No dependencies. --demo shows a quarter going from 100% to 88.89% correctness without any new failure occurring, which is Case Study 1 in one command.

  • Run one runbook drill this quarter. Exercise 26.18. One runbook, one colleague who did not write it, the author present and silent, and a clock. It costs an hour and finds things that otherwise cost a night.

  • Write your paging list, and show it to somebody who would be woken by it. Exercise 26.15. The conversation that produces is the whole exercise, and the part that is hard is writing down what is deliberately not covered.

A note on what to be skeptical of

Any on-call guidance that assumes a team of eight. Most of it does, silently, and the advice does not degrade gracefully: a rotation design that is humane at eight is punishing at four, and the usual recommendation — "hire more people" — is not available to the reader who most needs the advice. §26.1's answer is to shrink the coverage rather than the rotation, and it is the only answer this chapter is confident about.

And any suggestion that better runbooks make a small rotation viable. They make it less bad. Case Study 2's runbooks went from 35 blocking steps to 4 and the underlying arithmetic — four people, one week in four, forever — did not move at all.