Case Study 2: The Runbook That Only Worked for Its Author

"Step three said 'check the usual place.' I have no idea what the usual place is. It is 05:20 and the SLA is in forty minutes."

Executive Summary

Kestrel had eleven runbooks, all written by the engineers who had resolved the incidents they described, and all of them worked — for those engineers.

In August 2026 the platform's most experienced engineer took two weeks' leave. On the second morning a source arrived late, a Sev-2, and the engineer on the 05:00 window had never handled one.

The runbook took them from 05:12 to 06:34. The SLA was missed by 34 minutes. Nothing in the runbook was wrong. Six of its nine steps assumed context the reader did not have, and thirty-nine of the eighty-two minutes went on navigation — finding a dashboard, a list, a status page, a command — against twenty-three minutes of the build actually running.

The fix was a quarterly runbook drill — hand one to somebody who did not write it, on a quiet afternoon, and watch without helping. Every one of the eleven had at least one blocking step. The median was three.

Skills applied: runbook design (§26.5); the 05:00 window (§26.1); pre-authorized decisions (§26.8); and the fire-drill discipline from Chapter 19 Case Study 2, applied to documentation.

Background

The runbook, source-late.md, written in March after a similar incident. Nine steps, and here are the ones that failed:

# RUNBOOK: a source has not arrived

1. Check the source freshness dashboard.
2. If the source is late, check whether it is one of the flaky ones.
3. Check the usual place for the vendor's status.
4. If it looks like it will arrive soon, wait.
5. Otherwise start the partial build.
6. Let people know.
...

Read as its author, every line is clear. The freshness dashboard. The flaky ones — everyone knows which. The usual place — the vendor's status page, bookmarked. Soon — within the window. The partial build — the thing you run.

Read by someone who has not done this, six of those nine steps are questions.

The Problem

The timeline, reconstructed from the incident channel:

05:12  alert: source freshness, supplier_inventory
05:14  ack. opens source-late.md
05:16  step 1: which dashboard? Searches Slack. Finds a link. It 404s.
05:23  finds the freshness query in the dbt project instead. Confirms late.
05:26  step 2: "one of the flaky ones" — no list exists. Asks in channel.
       Nobody is awake.
05:31  step 3: "the usual place" — searches the vendor's name. Status page
       shows all green. (It is the wrong status page; the vendor has two.)
05:44  step 4: "arrive soon" — no definition. Decides to wait 15 minutes.
05:59  still absent. Step 5: "the partial build" — searches the repo for
       "partial". Finds nothing. It is a dbt selector in a Makefile target.
06:11  finds it. Runs it.
06:34  build completes. SLA missed by 34 minutes.
06:36  step 6: lets people know. First communication of the incident.

Thirty-nine of the eighty-two minutes were spent finding things — a dashboard, a list, a status page, a command. Twenty-three were the build actually running, fifteen were a deliberate wait, and five were acknowledging and confirming. Half the incident was navigation.

And step 6 was last, which §26.6 says explicitly it must not be. The engineer followed the runbook faithfully; the runbook's ordering was the defect.

⚠️ Failure Mode — a runbook is written at the moment its author has the most context they will ever have

This is the structural problem, and it explains why runbook quality does not improve with effort.

A runbook is written immediately after an incident, by the person who just resolved it, at the single point in time when everything about it is obvious. Every implicit reference is invisible to them, because "the usual place" is a complete instruction from inside that state.

The author cannot review their own runbook for this, in the same way you cannot proofread your own writing for a missing word. The context supplies itself.

Three classes of implicit reference, all present in source-late.md:

• The definite article. The dashboard, the flaky ones, the partial build. Every "the" in a runbook is a claim that the reader knows which one, and it is the fastest thing to grep for.
• A judgment with no criterion. "If it looks like it will arrive soon." Soon compared to what?
• A named thing that is not linked. A dashboard, a query, a command, a person — if it is not a link or a literal command, it is a search.

The only reliable detection is a reader who does not have the context, which is why this needs a drill rather than a review. Chapter 19 Case Study 2's fire drill, applied to documentation: a monitoring path never exercised end to end is a hypothesis, and so is a runbook.

The Analysis

The drill, run three weeks later across all eleven runbooks: an engineer who did not write it follows it on a quiet Tuesday afternoon against a simulated failure, with the author present and forbidden to speak.

runbook                     steps   blocking   worst
source-late.md                  9          6   "the usual place"
scheduler-down.md               7          4   "restart it"  (which process?)
grain-violation.md             11          2   "the tiebreaker" (which column?)
pool-exhausted.md               6          3   "the backfill pool" (name?)
daily-revenue-stale.md         12          1   step 5's retention query missing
cdc-slot-lag.md                 8          5   assumed psql access
quarantine-replay.md            5          4   the script's path had moved
schema-drift.md                 7          3   "the contract"  (where?)
export-failed.md                6          2   a vendor contact, no name
kafka-consumer-lag.md           9          4   "the runbook for that" (circular)
disk-full.md                    4          1   which volume
                              ----       ----
                                84         35

Thirty-five blocking steps across eleven runbooks. Median three. The best runbook had one, and it was the retention check §26.5 requires, which had been written but not tested and turned out to name a table that had been renamed.

Three findings beyond the count.

The circular reference. kafka-consumer-lag.md step 6 said "see the runbook for that," and the runbook for that did not exist. It had been planned.

The moved script. quarantine-replay.md gave a path that had been correct when written and had moved in a repository reorganization four months later. Nothing connects a runbook to the code it references, so the reorganization could not have known.

The permission assumption. cdc-slot-lag.md assumed psql access to the source replica. Two of the four engineers did not have it, and neither knew until the drill.

🔎 Read the Plan — the drill finds three classes of decay, and only one is about writing

The blocking steps sorted into three kinds, and the proportions surprised the team:

implicit context — "the usual place"	19	a writing problem
drift — a moved path, a renamed table, a dead link	11	a maintenance problem
missing access — a permission the reader lacks	5	an onboarding problem

Only 19 of 35 were about how the runbook was written. The other 16 were correct when written and had rotted, and no amount of careful authorship prevents either.

Which means the drill is not a proofreading exercise, and treating it as one misses half its value. Drift is found only by execution, and missing access is found only by a different person executing.

The corollary for frequency: implicit context is fixed once and stays fixed; drift accrues continuously. So the drill has to repeat, and quarterly is the interval Kestrel arrived at by noticing that eleven runbooks × three months produced about four findings — small enough to fix in an afternoon, frequent enough that nothing rots for a year.

The Decision

Five changes.

One: rewrite against a template, whose first four fields exist entirely to kill the three classes above:

# RUNBOOK: <symptom, as the alert words it>

**Owner:**       #data-eng          **Last drilled:** 2026-08-19
**Reader needs:** psql on replica-2, Airflow admin, #vendor-status access
**Prerequisites checked?** If you lack any of the above, escalate NOW —
                           do not start and discover it at step 6.

## 0. Is this actually the problem?
   <a command, and what its output should be>

"Reader needs" is the field that fixed the permission class, and it is at the top rather than at the step where the permission is needed, because discovering it at step 6 has already cost the window.

Two: every reference is a link or a literal command. The rule, enforced in review: no bare noun phrase for a thing you have to find.

❌  Check the usual place for the vendor's status.
✅  Check https://status.example-supplier.com/api  (NOT status.example-
    supplier.com, which is their marketing status page and is always green)

The parenthetical is not padding. It is the exact trap the engineer fell into at 05:31, written down.

Three: every judgment gets a criterion.

❌  If it looks like it will arrive soon, wait.
✅  Check the margin:  python platform/observability/health.py --job kestrel_daily
    Wait ONLY IF margin > 45 minutes AND the vendor's status page shows an
    active incident with an ETA. Otherwise go to step 5.

Four: communication moves to step 1. §26.6. Not step 6.

Five: the quarterly drill, with three rules that were argued about and all of which matter.

📐 Design Decision — the author must be present and silent

The three drill rules:

The reader has not written it and has not handled that incident. Obvious, and the constraint that makes an eleven-runbook rotation across four people awkward — by the third year everyone has seen everything. Kestrel's answer is that new joiners drill first, which also makes the drill double as onboarding.

The author is present and may not speak. This is the rule that produced the argument, and it is the one that matters. An author who can answer a question converts a blocking step into a conversation, and the step is then recorded as fine. Sitting silently while someone struggles for nine minutes with "the usual place" is uncomfortable, and the discomfort is the signal being generated.

The clock runs, and the time is recorded. "Thirty-nine of the eighty-two minutes were navigation" is the finding; a list of confusing steps is not. Time makes the cost legible to whoever has to fund the fix.

What was argued: several people felt the silence rule was theatrical, and that an author answering questions and noting them down would get the same information more efficiently.

The counter-argument won on evidence. The first drill was run both ways on two runbooks. With the author answering, the reader asked four questions and both runbooks were rated "mostly fine." Silent, the same two runbooks produced nine blocking steps — because a question asked and answered in two seconds does not feel like a defect, and at 05:20 there is nobody to ask.

What Happened

The next quarterly drill found four new blocking steps, all of them drift: two moved paths, a renamed table, and a Slack channel that had been archived. None was an authorship failure, which is the ⚠️ callout's proportion inverting once the writing problems are fixed once.

Three further findings:

Two runbooks were deleted. The incidents they described had been made impossible by other fixes — one by the dag_lint rule from Chapter 24, one by the pool configuration from §24.8. Nobody had connected the fix to the runbook, and a runbook for an impossible incident is a trap for a tired reader who will follow it.

One runbook was found to be missing entirely. The drill's coverage check compared runbooks against the paging list from §26.4, and one alert that pages had no runbook. It had been added six months earlier by someone who correctly created the alert and did not know the convention.

The drill became onboarding. New joiners now drill three runbooks in their first fortnight, which serves both purposes and — the team's own note — is the only onboarding exercise anyone has rated useful in an exit interview.

Lessons

1. A runbook is written at the moment its author has the most context they will ever have, which is why every implicit reference is invisible to them.

2. The author cannot review their own runbook for this. The context supplies itself.

3. Three classes of implicit reference: the definite article ("the usual place" — grep for "the"), a judgment with no criterion ("soon"), and a named thing that is not a link or a literal command.

4. Only 19 of 35 blocking steps were writing problems. Eleven were drift — correct when written, rotted since — and five were missing access. Careful authorship prevents neither.

5. Which is why it must be a drill and not a review: drift is found only by execution, and missing access only by a different person executing.

6. Put "Reader needs" at the top, not at the step where the permission is needed. Discovering it at step 6 has already cost the window.

7. The author must be present and silent. Run both ways, the same two runbooks produced four questions and "mostly fine" with the author answering, and nine blocking steps in silence — because a question answered in two seconds does not feel like a defect, and at 05:20 there is nobody to ask.

8. Record the time. "Thirty-nine of eighty-two minutes were navigation" funds a fix; a list of confusing steps does not.

9. Communication is step 1, not step 6, and a runbook that orders it last will be followed faithfully into a silent incident.

10. A runbook for an incident that can no longer happen is a trap. Two were deleted; nobody had connected the fix to the runbook.

11. Check runbook coverage against the paging list. One alert that pages had no runbook at all.

12. The drill doubles as onboarding, and is the only onboarding exercise Kestrel's leavers have called useful.

Questions for Discussion

1. The runbook was faithful, complete, and correct, and it cost 34 minutes of SLA. Where should the responsibility for that sit?

2. The silence rule was called theatrical and was validated by running the drill both ways. What other process disagreements could be settled that cheaply, and why are they usually not?

3. Eleven of 35 findings were drift. What would connect a runbook to the code and infrastructure it references, and what would that cost?

4. Two runbooks described impossible incidents. How would you detect that class systematically, rather than by drilling?

5. By year three, everyone on a four-person team has seen every incident. What happens to the drill then, and is "new joiners drill first" a sufficient answer?

6. The drill is quarterly because eleven runbooks produce about four findings in three months. How would you set the interval for a platform with sixty runbooks?

7. Chapter 19 Case Study 2's fire drill tested alert routes; this tests documentation. What else in your platform is a hypothesis that has never been exercised end to end?