Exercises: Monitoring, Alerting, and Observability

More of this chapter's exercises are about your systems than any other in the book, because the central claim — that the number you need is usually already being collected — can only be tested against something real.

code/health.py needs no dependencies and computes every derived signal in §25.4, §25.5, and §25.6 from a five-column run history.

Difficulty: ⭐ warm-up · ⭐⭐ standard · ⭐⭐⭐ deeper. Solutions: daggered (†) and odd-numbered problems are in appendices/answers-to-selected.md.

Part A — Warm-ups ⭐

25.1 † State the three-part question monitoring must answer. Which part do most teams not have?

25.2 Why are the four golden signals wrong for a batch pipeline? Give the four that replace them.

25.3 † A test and a metric measuring the same quantity want different thresholds. Explain, using the volume floor at 3,000 against a median of 17,753.

25.4 What makes a metric label dangerous? Give three labels that are safe and three that are not.

25.5 † Why alert on a duration ratio rather than a threshold? Give the four properties that make it work.

25.6 Why is r < 0.5 on that ratio usually not good news?

25.7 † Define the margin. Why can a 100% SLA compliance rate coexist with an outage four months away?

25.8 Why alert at 60% of a container limit rather than 90%?

25.9 † Give §25.8's four requirements for an alert. Which one do most alerts fail?

25.10 Name the three absence-shaped failures in this book and what a heartbeat proves.

Part B — Standard ⭐⭐

25.11 Run health.py --demo and read the output. Then change MEMORY_FRACTION to 0.90 and rerun. Report which signals disappear, and say which incident in this book each of them corresponds to.

25.12 † Export a run history from your own scheduler into the five-column format and run health.py against it. Report the signals. If you cannot export it, that is the finding — say what is missing.

25.13 Compute the margin to a deadline for a job you own, for as far back as you have data. Plot it. Is the slope negative? What is the date?

25.14 † Take a green indicator in your systems and answer §"The Analysis"'s question from Case Study 1: what number was thresholded to produce this? Plot that number for 90 days and report what you see.

25.15 Convert one unstructured log line in your codebase to a structured event. Then write the query it makes possible that a regular expression could not.

25.16 † Count last month's alerts. For each, determine whether anyone acknowledged it and whether anyone acted. Report the three numbers. Compare to Case Study 2's 11% / 53% / 36%.

25.17 Take the ten noisiest alerts from 25.16 and apply §25.8's test: is there something a person should do, now? Reclassify each as alert, dashboard row, weekly digest, or delete.

25.18 † Run §25.12's consumer query against your warehouse over a 400-day window. Report how many tables were read by nobody — and investigate three of them before deleting any.

Part C — Deeper ⭐⭐⭐

25.19 Case Study 1 decomposed a two-year margin slope into six changes by matching steps against a deploy log. Do this for a trend in your own systems — cost, duration, or row count — and report how many steps you can attribute.

25.20 † §25.9 says a heartbeat proves only its own chain, and Kestrel's trivial canary passed while the worker pool could not scale. Enumerate the chains in your platform and design the minimum set of heartbeats that covers all of them. State what remains uncovered.

25.21 Design the retention policy for your own telemetry using §25.11's asymmetry. Price it, in both directions: what does keeping cost, and what question becomes unanswerable if you do not?

25.22 † §25.6 says to attribute cost per job per run. Build it for one job — from QUERY_HISTORY, INFORMATION_SCHEMA.JOBS, or executor-seconds — and alert on the ratio rather than the amount. Report what you find in the first week.

Part D — The Kestrel Platform ⭐⭐⭐

25.23 — Increment 25: the health signal.

(a) Every scheduled job writes a run record with the five columns. Nothing else in this increment works without it, so do it first.

(b) Compute Kestrel's margin for the last 90 days and extrapolate it to zero. If the answer is a date inside the next two quarters, that is the most important number your platform produces and nothing else on the dashboard competes with it. This is the exercise that carries the chapter.

(c) Duration and cost alert on the ratio to a trailing 14-run median, above 2.0 and below 0.5. Peak memory over container limit, at 0.60.

(d) Two canaries — trivial and resource-sized — with an external monitor.

(e) Build the one dashboard from §25.7. Today and trend side by side, fitting on a screen. Then delete the others, and record which ones and who objected.

(f) Every alert names a runbook and an owner. Route by who fixes it, into four destinations.

25.24 † Add the margin-impact line to the pull-request template, measured on staging. Then apply it retroactively to the last ten merged changes and report the total.

Expect the total to be larger than anyone would have guessed. Case Study 1's sixteen sensors were sixteen individually unarguable pull requests adding 55 minutes between them, and the aggregate was invisible at every point where it could have been questioned.

Reflection

A. Case Study 1's metric had been collected for two years and never queried; Case Study 2's alerting had never been measured at all. Both are failures of the same kind — instrumentation without a consumer. Where else in this book has that shape appeared, and what does the count suggest?

B. §25.7 argues for one dashboard and against the other nine. Count the dashboards in your organization and estimate how many are opened weekly. Then ask what the ones nobody opens were for — and whether the answer is a decision that stopped being made, or a decision that was never made.