Self-Assessment Quiz: Monitoring, Alerting, and Observability

Twenty questions. Aim for 16 or more. Questions 7, 8, and 16 are the three that separate a platform that finds out from one that is told.

Question 1

The question monitoring must answer has three parts. The one most teams lack is:

• A. Is the data current?
• B. Is the data right?
• C. Will it still be current tomorrow?
• D. Did the job run?

Question 2

The four golden signals are wrong for a batch pipeline because:

• A. They are too old
• B. All four are properties of request handling, and a batch pipeline handles no requests
• C. They need a service mesh
• D. They only apply to HTTP

Question 3

The four signals for data are freshness, volume, distribution, and:

• A. Latency
• B. Schema / lineage
• C. Saturation
• D. Cost

Question 4

A test and a metric measuring the same quantity should have:

• A. The same threshold
• B. Different thresholds — the test protects the pipeline, the metric informs a person
• C. Only the test
• D. Only the metric

Question 5

Labelling a metric with order_id:

• A. Improves debuggability
• B. Creates one time series per order — 2.4 million a year — and takes down your metrics backend
• C. Is required for tracing
• D. Is free

Question 6

Structured logging's advantage is that:

• A. It is smaller
• B. "Show me every load under 5,000 rows this quarter" is a filter rather than a regular expression
• C. It is faster to write
• D. It replaces metrics

Question 7

Alerting on duration should use:

• A. An absolute threshold above the worst legitimate run
• B. The ratio to a trailing median
• C. The mean of all runs
• D. The scheduler's SLA feature

Question 8

A ratio below 0.5 usually means:

• A. Good news — the job got faster
• B. Not good news — the run probably processed less than it should
• C. A clock error
• D. Nothing

Question 9

The four numbers worth publishing per run are duration, peak RSS, rows out, and:

• A. Exit code
• B. finished_at
• C. CPU seconds
• D. Log volume

Question 10

The margin is:

• A. Whether the SLA was met
• B. The headroom between finishing and the deadline, as a number
• C. The error budget
• D. The retry count

Question 11

Kestrel's SLA compliance over the two years its margin fell from 4h41m to 41m was:

• A. Falling
• B. 100%
• C. 94%
• D. Not measured

Question 12

An Airflow task-level SLA miss did not fire during the backfill starvation because:

• A. SLAs were off
• B. No individual task was slow — the DAG was slow because it started late
• C. The threshold was wrong
• D. Backfills are exempt

Question 13

Alert on memory at 60% of the container limit rather than 90% because:

• A. 90% is unreachable
• B. The gap is the time you have to act, and it is where a busy Monday lives
• C. Containers round up
• D. 60% is the OOM threshold

Question 14

Cost should be alerted on:

• A. The monthly total
• B. The ratio to a trailing median, per job per run
• C. An absolute dollar threshold
• D. Only at quarter end

Question 15

An alert is:

• A. Any notification
• B. A request that a person do something
• C. A log line with high severity
• D. A failed test

Question 16

In Kestrel's alert review, the proportion of 412 alerts that produced no action was:

• A. 11%
• B. 36%
• C. 53%
• D. 89%

Question 17

A rule with a 0% acknowledgment rate:

• A. Should fire more often
• B. Provides the assurance of a rule that does not exist, even when every firing is correct
• C. Is well tuned
• D. Should be deleted immediately

Question 18

"We need an alert for this" as a postmortem action item is:

• A. Always right
• B. A ratchet — every incident adds one and nothing removes one
• C. Too slow
• D. A routing decision

Question 19

A heartbeat proves:

• A. Everything is healthy
• B. Only its own chain — a trivial canary can pass while the worker pool cannot scale
• C. That data is correct
• D. The scheduler's version

Question 20

Run records should be retained:

• A. 30 days
• B. Forever — 1.44 MB a year, and they cannot be recreated
• C. As long as logs
• D. One year

Answer Key

1. C — §25.1. And it is where every incident in Chapters 21, 22, and 24 was visible months ahead.

2. B — §25.2.

3. B — §25.2. The same four as Chapter 23's assertions, surfaced continuously rather than as a pass/fail at build time.

4. B — §25.2. A 9,000-row day passes a floor set at 3,000 and is still a 49% drop somebody should see.

5. B — §25.3. Labels must be low-cardinality: DAG, task, table, source, status.

6. B — §25.3.

7. B — §25.4. Scale-free, outlier-robust, and it fires on the day of the change.

8. B — §25.4. Chapter 19 Case Study 1's missing day.

9. B — §25.4's 🔎 callout. Four raw numbers, four derived signals, and it needs a table with five columns rather than a platform.

10. B — §25.5.

11. B — Case Study 1. Every month on that chart is a 100% compliance figure, which is the case study's whole point.

12. B — §25.5 and Chapter 24 §24.8. Put the SLA on the DAG's completion time.

13. B — §25.6.

14. B — §25.6. A job going from $0.06 to $0.18 is the same defect as $600 to $1,800, and only one of them appears on a bill.

15. B — §25.8. If nothing should be done, it is a metric.

16. D — Case Study 2. 53% acknowledged with no action plus 36% never acknowledged. And two of the 36% were real.

17. B — Case Study 2's 🔎 callout. Chapter 23's "measure whether a control is operating, not whether it exists," from a fourth direction.

18. B — Case Study 2. Pair it: add an alert and name one to remove or downgrade, or state why the total should grow.

19. B — §25.9. Hence a second canary that requests a realistic resource envelope.

20. B — §25.11. Logs can be re-derived by a rerun and metrics re-aggregated; the duration of a run in March 2025 exists in one place or in none.

Topic map