Case Study 2: The Weekend Nothing Failed

"Fifty-six hours, and not one red square. There were no squares at all, and that is a much harder thing to notice."

Executive Summary

At 01:47 on Saturday 2026-07-11, Kestrel's Airflow metadata database filled its 100 GB volume. The scheduler crashed, restarted, crashed again, and entered a restart loop.

Nothing ran for 55.9 hours. Not the daily DAG, not the 56 hourly clickstream runs, not the freshness checks. And nothing failed, because a task cannot fail if it is never scheduled — so there were no alerts, no red squares, and no email.

The dashboard showed Friday's numbers, correctly labelled with Friday's date, which nobody read.

It was found at 09:40 on Monday by an analyst who noticed a "last updated" timestamp that was two days old.

The database was 95.9 GB, of which 83.9 GB was the xcom table — a kestrel_hourly task had been returning an 8.4 MB DataFrame twenty-four times a day for fourteen months.

Skills applied: XCom and what not to put in it (§24.9); how Airflow breaks (§24.12); separating orchestrator alerts from pipeline alerts (§24.12's 🏭 callout); the canary DAG.

Background

The task, added in May 2025 to a DAG that summarizes the clickstream hourly:

@task
def summarize_hour(data_interval_start=None) -> pd.DataFrame:
    df = pl.scan_parquet(...).filter(...).group_by(...).collect().to_pandas()
    return df                # ~62,000 rows, 11 columns

@task
def write_summary(df: pd.DataFrame):
    df.to_parquet(f"{LAKE}/gold/hourly_summary/dt={...}/")

It is clean, readable code, and it is exactly the pattern the TaskFlow API's documentation encourages: return a value, take a value, let Airflow wire them up.

What it does is serialize a 62,000-row DataFrame into the metadata database, twenty-four times a day, and never delete it:

$$8.4\ \text{MB} \times 24/\text{day} \times 426\ \text{days} = 83.9\ \text{GB}$$

Nothing removed it, because airflow db clean had never been run — it is not on by default, it is not in any getting-started guide, and it produces no symptom until it does.

The alerting. Container restarts were monitored and routed to #data-alerts, the same channel as DAG failures. The scheduler's restart alert fired 41 times over the weekend, which is the detail this case study turns on.

The Problem

Sat 01:47   metadata DB volume 100% full
Sat 01:47   scheduler crash → restart → crash → restart …
Sat 01:52   [#data-alerts] airflow-scheduler restarted (1)
Sat 02:31   [#data-alerts] airflow-scheduler restarted (2)
   ⋮                       …41 of these…
Mon 09:40   an analyst asks why "last updated" says Friday

No DAG failed. No task failed. No SLA-miss callback fired, because SLA misses are evaluated by the scheduler.

The dashboard was not empty. It showed Friday's figures against Friday's date, which is exactly what a correct dashboard shows when there is no newer data — and which reads as normal to anyone scanning a chart rather than a timestamp.

⚠️ Failure Mode — "nothing failed" and "nothing ran" produce identical alerting

Every alerting system in this book is built on events: a task failed, a test failed, a threshold was breached, a job exited non-zero. All of them are things that happen.

A scheduler that is not running produces no events at all, and an alerting system built on events cannot distinguish "a quiet, healthy weekend" from "the machine that produces events is dead."

The three failures with this shape, and they are the hardest three in operations:

• The scheduler is down. Nothing runs, nothing fails.
• The alert route is broken. Chapter 19 Case Study 2's archived Slack channel — things fail, nobody hears.
• The check was never invoked. Chapter 19 §19.8's freshness check — the control exists and is not running.

All three are absences, and every monitoring system is built to notice presences.

The fix is a heartbeat: something that must arrive, so its absence is an event.

```python

dags/canary.py -- the whole file.

@dag(dag_id="canary", schedule="/15 * * * ", catchup=False, start_date=pendulum.datetime(2026, 1, 1, tz="UTC")) def canary(): @task def beat(): requests.post(HEARTBEAT_URL, timeout=10) # an external monitor beat() canary() ```

Four lines, and it exercises the scheduler, the executor, a worker, and the metadata database end to end. An external monitor alerts if the beat is more than thirty minutes old.

The monitor must be outside Airflow. An alert Airflow sends you when Airflow is down is not an alert, and this is the single most common way teams get this wrong — the canary DAG exists, and its failure notification is routed through the thing that is broken.

The Analysis

Step 1: why did the scheduler crash? Thirty seconds, once anyone looked at the host:

$ df -h /var/lib/postgresql
Filesystem      Size  Used Avail Use%
/dev/nvme1n1    100G  100G     0 100%

Step 2: what is 100 GB?

SELECT relname, pg_size_pretty(pg_total_relation_size(relid)) AS size
  FROM pg_catalog.pg_statio_user_tables ORDER BY pg_total_relation_size(relid) DESC LIMIT 5;

relname                          size
xcom                             83 GB     ←
log                              6.1 GB
task_instance                    2.9 GB
rendered_task_instance_fields    1.8 GB
dag_run                          412 MB

xcom is 87% of the database.

SELECT dag_id, task_id, COUNT(*), pg_size_pretty(SUM(length(value))::bigint)
  FROM xcom GROUP BY 1,2 ORDER BY SUM(length(value)) DESC LIMIT 3;

dag_id            task_id           n        total
kestrel_hourly    summarize_hour    10,224   83 GB     ← 8.4 MB each
kestrel_daily     extract_orders    426      41 KB
kestrel_daily     list_files        426      12 KB

One task. 10,224 rows. 8.4 MB each.

Step 3: why did nobody see the 41 restart alerts?

They were in #data-alerts, alongside every DAG failure notification, and the team had learned — correctly — that an occasional scheduler restart is benign. A restart alert is not actionable on its own, and forty-one of them over a weekend read as the same non-event as one of them.

🏭 From the Pipeline — an alert that is usually benign trains people to ignore it

The restart alert was correct, fired, and was routed to a channel a human was in. It failed anyway, and the mechanism is worth being precise about because "add an alert" would not have helped.

A scheduler restart is genuinely benign most of the time — a deploy, a node rotation, a memory reclaim. So the team's response to seeing one was correct on every previous occasion, and the correctness of that response is exactly what made the forty-first indistinguishable from the first.

Three changes that make a benign-but-sometimes-serious signal usable:

• Alert on the RATE, not the event. One restart an hour is noise; five in an hour is an outage, and no human should be doing that arithmetic in a chat window at 02:00.
• Route it somewhere different. Platform health and pipeline failures go to different channels with different expectations, because they have different owners and different urgency (§24.12).
• Pair it with the heartbeat. A restart alert says something happened; the canary says nothing is happening. Together they are diagnostic; separately the first one is noise.

The general shape: an alert whose usual meaning is "ignore me" will be ignored when its meaning changes, and the fix is never to make people read more carefully. It is to change what fires.

The Decision

Six changes.

One: the task returns a path. §24.9, and it is a four-line diff:

@task
def summarize_hour(data_interval_start=None) -> str:
    df = ...
    path = f"{LAKE}/staging/hourly_summary/dt={data_interval_start}/part.parquet"
    df.write_parquet(path)
    return path                      # ← 74 bytes instead of 8.4 MB

@task
def write_summary(path: str):
    ...

Two: airflow db clean, scheduled. In kestrel_maintenance, weekly, retaining 90 days:

airflow db clean --clean-before-timestamp "{{ macros.ds_add(ds, -90) }}" --yes

Three: a disk alert on the metadata volume, at 70%. Not 90% — the gap between 70% and full is the time available to act, and at this database's growth rate 90% was about four days.

Four: the canary DAG, with an external monitor. §"The Problem"'s callout.

Five: separate routing. #data-platform for scheduler, database, and worker health; #data-alerts for DAG and task failures. Different channels, different owners, different expectations.

Six: dag_lint.py --rule fat-xcom in CI.

📐 Design Decision — a custom XCom backend was proposed and rejected

Airflow supports a custom XCom backend that transparently writes large values to S3 and stores only a reference. It would have fixed this without touching the DAG.

It was rejected, and the reasoning generalizes:

It removes the symptom and keeps the shape. A task returning an 8.4 MB DataFrame is still a task whose intermediate result is invisible, unqueryable, and tied to the orchestrator's lifecycle. The path version is better for reasons that have nothing to do with the database size — the intermediate result is a file you can inspect, re-use, and hand to a retry.

It makes the wrong thing easy. With a transparent backend, returning a DataFrame from a task becomes free, so more people do it, and the next constraint you hit is subtler.

And the honest counter-argument, which the team recorded: a custom backend is the right answer for an installation with hundreds of DAGs written by teams you do not control, where fixing every one is not available. Kestrel has eleven DAGs and four engineers. The fix that scales is not always the fix that is correct at your size, and choosing the enterprise answer at a small size is its own mistake.

The recovery. The volume was extended to 200 GB, db clean was run manually (taking the database to 9.4 GB), the scheduler came up, and Airflow immediately tried to schedule three days of missed runs.

catchup=False on kestrel_daily meant only the most recent interval ran — which was correct and also meant Saturday and Sunday were not processed. Those were backfilled deliberately (§24.4) after the platform was stable, in that order.

kestrel_hourly had catchup=True, correctly, because each hour is an independent window. 56 runs queued at once, and the backfill pool from §24.8 kept them from starving Monday's daily run. The pool was doing exactly what it was for, on the first occasion it mattered.

What Happened

Nothing was lost. Every source retained the window — the CDC stream at 18 months, the clickstream in the lake, the supplier feeds on SFTP — so the backfill was complete. That is luck, and the postmortem says so: a source with a 48-hour retention would have made 56 hours of outage into permanent data loss, and Kestrel has one such source (the payment processor's webhook replay window is 72 hours).

The canary caught the next one. Eleven weeks later a worker node pool failed to scale and tasks queued indefinitely — a different failure with the same signature, nothing running and nothing failing. The canary's beat stopped, the external monitor alerted at the 30-minute mark, and the incident was 34 minutes instead of 56 hours.

The audit found two more absence-shaped gaps:

The kestrel_quality DAG had no monitor of its own. It is the DAG that runs Chapter 19 §19.8's freshness checks — the thing that detects other things not running — and nothing detected it not running. It now has a heartbeat.

The dbt Cloud-equivalent export job ran on a separate schedule outside Airflow entirely, and its failure notification went to an individual's email rather than a channel. That person was on holiday.

Lessons

1. A task that returns a DataFrame writes it into the shared metadata database. 8.4 MB × 24/day × 426 days = 83.9 GB, and the failure presents as Airflow being broken.

2. airflow db clean is not on by default, is in no getting-started guide, and produces no symptom until it does. Schedule it.

3. "Nothing failed" and "nothing ran" produce identical alerting, because every alerting system is built on events and an absence is not one.

4. Three failures have that shape: a dead scheduler, a broken alert route, and a check that is never invoked. All three appear in this book, in different chapters, and all three are absences.

5. The fix is a heartbeat — four lines of DAG plus an external monitor. It exercises the scheduler, executor, worker, and database end to end.

6. The monitor must be outside the thing it monitors. An alert Airflow sends you when Airflow is down is not an alert.

7. An alert whose usual meaning is "ignore me" will be ignored when its meaning changes. Forty-one correct restart alerts fired into a channel where one restart is genuinely benign.

8. Alert on the rate, not the event, and do not make a human do that arithmetic at 02:00.

9. Set the disk alert at 70%, not 90%. The gap is the time available to act; here 90% was four days.

10. A custom XCom backend fixes the symptom and preserves the shape. The path version is better for reasons unrelated to database size. But the enterprise answer can be right at enterprise size — eleven DAGs and four engineers is not that.

11. catchup decided what recovery looked like. False on the daily DAG meant a deliberate backfill; True on the hourly meant 56 queued runs, held back by the pool from §24.8 doing its job on the first occasion it mattered.

12. Nothing was lost, and that was luck. One source has a 72-hour replay window against a 56-hour outage.

13. The DAG that detects other things not running had nothing detecting it not running.

Questions for Discussion

1. The task returning a DataFrame is exactly what the TaskFlow documentation encourages. What should the documentation say, and what would it cost in clarity?

2. The dashboard showed Friday's data with Friday's date — correct behaviour. What would make staleness visible to someone scanning a chart rather than reading a timestamp?

3. Forty-one alerts fired and were correctly ignored. Is rate-based alerting a complete fix, or does it move the problem?

4. The canary tests the scheduler, executor, worker, and database. What does it not test, and would you add a second canary?

5. The custom XCom backend was rejected at eleven DAGs and four engineers. At what size does the answer flip, and what would tell you that you had crossed it?

6. Recovery depended on every source retaining more than 56 hours, and one retains 72. How would you set a maximum tolerable outage from your sources' retention windows?

7. This chapter's two case studies are a failure where every task was green and a failure where there were no tasks at all. Which is harder to detect in your systems, and does your monitoring reflect that?