Chapter 24 — Key Takeaways (Apache Airflow)

The page to read before writing a DAG, and again before clearing a task from last spring.

When, and what it costs

Cron is right until the fourth dependent step, or until a step's duration varies enough that a fixed offset is a guess.

📐 The honest cost: a scheduler, a metadata database, a web server, workers — Kestrel's managed Airflow is $310/month before any task runs — plus a new failure surface (the orchestrator down looks identical to every pipeline broken) and a place for logic to accumulate where it does not belong.

Not "you need Airflow because you are a data team." Rather: "you have nine tasks with a dependency graph, and it lives in five cron offsets and one person's head."

The model

DAG (a file, parsed repeatedly) · task instance (what succeeds, fails, retries) · operator (what it does) · executor (where it runs).

TaskFlow derives the graph from the call structure — the same idea as dbt's ref() and as datasets: the graph comes from the code rather than beside it.

⚠️ A DAG file executes on every parse — every 30 seconds, on the scheduler. Import, define, nothing else. time python your_dag.py < 1 second. Module-level work makes the scheduler do your I/O, delays scheduling for every DAG, and changes the graph without a deploy.

The date model

data_interval_start   data_interval_end   runs
2026-03-17 00:00      2026-03-18 00:00    2026-03-18 00:00
└─── the data it is about ───┘            └── when ──┘

A run labelled 2026-03-17 executes on the 18th and produces the 17th's numbers. Confusing, and correct: a run that processes a day cannot start until the day is over.

⚠️ datetime.now() in a task is the most consequential line in this chapter. A retry processes a different window; a rerun writes today's data into last March's partition; and every idempotency guarantee from Chapter 20 is defeated. data_interval_start/_end are stable across every retry and rerun, forever. Grep for now(), today(), utcnow(). Every hit is a bug or needs a comment.

Idempotency and recovery

catchup=True with a two-year start_date schedules 730 runs on deploy. Right for independent windows; wrong for anything maintaining current state.

🔁 Retries make non-idempotency a certainty, not a risk. Three hazards: a write that succeeded and lost its acknowledgment (the normal failure of a network) · a partial completion · a human clearing a task from a month ago.

airflow tasks test dag task 2026-03-17     # then again, then DIFF the target

A task should be the smallest unit you would want to retry independently — the unit of failure is the unit of design. Task groups organize without adding boundaries; every boundary is a scheduling round trip.

Waiting

💸 16 sensors × 41 min = 656 slot-minutes a night out of 32 slots. The symptom was the DAG finishing at 04:50, and the sensors showed as "running." deferrable=True is one keyword.

⚠️ ExternalTaskSensor with no execution_delta looks for a run at the same logical date — none exists if the schedules differ, and it times out reporting an upstream failure that did not happen. An offset between two cron expressions is a dependency waiting to be wrong.

Concurrency

| parallelism | whole installation | | max_active_runs | concurrent runs of one DAG | | max_active_tasks | tasks within one DAG | | pool slots | tasks sharing a named resource, across DAGs |

⚠️ A backfill in the default 128-slot pool starved the nightly run and missed the 6am SLA by 41 minutes — with every task green. Fixes: a dedicated backfill pool (4 of 32) · max_active_runs=1 · priority weight · and an SLA on the DAG's completion time, because no individual task was slow.

XCom

⚠️ XCom carries references, not data. A DataFrame goes into the shared metadata database and the failure presents as Airflow being broken. Anything you would not put in a log line does not go in XCom.

Write to storage, pass the path — which is also what makes the task independently retryable.

What belongs where

In Airflow: when things run · what depends on what · retries and alerting · concurrency · the audit trail. Not in Airflow: business logic · transformation SQL · validation rules · anything you would unit-test.

The test: could this run correctly outside Airflow? A @task should be four lines calling an importable function — which makes it testable with pytest, runnable from a laptop, and portable.

Testing, in four tiers

1. imports, and parses in under a second
2. structural policy — retries, owners, max_active_runs, no wall clock
3. the logic, as ordinary pytest — the tier §24.10 exists to make possible
4. airflow tasks test, twice

🧪 Four commands audit a DAG you already have: list-import-errors · time python dag.py · grep -rn "now()" · tasks test twice. Command 3 finds something in most repositories.

How Airflow itself breaks

🏭 Rows one and two produce no failure notification at all, so they must be monitored from outside Airflow. A four-line canary DAG plus an external heartbeat monitor covers the scheduler, executor, worker, and database end to end.

The two case studies

The rerun that overwrote March. Four red squares from a handled incident sat eleven weeks; someone cleared them; datetime.now() wrote 71,012 August rows into March partitions and the overwrite deleted March's. All six of Chapter 23's assertions passed — they check rows against themselves, and none asks whether a row is in the right partition.

Assert the relationship between a record's content and its location, with a tolerance matching the lookback. And add _ingested_at, _source, and run_id to every bronze row — the run identifier is the one people omit and the most useful.

An unresolved state in an operational tool decays into noise, and noise is where the next incident hides. Red squares, mutes, known-issues entries: a maximum age and a required explanation.

The weekend nothing failed. The metadata DB filled (83.9 GB of it xcom), the scheduler crash-looped, and nothing ran for 55.9 hours with zero failures reported. 41 correct restart alerts fired into a channel where a restart is genuinely benign.

"Nothing failed" and "nothing ran" produce identical alerting, because every alerting system is built on events and an absence is not one. Three failures share that shape — a dead scheduler, a broken alert route, a check never invoked — and all three appear in this book.

Alert on the rate, not the event. Set the disk alert at 70%, not 90% — the gap is the time you have to act.