Exercises: Apache Airflow

pip install apache-airflow==2.10.5 and airflow standalone gives you a working scheduler, web server, and SQLite metadata database in about two minutes. Everything in Parts A–C runs against it.

code/dag_lint.py needs no Airflow at all — it parses the source with ast — so it runs in CI on a laptop.

Difficulty: ⭐ warm-up · ⭐⭐ standard · ⭐⭐⭐ deeper. Solutions: daggered (†) and odd-numbered problems are in appendices/answers-to-selected.md.

Part A — Warm-ups ⭐

24.1 † At what point does cron stop being adequate? Name the five things it cannot do, and the honest costs of the alternative.

24.2 Distinguish a DAG, a task, a task instance, an operator, and an executor.

24.3 † A DAG with schedule="@daily" and a data interval of 2026-03-17: when does it run, and what data is it about? Why is that design correct despite being confusing?

24.4 What does datetime.now() inside a task break? Give all three consequences.

24.5 † What does catchup=True do on deploy? When is it right and when is it wrong?

24.6 Why do retries turn non-idempotency from a risk into a certainty? Name the failure mode that makes it the normal case.

24.7 † What is the right size for a task, and what does splitting cost?

24.8 A sensor, a deferrable operator, and a dataset all express waiting. Rank them and say why.

24.9 † Name the four concurrency limits and which one is Chapter 20's lock.

24.10 What belongs in XCom, and what is the test?

Part B — Standard ⭐⭐

24.11 Write a three-task DAG with the TaskFlow API: extract to a file, load from that file, and report a row count. Pass a path, not data. Run it and paste the graph.

24.12 † Add catchup=True and a start_date thirty days ago to a DAG in a local instance. Report how many runs are scheduled and how long the scheduler takes to clear them. Then set it back.

24.13 Reproduce Case Study 1. Write a task using datetime.now() to choose its window and {{ ds }} for its output path. Run it for today, then clear a run from two weeks ago. Report what the two-week-old partition now contains.

24.14 † Fix that task. Then run airflow tasks test twice against a historical interval and diff the output. Report both.

24.15 Run dag_lint.py against a DAG repository you have access to. Report the findings, and for each one decide whether it is real. How many false positives?

24.16 † Build the §24.8 starvation scenario locally: a DAG with max_active_runs unset, a backfill, and a second DAG that needs the same worker slots. Report the second DAG's start time with and without a dedicated pool.

24.17 Replace an ExternalTaskSensor with a Dataset. Report what got shorter, and what assumption disappeared.

24.18 † Write the four-line canary DAG from Case Study 2 and point it at a free external heartbeat service. Then stop your scheduler and time how long until you are told.

Part C — Deeper ⭐⭐⭐

24.19 §24.10 says a task should be four lines calling a library function. Take a real PythonOperator with substantial logic and refactor it. Report the test you could then write that you could not write before.

24.20 † Case Study 2 argues that every alerting system is built on events and cannot see an absence. Enumerate the absences in a system you own — things that would produce no signal if they stopped — and design the minimum set of heartbeats that covers them.

24.21 Evaluate Dagster or Prefect against a DAG you have written in Airflow. Rewrite one DAG and report what became easier, what became harder, and what you would need to believe to migrate.

24.22 † §24.3's date model is confusing and correct. Design an alternative that is both correct and intuitive, then find the case that breaks it. (There is one. Late-arriving data is a good place to start.)

Part D — The Kestrel Platform ⭐⭐⭐

24.23 — Increment 24: kestrel_daily and kestrel_hourly.

(a) Write the four DAGs from §24.14's layout. Every task body under fifteen lines, calling into an importable kestrel/ package.

(b) kestrel_quality runs at 02:45, before the build, as its own DAG. A stale source must stop the build rather than be discovered inside it.

(c) A warehouse_writes pool of 1; a backfill pool of 4 of 32; max_active_runs=1 everywhere a shared target is written; priority weights favouring the nightly run.

(d) Every cross-DAG dependency is a Dataset. Any remaining sensor is deferrable and carries a comment saying why it is not a dataset.

(e) Run airflow tasks test twice on the same interval for every task in kestrel_daily, and diff the target after each. Any task whose second run changes the output will corrupt data on its first retry — and retries are configured, so it will. This is the exercise that carries the chapter.

(f) dag_lint.py runs in CI and fails on any ERROR. test_dags.py asserts no import errors, a parse under two seconds, retries and owners on every DAG, and no wall-clock use without an exemption comment.

24.24 † Add the canary DAG and an external monitor, plus airflow db clean in kestrel_maintenance and a 70% disk alert on the metadata volume. Then kill your scheduler and record the time to notification.

Reflection

A. This chapter's two case studies are a failure where every task was green and a failure where there were no tasks at all. Which would your monitoring catch, and how quickly? Answer with a number, not a judgment.

B. §24.11's argument is that Airflow code is testable if the logic is not inside the operators. That is the same argument as Chapter 19 §19.9's about documentation, Chapter 22 §22.12's about notebooks, and Chapter 23's about controls that exist versus operate. State the shared principle in one sentence, and then find the place in your own systems where it is most violated.