Further Reading: Apache Airflow

Sources are tagged Tier 1 (confident it exists, recommended without reservation) or Tier 2 (real and worth seeking, but confirm the current edition, version, or URL yourself).

Airflow's documentation is good and its community material has a version problem worse than any other topic in this book. Airflow 1.10 material describes execution_date, schedule_interval, provide_context=True, and a plugin system that no longer exists — and it is still the top search result for many questions, because it has been accumulating links since 2018. Check the version before the author, every time.

The primary sources

• The Apache Airflow documentation, and specifically these pages read end to end rather than searched: "DAG Runs" (which is §24.3's date model, from the source, and the only place it is explained properly), "Concepts → Tasks", "Deferrable Operators & Triggers", and "Data-aware scheduling". The DAG Runs page in particular repays a slow read — the confusion it resolves is worth more than any tutorial. Tier 1 — versioned; make sure you are on 2.10 or later.

• The "Best Practices" page in the same documentation. It says most of §24.2 and §24.10, including the top-level-code warning, and it is a page nobody reads because its title promises platitudes. It does not deliver platitudes. Tier 1.

• The Airflow source, airflow/jobs/scheduler_job_runner.py. An unusual recommendation, and worth it once: the scheduler's main loop is readable, and reading it makes "why is my task queued" (§24.12) a question with an answer rather than a mystery. Skim it; do not study it. Tier 1.

• The Astronomer documentation and guides. A vendor, and the best free Airflow teaching material that exists — the guides on dynamic task mapping, datasets, and deferrable operators are clearer than the official pages and are kept current. Tier 2 — vendor-maintained; check dates.

Books

• Bas Harenslak and Julian de Ruiter, Data Pipelines with Apache Airflow (2nd ed., Manning, 2023). The book on Airflow. The second edition covers 2.x, including the TaskFlow API and deferrable operators. Chapter 3 on scheduling is the careful version of §24.3, and if that section left you unsure, this is where to go. Tier 1 — confirm the second edition; the first is a 1.x book.

• Joe Reis and Matt Housley, Fundamentals of Data Engineering (O'Reilly, 2022), on orchestration. Short, and useful for the part this chapter treats briefly: what an orchestrator is for, independent of which one, and how the category came to exist. Tier 1.

• Google's Site Reliability Engineering, on monitoring and on "the four golden signals." Case Study 2's argument — that alerting is built on events and cannot see an absence — is a data-shaped version of an operations problem this literature solved first. The heartbeat/canary pattern comes from there. Free online. Tier 1.

The alternatives, read fairly

• The Dagster documentation, particularly on Software-Defined Assets. §24.7's datasets, taken all the way: you declare the assets and their dependencies, and the tasks are derived. Read it even if you will never migrate — it is the clearest available statement of what an orchestrator for data should look like, and it changes how you write Airflow DAGs. Tier 1.

• The Prefect documentation. Python-first, far less ceremony, and dynamic behaviour that Airflow makes awkward is trivial. Worth an afternoon to see what the ceremony was buying and what it was not. Tier 2 — Prefect has had significant API changes across major versions.

• Any honest comparison of the three. They are rare; most are written by one of the three. The question that actually decides it is rarely technical: who else uses it, what your next engineer has used, and what your tools integrate with. §24.13 says so plainly, and most comparisons will not.

On the specific failures

• The Airflow documentation on airflow db clean and database maintenance. Case Study 2. It is not on by default, it is in no getting-started guide, and it produces no symptom until the volume is full. Tier 1.

• The PostgreSQL documentation on table sizes and pg_total_relation_size. Two queries from Case Study 2's analysis, and the ones to have in a runbook. Tier 1.

• Anything on alert fatigue — the same recommendation as Chapter 23, for the same reason. Case Study 2's forty-one correct restart alerts are the canonical shape: an alert whose usual meaning is "ignore me" will be ignored when its meaning changes, and the fix is never to read more carefully. Tier 2.

Practice

• code/dag_lint.py in this chapter. Parses DAG source with ast — no Airflow required — and finds module-level I/O, wall-clock use in tasks, DataFrames in XCom, missing max_active_runs, non-deferrable sensors, and ExternalTaskSensor without an offset. Point it at a repository you have access to and count the findings.

• airflow standalone. A scheduler, web server, and SQLite metadata database in one command and about two minutes. Everything in this chapter's Parts A–C runs against it, including breaking the scheduler on purpose, which is Exercise 24.18 and is the fastest way to understand §24.12.

• Clear a historical task in a local instance, deliberately. Exercise 24.13. It takes five minutes and it makes Case Study 1 something you have seen rather than something you have read.

A note on what to be skeptical of

Any Airflow advice that does not say which version. More than in any other chapter here, the answer changes: execution_date versus data_interval_*, schedule_interval versus schedule, plugins, the pre-2.0 scheduler's performance characteristics, and whether datasets exist at all.

And "Airflow is bad, use X" as a genre. Much of the criticism is accurate — the date model is confusing, the UI has been weak, top-level code is a trap — and none of it settles the decision, because Airflow's advantage was never technical. §24.13 states it: your next engineer has used it, every tool integrates with it, and every failure you will have has been had publicly by someone else. That is worth a great deal and it is rarely on the comparison table.