Part V — Orchestration and Operations
Chapters 24–28
Everything so far has been about making something work. This part is about making it keep working while you sleep.
That is a genuinely different engineering problem, and it is the one that separates a data engineer from someone who writes data scripts. A pipeline that runs correctly when you invoke it by hand and watch the output is not finished. It is a prototype of the thing that has to run at 02:00 on a holiday weekend, fail in a way somebody can diagnose from a phone, and be safe to re-run by a person who did not write it.
The 6am SLA becomes concrete in this part. So does the on-call rotation.
Chapter 24 is Apache Airflow: DAGs, operators, the TaskFlow API, scheduling, dependencies, retries, sensors, dynamic task mapping, and backfills. Airflow is the field's default orchestrator and its concepts transfer to every alternative, which matters because there are several good ones.
Chapter 25 is monitoring and observability, and its central argument is that monitoring the pipeline is not enough. Freshness, volume, distribution, and schema are the four data-level signals, and a platform that watches only job status is the platform the duplicate-rows incident happened to.
Chapter 26 turns the 6am SLA into an SLO with an error budget, then builds the runbook, the alerting policy, the escalation path, and the incident review process around it. It is adapted SRE practice, specialized for a domain where failures are silent and the fix is often a backfill.
Chapter 27 is CI/CD for data: testing pipeline code, testing SQL, building a slim clone of the warehouse for pull-request validation, deploying dbt and Airflow safely, and the review gate that catches a one-off script being promoted to a schedule.
Chapter 28 is infrastructure as code: Docker, Terraform, and enough Kubernetes to know what you are looking at. Reproducibility is the goal — the ability to rebuild the entire platform from a repository, which is also what makes disaster recovery a procedure rather than an archaeology project.
What you will have built by the end of Part V: two production Airflow DAGs with real dependency structure, freshness and volume monitoring with calibrated alerts, a written SLO and runbook, a GitHub Actions pipeline that runs the dbt build against a slim clone on every pull request, and Terraform that stands up the cloud version of the platform.
The single idea to carry out of this part: design for the person who gets paged, and assume it is not you. Every alert should name what broke, what it affects, and what to do — and every pipeline should be safe for that person to re-run without thinking hard.