Acknowledgments

To the people who wrote the field down first

This book stands on three works that did the hard part.

Joe Reis and Matt Housley, Fundamentals of Data Engineering (O'Reilly, 2022), gave the field its organizing idea: that data engineering is a lifecycle — generate, ingest, store, transform, serve — with undercurrents running beneath every stage, and that this framing survives the tools. Before that book, most data engineering writing was tool documentation with ambitions. Chapter 2 of this book is a deliberate restatement of their framework, applied to one concrete system so that a reader can hold all five stages at once.

Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017), is the reason Chapter 4 exists at all. Every genuinely hard bug in this field is a distributed systems bug, and Kleppmann is the person who made that body of theory legible to working engineers without diluting it. Where this book compresses something to a page, DDIA is where to go for the chapter.

Ralph Kimball and Margy Ross, The Data Warehouse Toolkit (Wiley, 3rd ed. 2013), settled dimensional modeling so thoroughly that thirty years of new storage technology has not dislodged it. Star schemas outlived the hardware they were designed for, outlived the ETL tools built around them, and are quietly at the center of most well-run modern warehouses. Chapter 6 and Chapter 20 are Kimball with different syntax.

To the projects

This book is possible because an enormous amount of genuinely excellent infrastructure software is free. PostgreSQL, which is the reason a beginner can have a real database in ninety seconds. Apache Kafka, Apache Spark, Apache Airflow, Apache Parquet, Apache Avro, and Apache Iceberg, and the Apache Software Foundation's model of stewardship that keeps them independent of any one vendor. DuckDB, which made a real analytical database small enough to teach with and good enough to ship with. dbt, which changed the sociology of the field more than its technology. Delta Lake, Debezium, Great Expectations, Polars, Arrow, MinIO, Docker, and Terraform.

Maintaining infrastructure software is thankless in a specific way that data engineers should recognize instantly: when it works, nobody thinks about you. Thank you anyway.

To the on-call engineers

Every ⚠️ Failure Mode and 🏭 From the Pipeline callout in this book is a compressed version of somebody's bad night. The duplicate-rows incident, the four-thousand-dollar Spark job, the timezone that shifted a daily boundary by an hour twice a year, the schema change deployed on a Friday, the DAG that was green for a month while producing nothing — these are the field's actual curriculum, and they are almost never written down, because the incident review is internal and the conference talk is about the successful redesign that came after.

If you have ever explained to a stakeholder why the number changed, you have contributed to this book.

To the DataField series

This is the twenty-somethingth volume in an ongoing open textbook project, and it inherits a great deal from its predecessors: the practitioner voice from the photography and video production books, the running-project structure from the quantum programming book, the sealed-capstone technique from the medical billing book, and — most importantly — a set of hard-won rules about numbers that the whole series now writes under.

The most valuable of those rules came from failures. Harvest, do not recall: a summary must reproduce figures from a generated source file, never from an author's memory, because three different plausible-looking tallies is what happens otherwise. Re-deriving from a printed, rounded value is how one figure becomes two. A number reused as a different quantity is the error no proofread catches. Those lessons cost real work to learn and they shaped how every quantitative claim in this book was produced.

To the reader who reports an error

A book this dependent on version-specific behavior and current cloud pricing begins drifting the day it is published. The most useful contribution anyone can make is a correction — a moved API, a price that changed, a command that no longer works, and above all a number in this book that disagrees with another number in this book.

That last category is the one that matters most, and it is the reason scripts/validate.py checks frozen figures mechanically. Mechanical checks catch a lot. They do not catch everything. You might.

Licensing

Released under CC BY-SA 4.0 for the text and MIT for the code, so that it can be taught, translated, excerpted, forked, and built upon without asking anyone's permission. That was the point of writing it.

Kestrel Supply Co. is fictional. Any resemblance to a real retailer is coincidental. The failure modes are not coincidental at all.