Every data science course starts with a clean CSV. Somebody put that CSV there, and what they did to produce it — extracting from a transactional database without taking it down, landing the result somewhere durable, reconciling schemas that changed underneath them, deduplicating, conforming, testing, scheduling, monitoring, and being paged at 05:41 when it broke — is the subject of this book. Data engineering is among the fastest-growing job titles in technology and has had no free, comprehensive textbook. This one runs forty chapters and fourteen appendices, and it is the foundation volume of the DataField data science sequence: the introductory data science book begins with data in a file, and this one explains how the data got there.
The book is built around a platform you build yourself. Across its chapters you assemble the Kestrel Data Platform — a complete modern stack for a fictional online outdoor-gear retailer with 2.4 million orders a year and 14 million clickstream events a day — entirely in Docker on a laptop, with no cloud account and no budget required. PostgreSQL as the source system feeds both batch extracts and Debezium change data capture; Kafka carries clickstream events; MinIO holds bronze, silver, and gold layers as Delta tables; DuckDB serves as the warehouse, with dbt models and tests, Great Expectations checks, and Airflow DAGs on top. By Chapter 38 it runs end to end, is tested and documented, reconciles to the source to the cent, and has a cost model attached.
Along the way the book covers the data engineering lifecycle and the source systems it starts from, storage and file formats, batch and streaming ingestion, dimensional modeling and transformation, orchestration and operations — monitoring, alerting, incident response, and the on-call reality of owning a pipeline — then advanced topics and the architecture patterns that recur across real platforms. It is written for aspiring data engineers moving over from software engineering or analysis, data scientists tired of being handed broken data, backend engineers whose services produce data that has to flow downstream, analysts hitting walls, and CS students entering the field; engineering managers and CTOs making infrastructure decisions will find it useful for understanding what they are buying. Solid Python, basic SQL, and comfort at a command line are the only prerequisites.