Part I — Foundations
Chapters 1–6
Six chapters before you write a pipeline. That is deliberate, and it is the part of this book most likely to be skipped by someone impatient to build something.
The reason to resist is that data engineering has an unusually high ratio of decisions to code. The code that loads a table is rarely difficult. Deciding what the table's grain should be, where it sits between raw and curated, whether it is rebuilt or appended, what it guarantees to its consumers, and what happens when the source changes underneath it — those decisions determine whether the system is maintainable, and they are almost impossible to revisit cheaply once a hundred downstream objects depend on them.
So Part I is about building the model in your head that the rest of the book fills in.
Chapter 1 establishes what the discipline is: what a data engineer produces, how the role separated from database administration and ETL development, how it differs from its neighbors, and the two Kestrel incidents that motivate most of the practices in this book. It also introduces Kestrel Supply Co. itself — 2.4 million orders a year, 14 million events a day, a 6am deadline, and four engineers.
Chapter 2 gives the framework everything else hangs on: the lifecycle — generate, ingest, store, transform, serve — and the six undercurrents that run beneath every stage. Once you can place a problem in the lifecycle you can usually name the class of solution, and that skill outlives every tool in this book.
Chapter 3 is where the arguments live. Batch versus streaming. ETL versus ELT. Warehouse versus lake versus lakehouse. These are presented as trade-offs with named losing sides rather than as best practices, because the correct answer genuinely depends on things this chapter teaches you to notice.
Chapter 4 is the theory chapter, and it is short because it is aggressively selective. Partitioning, replication, consistency, and delivery semantics — only what you need to debug a real pipeline. Every genuinely hard bug in this field is a distributed systems bug wearing a costume, and this chapter is how you recognize the costume.
Chapter 5 surveys the tool landscape honestly, including what each tool costs to operate and
what you can safely leave out. It is also where the Kestrel Platform gets its first
docker-compose.yml.
Chapter 6 is data modeling: dimensional modeling, star schemas, fact grain, conformed dimensions. It is the oldest material in the book and the most durable. A well-modeled warehouse on mediocre infrastructure beats a badly modeled one on excellent infrastructure, every time, and the gap widens with age.
What you will have built by the end of Part I: a project charter, a repository, a running
docker-compose.yml with PostgreSQL and object storage, and — on paper — the complete dimensional
model for Kestrel's warehouse. No data has moved yet. That is on purpose.
If you are short on time: Chapters 1, 2, 3, and 6 are the Quick Start path through this part. Chapter 4 is mandatory if you are heading for streaming. Chapter 5 is the most perishable material in the book — read it for the shape of the landscape, not the product names.