Preface

The CSV problem

Open any introductory data science course. Somewhere in the first hour there is a line like this:

df = pd.read_csv("sales.csv")

And then the course proceeds — cleaning, joining, modeling, plotting, concluding. It is good material. Thousands of people have started useful careers on the back of it.

But that first line is a lie of omission, and it is a large one. Somebody produced sales.csv. Producing it meant answering questions nobody in the course will ask: Which system is the source of truth for a sale — the order service, the payments processor, or the ledger the finance team reconciles against? What counts as a sale: an order placed, an order paid, or an order shipped? What happens to the row when the customer returns half the order six weeks later? How do we pull the data out without locking a table that a checkout page needs? Where do we put it so that it is still there in two years and still cheap? What do we do when the upstream team renames a column on a Tuesday afternoon with no notice?

Answer those questions well and the data scientist gets a clean CSV and never thinks about you. Answer them badly and everything downstream is wrong in ways that are extremely difficult to detect, because wrong data looks exactly like right data. It has the same column names. It renders in the same dashboard. It just quietly says the wrong thing, confidently, for a month.

That is data engineering. This book is about doing it well.

What makes this book different

There are three kinds of data engineering material available today, and each is missing something.

Vendor documentation is excellent at telling you what a tool does and nearly useless at telling you whether you should be using it. It has an obvious structural bias: no product's documentation will tell you that the product is overkill for you, which is the single most valuable thing a junior engineer could hear.

Conference talks and blog posts describe what worked at companies whose scale you do not have. The architecture that Netflix needs is not the architecture you need, and copying it is one of the most expensive mistakes in this field. Much of what reads as best practice is really scale-specific practice, presented without its scale.

Academic database texts are rigorous and enduring and they mostly stop before the parts that consume your working life. They will teach you the relational algebra behind a join and say nothing about what to do when the join key is null for 3% of rows because an upstream service started writing empty strings.

This book tries to sit in the space those three leave open. It is organized around the lifecycle — generate, ingest, store, transform, serve — rather than around a tool list, because the lifecycle outlives the tools. Every tool in this book will be replaced. The problems will not be. Kafka is a particular answer to a durable question about buffering producers from consumers; when Kafka is gone, that question will still be there and something else will be answering it. So each chapter teaches the problem first, the general shape of solutions second, and a specific tool third.

It is also organized around failure. The technique sections in this book do not exist because somebody thought they were elegant. They exist because a specific thing broke at a specific hour and the fix generalized. Idempotency gets a chapter's worth of attention not because it is intellectually interesting but because a non-idempotent backfill can inflate your company's reported revenue by 11.4% for thirty-one days without a single alert firing. You will meet that incident in Chapter 1 and it will keep coming back, because it is the best argument in the book for almost everything the book asks you to do.

The three commitments

Everything runs on your laptop. No cloud account, no credit card, no free-tier clock ticking down. PostgreSQL, MinIO standing in for S3, Kafka, Spark, DuckDB standing in for Snowflake, dbt, Great Expectations, and Airflow all run in Docker on a machine you already own. Where the cloud version differs — and sometimes it differs a lot — the book says so explicitly rather than pretending the local version is the real thing.

Every number is sourced. This turns out to be a much harder commitment than it sounds. A book about infrastructure is full of numbers, and the easy way to write them is from memory: Parquet is about ten times smaller than JSON, Spark is faster than pandas above a few gigabytes, that job cost a few thousand a night. Numbers written that way are wrong at a rate that would horrify the person writing them. So every quantitative claim here is one of exactly four things: a figure from the book's frozen anchor set, arithmetic performed on the page from those figures, output produced by code you can run from the book's own repository, or a citation to a real source. Where none of those was available, the sentence is written qualitatively instead — "much smaller," not "10× smaller."

Cost is an engineering property. Not a finance concern that arrives later. The difference between a well-designed pipeline and a poorly designed one that produce identical output is routinely a factor of ten on the monthly bill, and the decision that creates that difference is made by an engineer, in code, in about four seconds, usually without noticing. So cost arithmetic appears throughout rather than being quarantined in one chapter. There is a chapter on it too — Chapter 33 — but by then it should feel like a summary of a habit rather than a new topic.

What this book is not

It is not a tool certification. If you need to pass a specific vendor's exam, use their material.

It is not a scale trophy case. There is no chapter about handling a petabyte because you almost certainly do not have a petabyte, and the architecture that handles one badly serves the volume you actually have. Where scale genuinely changes the answer, a 📏 Scale Note says so and says what changes.

It is not neutral. It has opinions: that ELT beats ETL for most analytical workloads and that the exceptions are predictable; that your first streaming pipeline should probably have been a batch job running every fifteen minutes; that dbt's real contribution was social rather than technical; that most data mesh implementations are org charts wearing an architecture costume; and that the single highest-leverage thing a junior data engineer can learn is idempotency. Each of those opinions is argued rather than asserted, and each chapter names the strongest case against its own position.

How the book is built

The book is written around one running example, the Kestrel Data Platform — the analytics infrastructure of a fictional online outdoor-gear retailer with 2.4 million orders a year and 14 million clickstream events a day. That is a deliberate choice of size. It is large enough that naive approaches genuinely fail, and small enough that everything still fits on a laptop and every number remains checkable by hand. You add one component per chapter. By Chapter 38 it runs end to end, reconciles to the source to the cent, has tests, has a runbook, and has a cost model.

Chapters are self-contained enough to read out of order, and Chapter 1 gives four routes through the book depending on why you are here. The full course is fifteen weeks. The fast route is six.

A word about the job

Data engineering is, on most days, plumbing. The work is unglamorous and largely invisible: when it goes well, nobody thinks about you, and the visible credit accrues to the people whose analysis your pipeline made possible. That is genuinely how it should be. Infrastructure that draws attention to itself is infrastructure that is failing.

But the invisibility can make the job feel small, and it is not. You are the reason a number on a screen means what someone thinks it means. Every model, every dashboard, every quarterly decision made from a chart rests on the assumption that the underlying data is what it claims to be — and almost nobody downstream is in a position to check. You are the check. That is not plumbing. That is custody.

Take it seriously and the rest of the book follows naturally.

Chapter 1 begins with what the job actually is, why it separated from software engineering and analytics into its own discipline, and the two incidents that frame everything that follows.