Chapter 1: What Is Data Engineering?

"Data engineering is the development, implementation, and maintenance of systems and processes that take in raw data and produce high-quality, consistent information that supports downstream use cases." — Joe Reis and Matt Housley, Fundamentals of Data Engineering, 2022

Overview

At 05:41 on a Tuesday, a phone buzzes. The alert says kestrel_daily.load_fct_order_item has been running for 94 minutes against a 22-minute baseline. The dashboard the chief executive opens at 06:15 shows yesterday's revenue, and it is built on the table that job writes. There are thirty-four minutes.

That is the job. Not the whole job — most days contain no alert at all, and the best data engineers are the ones whose phones are quietest — but it is the part that clarifies everything else. Every design decision in this book, every argument for testing data rather than pipelines, every insistence on idempotency, every warning about clever architecture, traces back to some version of 05:41.

This chapter establishes what the discipline is before we build anything. That is worth three hours because data engineering is unusually easy to misunderstand from the outside. It looks like a tool list — Airflow, Spark, Kafka, dbt, Snowflake — and people who approach it as a tool list end up learning five products and no engineering. It looks like a subset of software engineering, which is closer, but misses that the failure modes are fundamentally different: software fails loudly and data fails silently. It looks like a subset of analytics, which is the most common misreading of all, and the one that produces pipelines nobody can operate.

What data engineering actually is: the discipline of moving data from where it is produced to where it is used, at a scale and reliability that lets someone downstream treat it as true.

That last clause carries the weight. Anyone can move data. Moving it such that a person three teams away can build a model on it, or a chief financial officer can report it to a board, without either of them being in a position to verify it — that is the engineering problem, and almost all of it is about failure.

In this chapter, you will learn to:

• Describe what a data engineer produces: not "pipelines" but a specific set of artifacts — landed data, conformed tables, tests, schedules, contracts, and documentation — that other people build on.
• Trace the history of the role from database administration through ETL development and the "big data" era to what the job looks like now, and understand which parts of that history are still load-bearing.
• Distinguish data engineering from analytics engineering, data science, ML engineering, and SRE, using the sharpest available test: who is accountable when a number is wrong.
• Know Kestrel Supply Co. — the fictional retailer whose data platform you will build — well enough to reason about its constraints: 2.4 million orders a year, 14 million events a day, a 340 GB source database, and a 6am deadline.
• Understand two incidents that recur throughout the book: a backfill that silently inflated revenue by 11.4% for thirty-one days, and a nightly Spark job that cost $3,840.00 because of one cast in a WHERE clause.
• See the whole project you are building, and the criterion by which it will be judged complete.

Learning Paths

Forty chapters is a lot, and not everyone needs all of them. Pick a path now; you can always widen it later.

Three notes on choosing.

Chapter 4 is not optional on the streaming path. Every genuinely hard streaming bug is a distributed systems bug wearing a costume — a partition rebalance, a clock skew, a retry that duplicated a write. Reading Chapter 4 first turns three weeks of confusion into an afternoon.

Chapter 38 is on every path. The capstone is where the pieces stop being separate topics. Even if you skip half the book, do the capstone with what you have.

Quick Start is a real path, not a consolation prize. A person who can model a star schema, land data reliably, transform it in dbt, test it, and schedule it in Airflow is doing the job that most data engineering positions actually describe. The other twenty-six chapters make you better at it and open specific doors. They are not the price of entry.

1.1 The CSV That Somebody Made

Start where most people start.

import pandas as pd
df = pd.read_csv("orders_2025.csv")
df.groupby("category")["revenue"].sum().sort_values(ascending=False)

Eight seconds of work, and a result a business would pay for. This is the visible surface of data work, and it is genuinely valuable — the analysis is where a decision gets made.

Now ask the boring questions.

Where did orders_2025.csv come from? Somebody queried a system. Which one? A retailer of Kestrel's size has an order service, a payments processor, a warehouse management system, a customer service platform, and a finance ledger, and all five of them have a number they would call "revenue." They do not agree. They cannot agree, because they are measuring different events at different moments with different definitions of what counts.

What is a "sale"? An order placed is not an order paid. An order paid is not an order shipped. An order shipped is not an order that was kept — Kestrel's return rate means roughly one line in fifteen comes back. If your CSV counts orders placed and the finance team counts revenue recognized on delivery, your dashboard and their board deck will disagree by millions and both of you will be right.

What happens when it changes? Someone returns half an order six weeks after placing it. The row in the source database is updated in place: status goes from delivered to refunded. Your CSV was extracted four weeks ago. It still says delivered. So does every model trained on it.

How was it extracted without breaking anything? SELECT * FROM orders against a production database at 09:00 on Black Friday is, technically, a denial-of-service attack that you performed on your own employer. The extraction has to be scheduled, throttled, watermarked, and pointed at a replica.

Where does it live now? On somebody's laptop, in Downloads, named orders_2025 (3).csv. In two years, when someone asks how the 2025 number was computed, that file will be gone and the person will have left.

And when a column disappears? The upstream team renames total_amount to total_amount_cents on a Tuesday afternoon to fix a rounding bug — a genuinely good change. Nobody tells you. Your pipeline either crashes, which is the good outcome, or silently produces nulls, which is the one that costs a month.

Six questions, none of them glamorous, all of them the job. Data engineering is the discipline that answers them systematically instead of one crisis at a time.

🏭 From the Pipeline — The three-revenue problem

A mid-size retailer once found that its executive dashboard, its finance close, and its investor-relations deck reported three different revenue figures for the same quarter. All three teams were competent. All three numbers were "correct."

The dashboard counted orders at checkout, including ones later cancelled. Finance recognized revenue on delivery, per accounting standards. Investor relations used the payments processor's settlement total, net of chargebacks and processor fees.

The fix was not a pipeline. It was a definition: one table, fct_order_item, with a documented grain and three named measures — gross_revenue_cents, net_revenue_cents, settled_revenue_cents — and a rule that no dashboard may compute revenue any other way. The pipeline came after. The hard part of data engineering is usually the definition, and the code is the easy part that follows it.

The word "pipeline" hides the work

"Data pipeline" is the field's standard term and it is slightly misleading, because it suggests a tube: data goes in one end, comes out the other, and the engineering is plumbing.

A better mental image is a factory line with quality control. Material arrives in an unreliable condition from a supplier you do not control. It gets inspected, corrected, reshaped, combined with other material, inspected again, and packaged in a form the next station can use. Every station can fail, and the expensive failures are not the ones that stop the line — those get noticed. The expensive ones are the stations that keep running while producing defective output, because downstream stations accept it, build on it, and ship it.

That is the shape of nearly every serious data incident. Not "the pipeline broke." "The pipeline ran."

1.2 What the Job Actually Is

Ask a data engineer what they do and you will usually get a tool list. Here is a better answer, organized by what someone else can point at afterward.

The six artifacts

1. Landed data. Raw data from source systems, in durable storage, in a form that preserves what the source actually said. Not cleaned — preserved. When a downstream number is wrong, the first question is always "what did the source say?", and if you cleaned before you landed, you cannot answer it. Chapters 9 through 17 are about this.

2. Conformed tables. Cleaned, typed, deduplicated, joined data with a documented grain — one row per what? — and defined measures. This is the layer analysts and data scientists actually use, and its quality determines whether they trust you. Chapters 6, 18, 19, and 20.

3. Tests. Not tests of the pipeline code, though those too. Tests of the data: this column is never null, this key is unique, this table has between 5,000 and 60,000 rows a day, this sum reconciles to the source within a cent, yesterday's revenue is within four standard deviations of the trailing average. Chapter 23.

4. Schedules and dependencies. Something has to decide that the transform runs after the load, that a failure retries three times and then pages someone, that the Monday run knows the weekend runs succeeded. Chapter 24.

5. Contracts. An explicit, versioned agreement about what a producer will emit and what a consumer may rely on. Without one, every upstream deploy is a coin flip. Chapter 17.

6. Documentation and lineage. Where did this column come from, what does it mean, who owns it, what breaks if it changes. Chapter 30.

A data engineer who has produced all six for a domain has done the job. One who has produced only the first and fourth has built a machine that moves bytes, which is not the same thing.

A week, honestly

Roughly, for a working data engineer at a company Kestrel's size:

The proportions shift with seniority — staff engineers do more definition and less code — but the shape holds. If you are picturing a job that is mostly writing new pipelines, adjust. You will spend more time on data somebody else produced than on code you wrote.

⚠️ Failure Mode — The silent success

A software service that fails throws an exception, returns a 500, and shows up in a graph within seconds. Everyone finds out fast.

A data pipeline that fails silently succeeds. The DAG is green. The table has rows. The dashboard renders. Nothing anywhere is red. The only signal that something is wrong is that the numbers are wrong, and the only people positioned to notice are the ones who trust you to have already checked.

This asymmetry is the single most important structural fact about the discipline, and it is why Chapter 23 exists, why Chapter 25 measures data and not just jobs, and why "did it run?" is the wrong question. The right question is "is the output what it should be?" — and answering it requires knowing, in advance and in writing, what it should be.

What "production-grade" means here

Every substantial code sample in this book is written to a standard, and it is worth naming it now because it will look like overhead in Chapter 13 and like the bare minimum by Chapter 26.

• Logging at boundaries: what was read, from where, how many rows, how long it took.
• Error handling that distinguishes a transient failure (retry) from a permanent one (stop and page). Retrying a permanent failure three times just delays the alert.
• Idempotency: running it twice produces the same result as running it once. This is the property most often missing and most expensive to lack.
• Testability: pure functions where the logic lives, I/O at the edges, so the transformation can be tested without a database.
• No secrets in code. os.environ["KESTREL_DB_PASSWORD"], never a literal, from Chapter 1 onward. The build validator for this book fails on a hardcoded credential in a sample.

A script that lacks these is not "quick and dirty." It is a future incident with a delay fuse. Some appear in this book anyway, to show the contrast — always with a callout saying so.

1.3 Where the Discipline Came From

The role has existed under four names in forty years, and each layer is still visible in the tools.

The DBA era (1980s–1990s)

One database, on one machine, administered by one person. The database administrator owned schema, backups, indexes, performance, and access. Analytics ran against the same database as the application, at night, when nobody was using it.

The core insight of the era survives completely: schemas are contracts, normalization prevents whole classes of error, and a query plan is something you read rather than guess at. Chapter 7 is substantially DBA craft, and every hour you spend on it pays out for the rest of your career, because underneath the lakehouse there is still a B-tree.

The ETL era (1990s–2000s)

Analytics outgrew the transactional database. Bill Inmon and Ralph Kimball, arguing from different premises, both concluded that analytical data belongs in its own system, shaped for reading rather than writing. The data warehouse was born, and with it ETL — extract, transform, load — and a generation of GUI tools (Informatica, DataStage, SSIS) in which you built pipelines by dragging boxes onto a canvas.

Transformation happened before loading because warehouse storage and compute were expensive and tightly coupled: you bought a box, and every gigabyte you loaded and every CPU cycle you spent came out of the same fixed budget. Cleaning before loading was economically forced.

The modeling from this era is not merely alive, it is dominant. Kimball's star schemas, conformed dimensions, slowly changing dimensions, and fact-table grain are the vocabulary of Chapter 6 and Chapter 20. The tools died for a reason worth remembering: pipelines built by dragging boxes cannot be diffed, reviewed, tested, or rolled back. They were software that had escaped software engineering, and it went exactly as you would expect.

The big data era (2005–2015)

Google published the papers — GFS in 2003, MapReduce in 2004, BigTable in 2006 — Hadoop implemented them in the open, and for a decade the field reorganized around a genuine insight and an overcorrection.

The insight: some datasets do not fit on one machine, and when that is true, moving computation to the data beats moving data to the computation. That is permanently true and Chapter 21 depends on it.

The overcorrection: everybody built for that case. Companies with forty gigabytes of data stood up twelve-node Hadoop clusters, hired teams to operate them, wrote MapReduce jobs to do what a single Postgres query would have done in nine seconds, and called it modernization. An enormous amount of money and career time went into distributed systems solving non-distributed problems.

📐 Design Decision — Distributed by default is a bet, and usually a losing one

The instinct that survives from the big data era is "it might grow, so build it distributed." This is almost always the wrong trade, and it is worth naming what each side costs.

Distributed wins: genuinely unbounded scale, fault tolerance across machines, parallelism you cannot get otherwise.

Distributed costs: every failure mode in Chapter 4 (partial failure, network partitions, clock skew, rebalances), a cluster to operate, a scheduler to tune, minutes of startup latency per job, and — the one nobody prices — debugging that requires reading logs from thirty machines to find one null.

This book's position: start single-node. A modern laptop with DuckDB handles tens of gigabytes comfortably; a large cloud VM handles hundreds. Move to Spark when you have measured that you must, and Chapter 21 §21.2 gives the specific thresholds. What you give up by taking this position: if you genuinely do have a petabyte, you will rewrite. That is a real cost and it is smaller than the cost of the alternative, which is a distributed system operated by people who did not need one.

The modern era (2015–now)

Two changes reorganized everything, and they are both economic rather than technical.

Storage and compute separated. S3 made durable storage effectively unlimited and nearly free — $0.023 per gigabyte-month at the frozen rate this book uses. Snowflake and BigQuery made compute elastic and billed by the second. Once storage is cheap and compute is on demand, the economic argument for transforming before loading evaporates. ETL became ELT: land everything raw, transform in the warehouse with SQL, and keep the raw data forever because keeping it costs almost nothing and having it is the difference between answering "what did the source say?" and shrugging.

Transformation moved into version control. dbt's contribution was not technical — it compiles Jinja-templated SQL and runs it in dependency order, which is not a hard problem. Its contribution was social: it made transformations into files in a git repository, with tests, code review, CI, and documentation generated from the same source. It dragged the ETL developer's craft back inside software engineering, twenty years after the GUI tools dragged it out. Chapter 19.

The result is the role as it exists today: a software engineer who specializes in data systems, works mostly in Python and SQL, thinks in terms of a lifecycle rather than a tool, and is accountable for correctness in a domain where errors are silent.

🧭 Version Note — Everything in this section is dated

The stack described above is the 2020s consensus, and parts of it are already shifting: table formats (Iceberg, Delta) are absorbing responsibilities that belonged to warehouses; single-node engines (DuckDB, Polars) are reclaiming workloads that went to Spark by default; and streaming and batch are converging on shared semantics.

This is why the book is organized around the lifecycle rather than the stack. Generate, ingest, store, transform, serve is the same in 1995 and 2035. Which product performs each stage is a detail that will change under you at least twice in your career, and treating it as the subject is how you end up expert in a discontinued product.

1.4 Data Engineering and Its Neighbors

Five roles overlap with this one. The distinctions matter, because they determine what you are accountable for.

The clarifying question is not "who writes SQL" — everyone writes SQL. It is: when a number on a dashboard is wrong, whose job is it to find out why?

The line that actually matters: data engineering vs. analytics engineering

This is the boundary people ask about most, and it is genuinely blurry — many jobs titled one thing are mostly the other.

The most useful test is who fixes it at 5am. If the nightly load failed because the source database rotated a credential, that is data engineering. If the load succeeded but revenue is 8% high because the definition of "revenue" silently changed when someone added a promotions join, that is analytics engineering. If revenue is 11.4% high because a backfill duplicated rows for thirty-one days — the incident in §1.6 — that is data engineering, decisively, and the fact that it presented as a definition problem for two weeks is exactly why the roles need to talk.

At Kestrel's size the same four people do both, and this book covers both. At a large company they are separate teams with separate on-call rotations, and the interface between them is a contract about what the silver layer guarantees.

🎓 Interview Angle — "What's the difference between a data engineer and an analytics engineer?"

This is a filtering question, and the failure mode is answering with tools ("data engineers use Python and Spark, analytics engineers use dbt and SQL"). That is true and shallow, and every candidate says it.

A better answer names the accountability boundary: "Data engineering owns getting data in reliably and correctly — ingestion, storage, orchestration, quality, and the platform. Analytics engineering owns turning correct data into business-defined models people can use. The clean split is that data engineering is accountable for the data matching the source, and analytics engineering is accountable for the model matching the business. On a small team it is one person, and the reason to name the boundary anyway is that the two failure modes need different fixes."

Then say which side you prefer and why. Interviewers are trying to find out where you want to sit.

What data engineering is not

It is not data entry, and it is not report building. If the role you are offered is mostly building dashboards, it is an analyst role, whatever the title says. This matters when you are job hunting: title inflation in this field is severe, and "data engineer" is applied to jobs ranging from Excel maintenance to distributed systems work. Chapter 39 §39.6 covers how to tell before you accept.

It is not machine learning. Data engineers build the infrastructure ML depends on (Chapter 32), and many drift into ML engineering, but training models is a different job with different skills.

It is not DevOps. There is real overlap — Docker, Terraform, CI/CD, on-call, all in Part V — but an SRE is accountable for systems being up and a data engineer is accountable for data being right. A system can be perfectly up and completely wrong. That case is yours.

1.5 Kestrel Supply Co.

Everything in this book is built for one company. Meet it now, because from Chapter 5 onward you are building its platform, and every architectural decision refers back to these numbers.

Kestrel Supply Co. is an online retailer of outdoor and workwear gear, founded in 2016, headquartered in Denver, with fulfillment warehouses in Denver (DEN), Columbus (CMH), and Reno (RNO). It sells in the United States, Canada, and the United Kingdom. It is fictional. The scale is chosen deliberately: large enough that naive approaches genuinely fail, small enough that every number stays checkable by hand and the whole platform fits on a laptop.

The business, in numbers

Two of these matter more than the rest.

The peak-to-average ratio of 6.28×. This single number drives more architecture than the annual totals do. A system sized for 6,575 orders a day falls over on the one day of the year when falling over is most expensive. A system sized for 41,300 orders a day is idle 364 days out of 365 and you are paying for it the whole time. Chapter 3 §3.5 and Chapter 33 are largely about this tension, and elasticity — paying for capacity only while using it — is the single strongest argument for cloud infrastructure in a retail business.

The average order value of $75.83. Derived, not assumed: $182.0M ÷ 2,400,000 = $75.83. Every revenue figure in this book traces back to that division, and when you see a revenue number you can check it. That is deliberate. A textbook full of numbers you cannot verify teaches you to accept numbers you cannot verify, which is the opposite of the habit this job requires.

The systems

┌──────────────────────────────────────────────────────────────────────────────┐
│  SOURCE SYSTEMS                                                              │
│                                                                              │
│  ┌────────────────────┐   ┌──────────────────┐   ┌───────────────────────┐  │
│  │  kestrel_app       │   │  web + mobile    │   │  third-party APIs     │  │
│  │  PostgreSQL 16     │   │  clickstream     │   │  carriers, payments,  │  │
│  │  12 core tables    │   │  14M events/day  │   │  marketing, support   │  │
│  │  340 GB            │   │  peak 2,900/sec  │   │  rate-limited, flaky  │  │
│  └────────────────────┘   └──────────────────┘   └───────────────────────┘  │
└──────────────────────────────────────────────────────────────────────────────┘
                    │                  │                      │
                    ▼                  ▼                      ▼
         ┌───────────────────────────────────────────────────────────┐
         │  THE PLATFORM  —  what you build in this book             │
         │  ingest → store → transform → serve, plus the             │
         │  undercurrents: quality, orchestration, governance, cost  │
         └───────────────────────────────────────────────────────────┘
                    │                  │                      │
                    ▼                  ▼                      ▼
         ┌────────────────┐  ┌──────────────────┐  ┌────────────────────┐
         │  BI dashboards │  │  data science    │  │  operational       │
         │  incl. the     │  │  churn, demand,  │  │  reverse ETL to    │
         │  6am revenue   │  │  recommendations │  │  the app, support  │
         │  dashboard     │  │                  │  │  tooling, finance  │
         └────────────────┘  └──────────────────┘  └────────────────────┘

In words: three classes of source system — a transactional PostgreSQL database, a high-volume clickstream from web and mobile clients, and a set of third-party APIs — feed a platform whose job is to make their contents usable by three classes of consumer: business intelligence, data science, and operational systems that need data pushed back to them.

kestrel_app, PostgreSQL 16, 340 GB, twelve core tables: customers, addresses, categories, products, warehouses, inventory, orders, order_items, payments, shipments, returns, promotions. It is the source of truth for what was ordered. It is also serving the checkout page, which constrains everything about how you read from it. Chapter 7 covers the schema in full; Chapter 13 covers extracting from it without becoming an outage.

One convention, stated once and honored everywhere: money is stored in integer cents. Every amount column ends in _cents. Chapter 7 §7.4 explains why floating-point currency is a bug rather than a style preference, and the build validator for this book fails on a float money column in a code sample.

The clickstream: 14,000,000 events a day from web and mobile — page views, product views, searches, cart operations, checkout starts, purchases. That is 162 events per second on average and 2,900 per second at peak, a ratio of roughly 18× far spikier than orders, because browsing spikes harder than buying. At roughly 820 bytes of JSON per event, the raw stream is 11.48 GB a day and 4.19 TB a year.

Third-party APIs: carrier tracking, the payment processor, the marketing platform, the customer support tool. Rate-limited, occasionally down, and — the part that surprises people — occasionally wrong, retroactively. Chapter 16.

💸 Cost Check — What format choice costs at Kestrel's volume

One year of clickstream, at the frozen S3 Standard rate of $0.023/GB-month:

• Raw JSON: 4.19 TB = 4,190 GB × $0.023 = $96.37/month
• Parquet + zstd, which compresses this data about 12.3× (measured in Chapter 11 §11.6): 341 GB × $0.023 = $7.84/month

A saving of $88.54 a month, or $1,062 a year. On its own, an unremarkable number — nobody gets promoted for it.

Storage is not where the format decision pays. It pays in what you scan. A query that reads three columns out of forty reads three columns from Parquet and all forty from JSON, and at BigQuery's frozen on-demand rate of $6.25 per TiB scanned, the same monthly query pattern is the difference between tens of dollars and thousands. Chapter 11 §11.7 does that arithmetic properly. Watch for this shape throughout: the visible line item is rarely where the money is.

The team and the constraint

Four data engineers, three analysts, two data scientists. That team size is the most important constraint in the book. It is why the answer is usually the boring option: four people cannot operate a Kubernetes-hosted Flink cluster and a data mesh and a custom lineage service and still ship anything. Every ⭐ architecture recommendation in this book is filtered through "can four people run this at 3am."

And the deadline: the daily_revenue dashboard must be fresh for the previous calendar day by 06:00 America/New_York. The chief executive opens it at 06:15.

Fifteen minutes. That is the error budget as the business perceives it, and it is the recurring stake behind every reliability argument in this book. Chapter 26 turns it into a real SLO with real arithmetic; for now, hold it as the reason any of this matters.

📏 Scale Note — Where Kestrel sits, and what changes above it

Kestrel is a medium data platform: single-digit terabytes of analytical data, tens of millions of daily events, single-digit billions of rows in the largest fact table. This is where most data engineering jobs are, and the advice in this book is tuned for it.

At 10× Kestrel — 24M orders, 140M events a day — three things change: the source database needs a dedicated read replica or CDC becomes mandatory rather than preferred; single-node transformation stops being viable for the largest tables and Spark earns its complexity; and storage layout (partitioning, file sizing, compaction) shifts from a tidiness concern to the dominant cost driver.

At 100× Kestrel the organization changes shape before the technology does. One team cannot own all the data, ownership federates by domain, and you are in Chapter 35 (data mesh) whether you chose it or not.

Below Kestrel — under a few hundred gigabytes — much of this book is optional. Postgres and a nightly script are a legitimate answer, and Chapter 5 §5.7 says so plainly.

1.6 Two Incidents That Frame This Book

Two things went wrong at Kestrel. They recur throughout, because between them they motivate most of what the book asks you to do.

Incident one: the duplicate rows

What happened. An engineer wrote a backfill to repair three days of fct_order_item after a failed load. The backfill inserted rows for orders placed in a rolling three-day window. It worked. They scheduled it nightly "for a while, just in case," and moved on.

It had no delete step.

It ran every night from 2025-03-02 to 2025-04-01 — thirty-one days — each night re-inserting the last three days of order lines on top of rows that were already there. Reported revenue drifted upward by 11.4%.

Why nobody noticed.

• Every DAG was green. The backfill succeeded every night. It was doing exactly what it was written to do.
• Row counts grew, but they grow anyway, and the growth was smooth rather than a step.
• Revenue went up, in March, in a business whose revenue goes up in spring. It looked like a good quarter.
• Nobody was comparing warehouse revenue to source revenue, because why would you — the pipeline works.

How it was caught. A finance analyst reconciling the quarter against the payments processor found an unexplained gap and asked the data team where it came from. Two weeks of investigation followed, most of it spent looking at the definition of revenue, because that is where these discrepancies usually live. The actual cause — SELECT COUNT(*), COUNT(DISTINCT order_item_id) on one table — took about ninety seconds to find once someone asked the right question.

The cost. Two weeks of two people's time. A quarter's revenue reporting corrected after distribution. And the expensive part, which does not appear in any incident report: for the next year, whenever a number moved, somebody asked whether the pipeline was double-counting again. Trust is the actual product, and it is much harder to rebuild than a table.

🔁 Idempotency Check — The property that would have prevented all of it

An operation is idempotent when running it twice produces the same result as running it once.

The backfill was not:

sql -- NOT idempotent. Every run adds rows. INSERT INTO fct_order_item SELECT ... FROM staging.order_items WHERE placed_at >= CURRENT_DATE - INTERVAL '3 days';

The idempotent version is three lines longer and could not have caused this incident:

sql -- Idempotent. Any number of runs converges to the same state. BEGIN; DELETE FROM fct_order_item WHERE placed_at >= CURRENT_DATE - INTERVAL '3 days'; INSERT INTO fct_order_item SELECT ... FROM staging.order_items WHERE placed_at >= CURRENT_DATE - INTERVAL '3 days'; COMMIT;

Delete the window, then rewrite it, in one transaction. The pattern has a name — delete-insert, or insert-overwrite — and Chapter 20 §20.3 covers it along with the three other ways to get idempotency (merge/upsert on a natural key, partition replacement, and content-addressed writes).

Ask of every pipeline you write: what happens if this runs twice? It will. Retries, manual reruns, backfills, and duplicate scheduler triggers all guarantee it. The question is only whether you decided the answer or discovered it.

Incident two: the four-thousand-dollar job

What happened. A nightly Spark job aggregated the clickstream into session-level features. Everyone called it "the four-thousand-dollar job," which is roughly right and, characteristically, not the actual number. Metered:

$$\text{160 executors} \times \text{10.0 hours} \times \$2.400/\text{node-hour} = \$3{,}840.00 \text{ per night}$$

That is $1,401,600 a year for one job — using the frozen compute rate of $2.400 per node-hour that this book uses throughout ($2.016 for the instance plus $0.384 for the managed-Spark surcharge).

The cause. One line:

WHERE CAST(event_ts AS DATE) = '2025-03-14'

The table was partitioned by event_date. The filter was on CAST(event_ts AS DATE) — a function of a column, which the query planner could not match to the partition key. So it could not prune partitions. So it read every partition ever written: 4.2 TB, to produce one day of output.

The fix.

WHERE event_date = DATE '2025-03-14'

Filter on the partition column directly. The job then reads 34 GB — one day's compressed Parquet — and finishes on 24 executors in 1.3 hours:

$$\text{24} \times \text{1.3} \times \$2.400 = \$74.88 \text{ per night}$$

$3,840.00 → $74.88. A 51.3× reduction, $1,401,600 down to $27,331 a year, saving $1,374,269. From deleting four characters and moving a filter to a different column.

🔎 Read the Plan — The one line that told the whole story

Every cost problem of this shape is visible in the query plan, and reading plans is the highest return-on-time skill in this book. In Spark's physical plan the giveaway is the scan node:

text == Physical Plan == *(2) HashAggregate(keys=[session_id#12], ...) +- Exchange hashpartitioning(session_id#12, 200) +- *(1) Filter (cast(event_ts#3 as date) = 2025-03-14) +- FileScan parquet kestrel.bronze_events[...] PartitionFilters: [] ← EMPTY. Nothing was pruned. PushedFilters: [] ← nothing pushed to the reader ReadSchema: struct<session_id:string,event_ts:timestamp,...>

PartitionFilters: [] means the planner found nothing it could use to skip partitions, so the scan reads all of them. After the fix:

text +- FileScan parquet kestrel.bronze_events[...] PartitionFilters: [isnotnull(event_date#8), (event_date#8 = 2025-03-14)] PushedFilters: []

One populated bracket, 4.2 TB down to 34 GB. Learn to look at PartitionFilters and PushedFilters before you look at anything else — Chapter 21 §21.6 and Chapter 33 §33.4.

The general lesson is not about Spark. It is that cloud costs are engineering decisions made in code, usually in seconds, usually without noticing. The engineer who wrote that CAST was not being careless; they were writing correct SQL that produced correct output. It was correct and it cost $1.4M a year, and nothing in the development workflow surfaced that. The job ran, the tests passed, the output was right. The bill arrived a month later, attributed to a cluster, not to a line.

This is why cost arithmetic appears throughout this book rather than only in Chapter 33. A number you never compute is a number you cannot manage.

1.7 The Kestrel Data Platform: The Project You Will Build

You do not learn this by reading. You learn it by building one platform all the way through, hitting the problems in the order they actually arrive.

The goal

Build a production-grade data platform for Kestrel Supply Co. that runs entirely in Docker on your laptop, and that satisfies the same requirements the real thing would.

By Chapter 38 it will:

• Ingest from PostgreSQL by batch extract and by change data capture
• Ingest clickstream events through Kafka, with a dead-letter path for bad ones
• Ingest from a rate-limited third-party API without getting banned
• Land everything in object storage as Parquet, organized in bronze/silver/gold layers
• Transform with dbt, including incremental models and slowly changing dimensions
• Test the data — not just the code — with dbt tests and Great Expectations
• Orchestrate with Airflow, with retries, dependencies, and alerting that means something
• Monitor freshness, volume, and distribution, and page a human when they drift
• Serve a star schema for BI and a feature view for data science
• Be version-controlled, CI-tested, and documented, with a runbook someone else could use
• Reconcile to the source — and this is the acceptance test — with a documented variance

The acceptance criterion

One sentence, and it is the same standard a real platform is held to:

For any calendar month, total net revenue computed from the gold layer equals total net revenue computed directly from the source kestrel_app database, to the cent — and every difference is explained by a documented, tested rule rather than discovered after the fact.

Not "the pipeline runs." Not "the dashboard renders." The number is right, and you can prove it.

Chapter 38 computes that reconciliation for November 2025. Those figures do not appear anywhere before Chapter 38 — you compute them, you do not read them — and the build validator enforces it.

How it grows

One component per chapter. Nothing thrown away, everything refactored in place, exactly as a real platform evolves.

🧱 Kestrel Platform — Increment 1: the charter

Before any code, write down what you are building and how you will know it works. This is Part D of this chapter's exercises, and it is not a warm-up.

Create platform/CHARTER.md with five sections:

1. What this platform is for — three sentences, naming the consumers by role.
2. The acceptance criterion — the reconciliation statement above, in your own words.
3. The SLA — daily_revenue fresh for the prior day by 06:00 America/New_York.
4. Explicit non-goals — three things you are not building. Sub-second latency. A machine learning platform. Support for a second source database. Naming non-goals is how a scope stays finished.
5. The constraint — four engineers. Every design decision gets checked against it.

This takes twenty minutes and you will refer to it in Chapter 38. Every real platform that succeeded had this written down somewhere; every one that sprawled did not.

1.8 How This Book Handles Numbers, Tools, and Cost

Three commitments, because they will shape how you read everything that follows.

Numbers

Every quantitative claim here is one of exactly four things:

1. A frozen anchor figure about Kestrel — the tables in §1.5, fixed for the whole book.
2. Arithmetic performed on the page from those figures, with the working shown.
3. Output from code in this repository, which you can run.
4. A citation to a real, checkable source.

There is no fifth category. Where none applied, the sentence is qualitative: "much smaller," not "about 10× smaller."

This sounds like pedantry. It is not, and this series learned it the hard way. The failure mode is specific and recurrent: an author recalls a figure, writes it slightly differently in a second place, and the two coexist for a hundred pages. Worse is re-deriving from a printed, rounded value — taking a displayed 6.92 that was really 6.9234, multiplying by 4,096, and publishing a total that disagrees with the one computed from the unrounded number. Worst is reusing a number as a different quantity: a ratio recalled later as a count, correct-looking in both places, wrong in one.

You will do all three in your career, on dashboards, in decks, in incident reports. Watching a book try not to is useful practice.

🧪 Try It — Check the book

Every derived figure in §1.5 and §1.6 can be verified with arithmetic. Do three of them now:

1. Average order value. $182,000,000 ÷ 2,400,000 = ? Should be $75.83.
2. Orders per day. 2,400,000 ÷ 365 = ? Should round to 6,575.
3. The Spark job. 160 × 10.0 × $2.400 = ? And 24 × 1.3 × $2.400 = ? Then the ratio of the two: should be 51.3×.

Then find one figure in this chapter that is not derivable from the anchor tables, and decide which of the four categories it belongs to.

Do this again in a chapter that annoys you. Books are wrong, including this one; the difference between a good engineer and a credulous one is the reflex to check.

Tools

Every tool in this book will be replaced. Some are being replaced right now.

So each chapter teaches, in order: the problem, then the shape of solutions, then a specific tool. Chapter 15 is not really about Kafka; it is about the durable problem of decoupling producers from consumers with a replayable, ordered log, which Kafka is the current best answer to. When Kafka is gone the problem will still be there.

The practical consequence for you: when you read "Kafka," mentally substitute "a durable ordered log," and you will find that most of the chapter transfers to whatever your employer actually uses.

Local-first. Every tool runs on your laptop. PostgreSQL is PostgreSQL. MinIO stands in for S3 and speaks the same API, so the boto3 code is unmodified. DuckDB stands in for Snowflake or BigQuery, and where the substitution leaks — mostly around concurrency, scale-out, and cost model — the book says so explicitly rather than pretending.

Cost

Cost arithmetic appears in every part, against one frozen basis used book-wide:

Appendix J has the full table with the tiers and the caveats.

One basis, everywhere. That is deliberate and it comes from a mistake in an earlier book in this series, where a contribution ratio computed on one cost base was paired with a profit figure computed on another, and two readers independently produced the same wrong answer because the book had made it easy to. Mixing cost bases is how you get two contradictory conclusions from one architecture.

These prices are US list, us-east-1, on-demand, at the time of writing, and they are already drifting. That is fine. The arithmetic is the skill; every cost example shows its working so you can substitute today's numbers and get today's answer.

1.9 Summary

Data engineering is the discipline of moving data from where it is produced to where it is used, at a scale and reliability that lets someone downstream treat it as true. The last clause is the engineering problem. Anyone can move data.

The claims this chapter makes, restated as claims:

The clean CSV is the output of a process, not a starting point. Six unglamorous questions stand between a source system and a usable file: which system is authoritative, what counts as the event, what happens when it changes, how to extract without causing an outage, where it lives afterward, and what happens when the schema moves. Answering them systematically is the job.

A data engineer produces six artifacts, and a stakeholder can point at each: landed raw data, conformed tables with documented grain, tests of the data, schedules and dependencies, contracts with producers, and documentation with lineage. Producing only pipelines and schedules is building a machine that moves bytes, which is not the same thing.

Data failures are silent. A software service that fails throws an error; a data pipeline that fails succeeds — green DAG, populated table, rendering dashboard, wrong numbers. This asymmetry is the most important structural fact about the discipline and it is why "did it run?" is the wrong question.

The role's history is still load-bearing. DBA-era schema and query-plan craft, ETL-era dimensional modeling, big-data-era distributed processing, and the modern era's separation of storage from compute each left something permanent. What died was pipelines-as-GUI-artifacts, and they died because software that has escaped version control cannot be reviewed, tested, or rolled back.

The neighbors are distinguished by accountability, not tooling. Everyone writes SQL. The question is who finds out why the number is wrong: data engineering owns data matching the source, analytics engineering owns models matching the business, data science owns the conclusion, ML engineering owns the model in production, and SRE owns the system being up. A system can be perfectly up and completely wrong; that case is yours.

Kestrel Supply Co. is the whole book's example: $182.0M GMV, 2,400,000 orders, an $75.83 average order value, 14 million clickstream events a day peaking at 2,900 per second, a 340 GB PostgreSQL source, four data engineers, and a dashboard that must be fresh by 06:00 because someone opens it at 06:15. The 6.28× peak-to-average ratio and the four-person team drive more architecture than the totals do.

Two incidents frame everything. A backfill with no delete step ran for thirty-one days and inflated revenue by 11.4% with every DAG green — the standing argument for idempotency and for testing data rather than pipelines. And a CAST in a WHERE clause defeated partition pruning, turning a 34 GB scan into 4.2 TB and a $74.88 night into $3,840.00 — the standing argument that cloud cost is an engineering property decided in code, in seconds, usually without noticing.

You build one platform, all the way through, and it is judged by reconciliation to the source rather than by whether it runs.

What's next

Chapter 2 gives the framework the rest of the book hangs on: the data engineering lifecycle — generate, ingest, store, transform, serve — and the undercurrents that run beneath every stage: security, data management, DataOps, architecture, orchestration, and software engineering. You will map Kestrel onto it, which will show you exactly where the hard parts are before you build any of them.