Chapter 2: The Data Engineering Lifecycle

"The lifecycle is the invariant. Tools come and go, but data is generated, ingested, stored, transformed, and served." — Joe Reis and Matt Housley, Fundamentals of Data Engineering, 2022

Overview

Chapter 1 argued that data engineering is easy to misunderstand as a tool list. This chapter gives you the thing to understand it as instead.

The data engineering lifecycle is five stages — generate, ingest, store, transform, serve — with six undercurrents running beneath all of them. It comes from Reis and Housley's Fundamentals of Data Engineering, and it is the most useful organizing idea in the field, for a reason that has nothing to do with elegance: it is stable. Every tool named in this book will be replaced. The five stages were the same in 1995, they are the same now, and they will be the same when whatever replaces Kafka is being taught.

That stability has an immediate practical payoff. When someone brings you a problem — "the marketing team can't see yesterday's attribution data" — the lifecycle lets you locate it before you start guessing. Is the source not producing the event? Is ingestion failing? Is it landing but not being transformed? Is it transformed but not served in a form they can use? Four different questions with four different classes of answer, and the wrong guess costs a day.

Most junior engineers debug by reaching for the tool they know best. Most senior engineers locate the stage first. This chapter is about the second habit.

There is one thing to flag before we start, because it is the most common misreading of the model. The lifecycle is not a pipeline. It is not five boxes with arrows through which each row passes exactly once. Store sits underneath everything rather than between ingest and transform, because data is stored at every stage — you store what you ingested, you store intermediate transformations, you store what you serve. Drawing Store as a box between Ingest and Transform is the single most common way this diagram gets copied wrong, and the mistake propagates into architectures that treat storage as a waypoint rather than as the substrate.

In this chapter, you will learn to:

• Name and define the five stages, and state the specific question each one owns.
• Explain why Store is a substrate rather than a stage in sequence, and what goes wrong when it is modeled as a waypoint.
• Name the six undercurrents — security, data management, DataOps, data architecture, orchestration, software engineering — and give an example of each failing.
• Place a problem in the lifecycle before choosing a tool, which is the diagnostic habit this chapter exists to build.
• Map Kestrel's three source systems through all five stages and see where the risk actually concentrates.
• Recognize the Transform/Serve boundary, which is the one most often drawn in the wrong place, and understand what drawing it wrong costs.

Who needs this chapter: everyone, on every path. It is the shortest chapter in Part I and the one the other thirty-eight refer back to most.

2.1 Why a Lifecycle and Not a Tool List

Here is the same job description written two ways.

As a tool list: "Build and maintain data pipelines using Airflow, dbt, Spark, and Snowflake. Experience with Kafka and Terraform preferred."

As a lifecycle: "Own the movement of data from our transactional systems and event streams into a warehouse where it can be trusted — including how it gets extracted, where it lands, how it is cleaned and modeled, how it is tested, how it is scheduled, and how consumers get at it."

Both describe the same job. The first is how it is advertised; the second is what it is.

The problem with the first framing is not that the tools are unimportant. It is that the tool list gives you no way to reason about a problem you have not seen before, and it gives you no way to evaluate whether a tool is the right one, because you have not named what job it is doing.

📐 Design Decision — Learn the stage, then the tool

The alternative position is defensible and widely held: learn the tools your employer uses, deeply, because that is what you are paid for and abstraction without practice is empty.

There is real force in this. An engineer who understands the lifecycle beautifully and cannot debug an Airflow scheduler at 05:41 is not useful at 05:41.

This book's position: learn the stage first, then a tool that occupies it, then a second tool that occupies the same stage. The second tool is what converts tool knowledge into stage knowledge, because it forces you to separate what is essential to the problem from what is an artifact of one implementation. This is why Chapter 22 teaches pandas and Polars and DuckDB for the same job, and why Chapter 10 covers both Delta Lake and Iceberg.

What you give up: depth. A book that taught one tool for each stage could go deeper into each. That is a real cost, and it is why every chapter ends with further reading pointed at the primary documentation.

The five stages, in one sentence each

And the six undercurrents, which are not stages because they do not happen at a point in time — they run beneath all five:

Security · Data management · DataOps · Data architecture · Orchestration · Software engineering

The picture

                                                         ┌──────────┐
   ┌───────────┐    ┌────────┐    ┌───────────┐          │  SERVE   │
   │ GENERATE  │───▶│ INGEST │───▶│ TRANSFORM │─────────▶│          │
   │           │    │        │    │           │          │ BI / ML  │
   │ source    │    │ batch  │    │ clean     │          │ reverse  │
   │ systems   │    │ stream │    │ model     │          │ ETL      │
   └───────────┘    └────────┘    └───────────┘          └──────────┘
        │                │              │                     │
        ▼                ▼              ▼                     ▼
   ╔══════════════════════════════════════════════════════════════════╗
   ║                            S T O R E                             ║
   ║   object storage · warehouse · lakehouse · queue · cache         ║
   ╚══════════════════════════════════════════════════════════════════╝
   ┌──────────────────────────────────────────────────────────────────┐
   │ security │ data mgmt │ DataOps │ architecture │ orchestration │ SWE │
   └──────────────────────────────────────────────────────────────────┘

In words: generate, ingest, transform, and serve run left to right as a flow. Store runs beneath all four, because every one of them reads from and writes to storage — that is what makes it a substrate rather than a step. Beneath everything, the six undercurrents apply at every stage simultaneously.

If you take one image from this chapter, take that one. The lifecycle is a flow over a substrate, not a chain of five links.

2.2 Generate: The Stage You Do Not Control

Data comes from somewhere. That somewhere is almost always a system built by someone else for a purpose that is not analytics.

This is the defining property of the Generate stage: you are a guest. Kestrel's orders table exists to make checkout work. The clickstream exists so the front-end team can debug user journeys. The carrier API exists so customers can see where their package is. That any of it is useful for analytics is a happy accident that the systems' owners are under no obligation to preserve.

What you need to know about a source system

Six questions, and you should be able to answer all six about every source you depend on. Most teams cannot answer three.

1. What is the system of record for this fact? Not "where can I get order data" — which system is authoritative when two disagree. At Kestrel the order service is authoritative for what was ordered, the payments processor for what was collected, and the warehouse system for what shipped. When they disagree — and they will, constantly, because they are eventually consistent with each other through business processes rather than through a protocol — you need to know in advance whose answer wins for which question.

2. What is the update pattern? Are rows inserted and never changed (append-only), updated in place (mutable), or hard-deleted? This single answer determines your entire ingestion strategy. Append-only is easy. Mutable is manageable with a reliable updated_at. Hard deletes are the case that breaks naive ingestion completely, because a deleted row leaves no trace to detect — it is simply absent on the next read, and "absent" is indistinguishable from "not in this batch."

At Kestrel, orders is mutable (status transitions), order_items is mostly immutable but can be deleted when a customer edits an order before payment, and inventory is updated constantly. Those three different patterns need three different ingestion approaches, covered in Chapters 13 and 14.

3. Is there a reliable change timestamp? updated_at is the backbone of incremental ingestion and it fails in four specific ways: it is set by the application rather than the database and so misses direct SQL updates; it is set at transaction start rather than commit, so rows can commit out of timestamp order; it has second granularity in a system doing hundreds of writes per second; or it is simply not maintained on some code path nobody remembers. Chapter 13 §13.4 covers all four and how to detect them.

4. What is the schema and who can change it? Is there a review process? Will you be told? At most companies the honest answer is "no" and "no," which is why Chapter 17 exists.

5. What load can it take, and when? A production database serving checkout has capacity you are borrowing. Kestrel's peak is 6.28× its average, and the acceptable extraction window is exactly the inverse of the business's peak.

6. What does the data mean? The one nobody asks. What does status = 'picked' mean, exactly? Is placed_at when the customer clicked, when the order was written, or when payment authorized? Is quantity ever negative? These are answerable only by asking the team that owns the system, and the answers are rarely written down anywhere.

🏭 From the Pipeline — The column that meant two things

A discount_cents column had been populated by the checkout service since 2019. In 2023 a new promotions service began writing to the same column, but with the opposite sign convention: checkout wrote discounts as positive numbers to be subtracted, promotions wrote them as negative numbers to be added.

Both services were internally consistent. Both had tests. The analytics pipeline subtracted discount_cents unconditionally, so orders touched by the new service had their discounts added to revenue instead of subtracted.

The effect was small — about 0.4% of orders in the first month — and grew as the promotions service rolled out. It was found nine months later, by which point the trend line in a promotional-effectiveness analysis was pointing in the wrong direction and a marketing budget had been reallocated on the strength of it.

No pipeline test would have caught this. The values were valid integers in a plausible range. The only defense is question six — knowing what a column means — and the only mechanism that scales is the data contract in Chapter 17.

Kestrel's three generators

Three sources, three completely different ingestion problems. That is typical, and it is why Chapter 13 through Chapter 16 are four separate chapters rather than one.

2.3 Ingest: Getting Data In

Ingestion is where the lifecycle stops being conceptual. It is also, by a wide margin, where the incidents are.

The stage owns one question — how does data get from there to here, reliably and repeatably — and five decisions:

Batch or streaming? Chapter 3 §3.2 treats this properly. The short version, which will annoy people: most systems that should be batch are built streaming, because streaming is more interesting. The honest test is the one from Chapter 1's Case Study 1 — what decision changes if this is an hour old instead of a minute old?

Push or pull? Do you go and get it, or does the source send it? Pull gives you control over timing and load; push gives you lower latency and removes you from the source's capacity planning. Kestrel pulls from PostgreSQL and the carrier API, and receives pushes from the clickstream.

Full or incremental? Re-read everything every time, or only what changed. Full loads are gloriously simple and idempotent by construction, and they stop being viable somewhere between ten and a hundred gigabytes depending on your window. Incremental is what you graduate to, and it is where the watermark bugs live.

What is the delivery guarantee? At-most-once (may lose data), at-least-once (may duplicate), or exactly-once (expensive, and usually a lie about the whole system even when true about one hop). Chapter 4 §4.5 is the honest treatment. At-least-once plus idempotent writes is the pattern that actually works in production, and the reason is that it moves the hard problem from the transport layer, where it is a distributed systems problem, to the write layer, where it is a DELETE statement.

Where does it land, and in what shape? Raw, exactly as received, or lightly parsed? This book's answer is raw — Chapter 9 §9.4 argues it — because the first question in every investigation is "what did the source actually say," and you can only answer that if you kept it.

⚠️ Failure Mode — The watermark that lost 340 rows a night

The most common incremental ingestion pattern:

sql SELECT * FROM orders WHERE updated_at > :last_watermark; -- then: last_watermark = max(updated_at) from what we just read

This loses rows, silently, forever. Here is the sequence:

text t=100.0 transaction A begins, will write updated_at = 100.0 t=100.5 transaction B begins and commits, updated_at = 100.5 t=100.6 your extract runs: reads B, sets watermark = 100.5 t=101.0 transaction A commits — with updated_at = 100.0, which is < 100.5 A is now invisible to every future extract.

The row was written with a timestamp assigned at transaction start but became visible at transaction commit. Your watermark moved past it while it was still in flight. Nothing errors. Row counts look fine. The row is simply never seen again.

At Kestrel's write rate this pattern lost an estimated few hundred rows a night — about 0.005% of orders — which is small enough to never show up in a row-count check and large enough to break a reconciliation to the cent.

Three fixes, in ascending order of correctness: overlap the window (> watermark - 15 minutes, which turns the problem into a duplicate problem, which idempotent writes already solve); use a monotonic commit-ordered column such as a sequence or log position rather than a wall clock; or use CDC (Chapter 14), which reads the commit log itself and has no watermark at all. Chapter 13 §13.4 works through all three.

2.4 Store: The Substrate, Not a Stage

Store is drawn beneath the other four stages, and getting this right changes how you design systems.

Data is stored at every stage. You store what you ingested (bronze). You store intermediate transformations (silver). You store what you serve (gold). You store the queue that buffers ingest from transform. You store the cache in front of the serving layer. Storage is not a place the data passes through on its way somewhere else; it is the medium every stage reads from and writes to.

What storage decisions you actually own

The system. Object storage, warehouse, lakehouse, relational database, queue, key-value store, search index. Part II is six chapters on this because the choice is expensive to reverse.

The format. Row-oriented (CSV, JSON, Avro) or column-oriented (Parquet, ORC). Compressed with what. Chapter 11 measures it rather than asserting it.

The layout. Partitioning scheme, file sizes, sort order, clustering. This is the decision that most affects cost and is most often made by default. Kestrel's $3,840 Spark job from Chapter 1 §1.6 was a layout decision meeting a query that did not respect it.

The lifecycle policy. How long does data live, and where does it move as it ages? Standard → infrequent access → archive → deleted. This is both a cost decision and — once Chapter 31's retention requirements arrive — a legal one.

💸 Cost Check — What layout costs, using Kestrel's clickstream

One year of clickstream is 341 GB as Parquet + zstd (Chapter 1 §1.5). Storage at the frozen S3 Standard rate of $0.023/GB-month is $7.84/month. Trivial.

Now consider the file count, which is a layout decision nobody thinks of as one. Suppose the streaming consumer commits every 30 seconds:

$$\frac{86{,}400 \text{ s/day}}{30 \text{ s}} \times 12 \text{ partitions} = 34{,}560 \text{ files/day}$$

That is 12.6 million files a year, averaging 27 KB each. Every full-year scan issues at least one GET per file. At the frozen rate of $0.0004 per 1,000 requests:

$$12{,}614{,}400 \times \frac{\$0.0004}{1{,}000} = \$5.05 \text{ per full scan, in requests alone}$$

The requests now cost most of a month of storage every time somebody runs a full-year query — and the real damage is not the $5.05 but the latency and per-file overhead, which is why the small-files problem shows up as a performance incident long before it shows up on a bill.

Compacting to 256 MB files gives roughly 1,332 files a year. Same bytes, same format, same query results, request cost of essentially nothing. The bytes were never the problem. Chapter 9 §9.6 and Chapter 10 §10.5 cover compaction.

2.5 Transform: Making It Mean Something

Transform is where raw data becomes something a person can trust, and it is where most of a data engineer's code lives.

The stage owns two questions that are easy to conflate and should not be: what does this data mean, and what shape does it need to be in. The first is semantics and it is genuinely hard. The second is mechanics and it is usually straightforward.

The layers of transformation

Not all transformation is the same kind of work, and separating the kinds is what makes a warehouse debuggable:

Structural. Parsing JSON into columns, casting types, flattening nesting, renaming to conventions. Mechanical, testable, no business judgment. Should be automatic and boring.

Cleaning. Deduplicating, handling nulls, fixing known encoding problems, dropping test records, standardizing formats. Requires judgment, and every judgment should be written down — why was that row dropped?

Conforming. Making sources agree. The customer ID in the clickstream is a session-scoped anonymous identifier until login; the customer ID in orders is a database primary key. Stitching them is conforming, it is where the hardest bugs live, and it is the reason conformed dimensions exist.

Modeling. Building the fact and dimension tables consumers query — grain, keys, measures, history. Chapter 6 and Chapter 20.

Aggregating. Rollups, summaries, and metric computation, ideally derived from the modeled layer rather than computed independently, so that two dashboards cannot disagree.

Different layers fail differently and want different tests. A structural failure is loud — a cast throws. A conforming failure is silent and expensive. The medallion architecture in Chapter 34 is essentially this list, given layer names and enforced boundaries.

The transformation that will bite you

Joins that change grain. This is the single most common source of silently wrong numbers in analytics, and it deserves its own callout.

⚠️ Failure Mode — The join that changed the grain

fct_order_item has one row per order line. Someone joins promotions to attribute discounts:

sql SELECT oi.*, p.promotion_code FROM fct_order_item oi LEFT JOIN promotions p ON p.order_id = oi.order_id;

If an order can have two promotions applied — a site-wide sale and a personal code — this join emits two rows per order line. Every downstream SUM(net_revenue_cents) doubles for those orders.

The query is valid SQL. It returns rows. The values in them are correct. The grain changed from "one row per order line" to "one row per order line per promotion," and nothing in the type system, the query, or the tooling notices.

Three defenses, and you want all three:

1. Declare the grain of every table in its documentation, in words, at the top.
2. Test it — unique on the declared key, running in CI and in production. Nine lines of YAML in dbt (Chapter 23 §23.4).
3. Assert row counts across a join: if the row count changed and you did not intend it to, the build fails.

This is the same class of error as the duplicate-rows incident in Chapter 1 §1.6, arriving by a different route. The route does not matter. The grain test catches both.

2.6 Serve: The Only Stage the Business Sees

Everything upstream is invisible. Serving is the entire user interface of your work, and data engineers systematically under-invest in it because it feels like someone else's job.

There are five serving patterns, and a mature platform runs most of them:

1. Analytics and BI. A warehouse a BI tool queries. The highest-volume consumer and the one with the strongest opinions about naming, because analysts read column names all day.

2. Machine learning. Training data, and features that must be point-in-time correct — computed using only information available at the moment being predicted. Getting this wrong produces training/serving skew, which is a data engineering failure that presents as a modeling problem. Chapter 32.

3. Reverse ETL. Pushing warehouse data back into operational systems: customer lifetime value into the CRM, a churn score into the support tool, a product ranking into the app. Growing fast, and it inverts your risk profile — a bug in a dashboard is embarrassing, a bug in reverse ETL is in front of customers.

4. Data as a product / data sharing. Serving data to other teams or other companies as a first-class product with an SLA, a schema contract, and a version. Chapter 35.

5. Self-serve. A semantic layer, a metrics store, or a well-documented set of tables that non-engineers can query without asking you. The scaling answer for a four-person team facing a company that keeps growing.

The boundary people draw in the wrong place

The Transform/Serve boundary is the most commonly misplaced line in a data platform, and it is worth being explicit about the two failure modes.

Too far upstream: you build one enormous pre-aggregated table per dashboard. Fast queries, and a combinatorial explosion — every new question needs a new table, every definition change needs fifteen tables rebuilt, and nobody can tell which of the fifteen is authoritative.

Too far downstream: you serve raw-ish tables and let every consumer do their own joins and aggregations. Flexible, and now the definition of revenue lives in nineteen BI queries maintained by five people. This is precisely the situation Kestrel was in before its first data hire (Case Study 1).

The workable answer is a clean, well-modeled star schema at the serving boundary, plus a small number of aggregates for genuinely expensive queries, plus one place — a semantic layer, a dbt metrics definition, a documented view — where each business metric is defined exactly once.

🎓 Interview Angle — "Walk me through how data gets from our app to a dashboard"

An extremely common opener, and the lifecycle is the answer structure. Weak candidates start naming tools. Strong candidates walk the stages and ask questions at each one:

"Starting at generation — what's the source, and is it mutable or append-only? That determines whether I can use a watermark or need CDC. For ingest, what's the freshness requirement, because that decides batch versus streaming, and what delivery guarantee do we need. For storage, what format and partitioning, since that's most of the query cost later. For transform, what's the grain of the table the dashboard needs and where do the business definitions live. For serving, is it BI, ML, or reverse ETL — because point-in-time correctness matters for the second one."

That answer demonstrates a mental model rather than a memorized stack, and — the part that actually gets you the offer — it shows you know which questions determine the design.

📏 Scale Note — the same five stages at three company sizes

The stages do not change. Who owns them does, and that is the whole of what "seniority" means in this field.

text 3 people Kestrel (9) 300 people ───────────────────────────────────────────────────────────────────── Generation the app team a different team forty teams, and is you you can walk to a contract (ch 17) Storage one database Postgres + a lake a platform team + a warehouse owns the substrate Ingestion a cron script 4 mechanisms a self-serve (ch 13-16) onboarding flow Transform the same script a dbt project 50 dbt projects and a semantic layer Serve one dashboard 22 dashboards a BI team, and reverse ETL

What actually changes across the columns is the number of trust boundaries, and every boundary needs the thing Chapter 17 calls a contract. At three people there are none — you are the producer and the consumer, and the agreement is in your head. At Kestrel there are three, and the book's Part III is about all three. At three hundred there are hundreds, and the failure mode is not that any boundary is badly managed but that nobody can enumerate them.

Two things do not scale down, which is worth saying to anyone at a small company reading this book and wondering whether it applies:

The debugging order (§2.5) is the same at every size. Serving first, generation last, because it is ordered by cost to eliminate rather than by causality.

And the undercurrents are the same. A three-person team with no version control and no way to tell what production is running has a DataOps problem, and it has it at exactly the same severity as a three-hundred-person one. The undercurrents are properties, and properties do not have a minimum team size.

Reverse ETL, and the loop that closes

Serving usually means a human reads a number. Sometimes it means a machine acts on one, and that case deserves its own treatment because its failure profile is different from everything else in this chapter.

Reverse ETL pushes data from the warehouse back into an operational system — customer lifetime value into the support tool, a churn score into the CRM, a computed segment into the email platform. The mechanics are unremarkable: a query, a mapping, an API client with the retries and idempotency keys of Chapter 16.

What is remarkable is where the failure lands.

a wrong number in a DASHBOARD
    read by ~11 internal people, with context
    they say "that looks off"
    corrected with an email
    blast radius: a meeting

a wrong number in a SUPPORT TOOL
    read by ~40 agents, with NO context, as fact
    acted on immediately, at the speed the tool is used
    the actions leave the building
    blast radius: unbounded, and not retractable

Reverse ETL inverts your risk profile, and the inversion has three components worth separating.

The consumer loses the ability to disbelieve. An analyst looking at a dashboard has a model of where the number came from. A support agent looking at a field beside a ticket has no such model, and the field carries the authority of the tool it appears in.

The action is immediate and external. A dashboard error is discovered before anybody acts on it, sometimes. A field in an operational tool is acted on within minutes, by someone talking to a customer.

And the mis-keyed case is qualitatively worse than the wrong-value case. A join that attaches the wrong customer's lifetime value to a record does not produce a wrong number; it produces a correct number about the wrong person, shown to an agent who is speaking to a real human being.

The controls that follow are ones you would not bother with for a dashboard:

1. A pre-write assertion with a HARD STOP.
     refuse to write if more than X% of records changed by more than Y%
     refuse outright on any null or out-of-range value
2. An audit trail of what was written, when.
     "what did the agent see on Tuesday" must have an answer
3. A staged rollout.
     write to 1% of records, wait, compare, then the rest -- Chapter 27's
     shapes, applied to data rather than to code
4. And a documented rollback.
     what does "undo" mean when the destination is somebody else's system?
     Frequently the answer is "write the previous values back," which
     requires having kept them.

The general principle, which is the reason this belongs in the lifecycle chapter rather than in an appendix: reverse ETL turns an analytics artifact into a production dependency, and it should inherit production's controls rather than analytics'. The people who build it are usually the analytics team, using analytics tooling, on an analytics release cadence — and the thing they are shipping is operational.

2.7 The Six Undercurrents

The undercurrents are not stages because they do not happen at a point in the flow. They apply everywhere, all the time, and each one fails in a characteristic way.

Security

Least privilege at every stage. The extraction account is read-only on the tables it needs and nothing else. Secrets in a secret manager, never in code — Chapter 1 stated this and every code sample obeys it. Encryption in transit and at rest. Access to serving layers by role.

How it fails: an extraction account with SUPERUSER because it was easier during setup, still there four years later. A .env file committed in 2022 that nobody rotated.

Data management

Governance, cataloging, lineage, quality, metadata, master data, privacy. The undercurrent that answers "what data do we have, what does it mean, where did it come from, who owns it, who can see it." Chapters 30 and 31.

How it fails: nobody can find anything, so five teams build five overlapping versions of the same table, and a deletion request cannot be honored because nobody knows all the places a customer's email was copied to.

DataOps

Applying DevOps to data: automation, monitoring, incident response, continuous improvement. Chapters 25, 26, 27.

How it fails: pipelines deployed by hand from a laptop. Failures discovered by a stakeholder rather than an alert. No postmortems, so the same failure recurs quarterly.

Data architecture

The structural decisions: what systems, what patterns, what boundaries, what trade-offs. Chapter 3 and Part VII.

How it fails: accretion. Nobody made a bad decision; nobody made a decision at all, and eleven systems arrived one emergency at a time. Every one of them is now load-bearing.

Orchestration

Making things run in the right order at the right time with the right failure behavior. Chapter 24.

How it fails: cron. Twenty cron jobs with sleeps between them, hoping the upstream one finished. This works until it does not, and it fails on the day the upstream job is slow — which is always the day the data is biggest.

Software engineering

The one most often missing, and the one Chapter 1 argued is the discipline's actual foundation. Version control, testing, code review, modularity, documentation, CI/CD.

How it fails: a 2,000-line SQL script nobody can modify safely. Copy-pasted transformation logic in nine places, six of them updated when the definition changed. No tests, so every change is a deploy-and-pray.

🔐 Privacy & Governance — The undercurrent that becomes a deadline

Five of the six undercurrents fail gradually. Data management, when it involves personal data, fails on a statutory clock.

A GDPR erasure request obliges a controller to respond within one month of receipt (extendable by two further months for complex requests). That is not a target; it is a legal deadline with a supervisory authority behind it. To honor it you must be able to find every copy of a person's data — in the warehouse, in bronze Parquet files with no index, in Kafka topics inside their retention window, in a BI tool's extract cache, and in the backup taken last Tuesday.

A platform designed with this in mind handles it with a query. A platform that was not designed for it handles it with a two-week investigation, every time, forever.

Chapter 31 covers the mechanics. The point here is architectural: PII location is a design decision made in Part II, and the deadline arrives in Part VI. This book is not legal advice; confirm your obligations with counsel.

🧭 Version Note — what changed between the ETL era and now, and what did not

The lifecycle is presented here as though it were timeless. It is not, and knowing which parts are recent helps when you read older material — which is most of the material.

text ~2010 now ───────────────────────────────────────────────────────────────────── order of operations Transform, THEN Load Load, then Transform why storage was expensive it is not (ch 3 section 3.3) where transform runs a dedicated ETL server the warehouse itself what lands the modelled result everything the source sent the "staging area" a temporary schema bronze, retained (ch 34) schema decided before loading at read, then enforced at the silver boundary

The reordering is not cosmetic. In the ETL era, a field you did not model was a field you did not have — the transformation happened before anything was stored, so the decision about what mattered was made once, in advance, by someone who did not yet know the questions. The whole argument for bronze (Chapter 34) is that this decision is now deferrable, and Exercise 16.23 prices what the deferral is worth.

What did not change: the five questions. Older material asks them in a different order and with different tooling, and the "what does the source actually promise" question from §2.2 is asked in a 1998 data-warehousing text in almost the same words.

A practical reading tip. When you find older material, check whether it assumes storage is scarce. If it does, its conclusions about what to keep are unreliable and its conclusions about what data means are usually still excellent — Kimball's dimensional modelling (Chapter 6) is thirty years old and almost entirely intact.

2.8 Kestrel Through the Lifecycle

Now put it together. Here is Kestrel's order data, traced through all five stages, with the risk at each.

GENERATE   kestrel_app.orders (PostgreSQL 16)
           mutable · status transitions · updated_at maintained by app
           RISK: hard deletes on pre-payment edits leave no trace
              │
INGEST     Debezium CDC → kestrel.orders.cdc.v1 (6 partitions, compacted)
           at-least-once · keyed by order_id · ordered within partition
           RISK: connector restart replays; consumer must be idempotent
              │
STORE      bronze/orders/  Delta, partitioned by ingest_date
           append-only, source-shaped, retained 2 years
           RISK: small files from a 30s commit interval → compaction needed
              │
TRANSFORM  silver.orders   deduplicated on (order_id, lsn), typed, conformed
           gold.fct_order  grain: one row per order
           RISK: joins to promotions and payments can change grain
              │
SERVE      daily_revenue dashboard · DS feature view · finance reconciliation
           SLA: fresh for prior day by 06:00 America/New_York
           RISK: the 15 minutes between 06:00 and 06:15

In words: orders originate in a mutable PostgreSQL table, are captured by Debezium into a compacted Kafka topic, land as append-only Delta files in bronze, are deduplicated and modeled into silver and gold, and are served to three consumers under a 6am freshness deadline. Each stage carries a named, different risk.

Do the same for the other two sources and the picture is instructive:

Read down the risk rows. Notice where the risk concentrates: ingestion for the clickstream, transformation for identity stitching, and generation for the API. Not evenly spread, and not where a tool-first view would put it.

🧱 Kestrel Platform — Increment 2: the lifecycle map and the repo

Two artifacts, and the first one is not code.

1. platform/docs/lifecycle-map.md. Build the table above for all three sources yourself, in your own words, with a risk and an open question at every stage. You will not know all the answers yet — that is the point, and the open questions become your reading list for Parts II and III.

2. The repository skeleton. Directories that match the lifecycle, so a new engineer can find anything in one guess:

text platform/ ├── CHARTER.md # from Chapter 1 ├── docs/ │ └── lifecycle-map.md ├── seed/ # generate: synthetic source data (Ch. 7) ├── ingest/ # ingest: extractors, producers (Ch. 13-16) │ ├── batch/ │ ├── cdc/ │ └── api/ ├── storage/ # store: bucket layout, table defs (Ch. 9-11) ├── transform/ # transform: dbt project + Spark jobs (Ch. 18-22) ├── quality/ # tests and expectations (Ch. 23) ├── orchestrate/ # Airflow DAGs (Ch. 24) └── infra/ # docker-compose, Terraform (Ch. 5, 28)

Organizing by lifecycle stage rather than by tool is deliberate. Tools change; the stages do not, and a repository organized by tool needs restructuring every time one is replaced.

🧪 Try It — trace one number end to end, and time yourself

Pick one number your organisation reports. Revenue, active users, orders, tickets closed — anything a person looks at regularly.

Now trace it backwards through the five stages, and write down how long each hop takes you:

text SERVE which dashboard or report shows it? which query does that dashboard run? ____ min TRANSFORM which table does that query read? which job builds that table, and when? ____ min INGEST where does that table's input come from? how does it get there, and how often? ____ min STORE what format, where, retained how long? ____ min GENERATE which system produces it, and who owns that system? ____ min

Then answer one question at each hop: what would happen if this step silently produced 90% of the rows it should have?

Two things almost always come out of this exercise. The total time is longer than expected — typically thirty to ninety minutes for a number people quote daily — and at least one hop cannot be answered without asking somebody, which locates the boundary this chapter is about.

Keep the timings. Chapter 30 §30.1 asks what a catalog is for, and the answer is: turning that ninety minutes into ninety seconds. Having measured it once makes that argument concrete rather than theoretical.

🔁 Idempotency Check — the property that has to hold at every stage, not at one

Idempotency is introduced properly in Chapter 4 and it belongs here, because it is a lifecycle property rather than a technique — every stage can be run twice, and every stage has a different way of getting that wrong.

text stage run it twice, and the failure is... ──────────────────────────────────────────────────────────────────────── Generate out of your control. The SOURCE may emit an event twice, and the only defence is a producer-assigned id (ch 15). Ingest a duplicated batch. The classic: an append-mode land with no delete step -- Chapter 1's 31 days. Store a partially-written partition read as complete. The write is not the problem; the VISIBILITY is (ch 9, ch 10). Transform a merge with no unique tiebreak, which produces a DIFFERENT answer rather than a duplicated one (ch 18, ch 38). Serve a reverse-ETL push applied twice, which is the only stage where the second run leaves the building.

Notice that the five failures are not variations of one failure. Ingest duplicates, transform diverges, store shows a torn read, and serve escapes. A platform that has solved idempotency at the ingest layer has solved one of five, and the other four are usually assumed to be covered.

The single test that works at every stage is the same one: run it twice over the same input, and diff the output in both directions. One direction passes for an operation that loses rows, which is the most common way this check is done wrongly (Chapter 20 §20.12).

And the artifact that makes it tractable is Exercise 4.21's register — one row per write operation, with its target, its strategy, and its key. It is mostly blank when you first build it, and Chapter 38 asks you to reread it, because a blank row at that point is a write nobody has thought about.

🔎 Read the Plan — the lifecycle, read backwards from a query

A query's execution plan is a map of the last two stages, and reading one backwards tells you more about a platform's lifecycle than any diagram of it.

text FileScan parquet s3://silver/order_lines/ PartitionFilters: [(order_date = 2026-11-27)] <- STORE: partitioned ReadSchema: struct<order_line_id,net_cents,...> <- STORE: columnar Exchange hashpartitioning(customer_id, 200) <- TRANSFORM: a shuffle HashAggregate(keys=[customer_id]) <- TRANSFORM: the grain BroadcastHashJoin <- TRANSFORM: a dimension

Four things about the platform are visible in five lines, and none of them is in the SQL:

The data is partitioned by date and the query knows it. That is a storage decision (Chapter 9) being exercised by a transform, and if PartitionFilters were empty you would be looking at Chapter 1's job.

Only the named columns are read. A columnar format, chosen in Chapter 11, paying for itself here.

The grain is customer_id — the HashAggregate says so — and every additive measure below it is now at customer grain, which is Chapter 6's declaration, enforced by nothing except this query.

And a dimension was small enough to broadcast. That is a modelling fact (dimensions are small, §6.5) showing up as an execution strategy.

The habit worth forming: when a number is wrong, read the plan before reading the SQL. The SQL says what was asked for; the plan says what happened, and the two differ exactly where the lifecycle's earlier stages made a decision the query's author did not know about.

2.9 Summary

The lifecycle is five stages over a storage substrate, with six undercurrents beneath all of it. It is worth internalizing not because it is elegant but because it is stable: it survives every tool replacement, which no stack diagram does.

Generate is the stage you do not control. Source systems exist for other purposes and owe you nothing. Six questions to answer about every one: what is authoritative, what is the update pattern, is there a reliable change timestamp, who can change the schema, what load can it take, and — the one nobody asks — what does the data actually mean. The discount_cents sign-convention story is what question six catches and no test does.

Ingest is where the incidents are, because it is the boundary between your assumptions and someone else's reality. Five decisions: batch or streaming, push or pull, full or incremental, what delivery guarantee, and what shape it lands in. At-least-once delivery plus idempotent writes is the pattern that works, because it relocates the hard problem from the transport layer to the write layer, where a DELETE statement solves it.

Store is a substrate, not a stage. Drawn beneath everything because every stage reads and writes it. Four decisions you own — system, format, layout, lifecycle policy — and layout is the one made by default and paid for on every read. The small-files arithmetic in §2.4 is a layout decision costing request charges and latency that nobody attributes to layout.

Transform has five distinct layers — structural, cleaning, conforming, modeling, aggregating — that fail differently and want different tests. The characteristic failure is a join that changes the grain: valid SQL, correct values, doubled sums, and nothing in the tooling notices. Declare the grain, test it, assert row counts across joins.

Serve is the entire user interface of your work, and the Transform/Serve boundary is the line most often drawn wrong — too far upstream gives a combinatorial explosion of pre-aggregated tables, too far downstream scatters the definition of revenue across nineteen BI queries.

The undercurrents each fail characteristically: a superuser extraction account; five teams building the same table because nobody can find anything; cron with sleeps; a 2,000-line SQL script nobody dares change. And data management, uniquely, fails against a statutory clock once personal data is involved.

The diagnostic habit this chapter exists to build: when a problem arrives, locate the stage before choosing a tool. "Marketing can't see yesterday's attribution" is four different problems with four different classes of answer, and the wrong guess costs a day.

What's next

Chapter 3 takes the arguments the lifecycle exposes and settles them — or, more honestly, states the trade-offs sharply enough that you can settle them for your own situation. Batch versus streaming. ETL versus ELT. Warehouse versus lake versus lakehouse. Each is presented with its losing side named, because the correct answer genuinely depends on things Chapter 3 teaches you to notice.