Exercises: The Data Engineering Lifecycle

The point of these is diagnostic reflex: given a problem, place it in the lifecycle before reaching for a tool. Several problems below are deliberately underspecified — naming what you would need to know is the answer.

Difficulty: ⭐ warm-up · ⭐⭐ standard · ⭐⭐⭐ deeper. Solutions: daggered (†) and odd-numbered problems are worked in appendices/answers-to-selected.md.

Part A — Warm-ups ⭐

2.1 † Name the five stages of the lifecycle in order, and state in one sentence the question each one owns.

2.2 Why is Store drawn beneath the other four stages rather than between Ingest and Transform? Give one concrete consequence of modeling it the wrong way.

2.3 † Name the six undercurrents. For each, write a one-line example of it failing.

2.4 For each of Kestrel's three source systems, state whether the data is append-only, mutable, or subject to hard deletes — and say which of the three is hardest to ingest correctly and why.

2.5 † A stakeholder says: "the marketing dashboard is showing yesterday's data as blank." List the four lifecycle-stage hypotheses, in the order you would check them, with the single query or command you would run to eliminate each.

2.6 What is the difference between cleaning and conforming, as the chapter uses the terms? Give a Kestrel example of each.

2.7 † In one sentence each, define at-most-once, at-least-once, and exactly-once delivery. Then state which combination this book recommends and why.

Part B — Standard ⭐⭐

2.8 Trace a single clickstream purchase event through all five stages for Kestrel, naming at each stage: the system involved, the format the data is in, and one specific way that stage could fail. Use the §2.8 table as a model but do not copy it — write it for the purchase event specifically, which has properties a page view does not.

2.9 † The watermark failure in §2.3 loses rows because updated_at is assigned at transaction start and the row becomes visible at commit. Write the sequence out as a timeline for a case where two long transactions overlap, and state how many rows are lost. Then evaluate each of the three fixes against this specific case — does overlapping the window by 15 minutes always work?

2.10 Compute the small-files arithmetic from §2.4 for a different commit interval. If the clickstream consumer commits every 5 minutes instead of every 30 seconds: (a) How many files per day, and per year? (b) What is the average file size, given 341 GB/year of Parquet? (c) What does one full-year scan cost in GET requests at $0.0004 per 1,000? (d) Is this small enough to stop worrying about compaction? Justify with a number, not a feeling.

2.11 † Write the three defenses against a grain-changing join (§2.5) as concrete artifacts for fct_order_item: the documentation sentence, the dbt test in YAML, and the row-count assertion. You have not learned dbt yet — write the YAML from the shape shown and mark what you are guessing.

2.12 Take a data problem from your own experience or from a job posting you have read. Place it in the lifecycle. Then name which undercurrent, if it had been healthy, would most likely have prevented it.

2.13 † The chapter says the Transform/Serve boundary is drawn too far upstream when you build one pre-aggregated table per dashboard. Kestrel has 22 dashboards. Estimate the maintenance cost of that approach when the definition of net_revenue_cents changes: how many objects rebuild, how many could silently disagree, and how would you find out?

Part C — Deeper ⭐⭐⭐

2.14 The discount_cents sign-convention story in §2.2 was invisible to every test. Design a control that would have caught it within a week. You may not use "ask the upstream team" as the whole answer — describe an automated check. Then state its false-positive rate honestly: how often would your check fire on a legitimate change?

2.15 † The lifecycle is presented as tool-independent. Construct the strongest argument that it is not — that the five stages are an artifact of the batch-warehouse era and that a fully event-driven architecture (Chapter 36) collapses or reorders them. Then say what survives your own argument.

2.16 Kestrel's risk table in §2.8 concentrates risk in ingestion for the clickstream, transformation for identity stitching, and generation for the API. Suppose you have one engineer-month to reduce risk. Allocate it across the three and defend the allocation. What information would change your answer?

2.17 † Reverse ETL "inverts your risk profile" (§2.6). Write the incident scenario: a bug in a reverse-ETL job that pushes customer lifetime value into Kestrel's support tool. What does the customer see? What is the blast radius compared to the same bug in a dashboard? What control would you add that you would not bother with for a dashboard?

Part D — The Kestrel Platform ⭐⭐⭐

2.18 — Increment 2: the lifecycle map and the repository.

(a) The map. Create platform/docs/lifecycle-map.md. For each of Kestrel's three sources, build a five-stage table with four columns: stage, system/format, risk, open question. Do not copy §2.8 — write it yourself, and where you do not know something, write the question rather than a guess. Aim for at least twelve open questions total.

(b) The repository. Create the directory skeleton from the 🧱 callout in §2.8, organized by lifecycle stage. Add a README.md in each directory: one sentence saying what belongs there and which chapter fills it. Commit it.

(c) The trap. In platform/docs/lifecycle-map.md, add a final section: "Where I expect to be wrong." Name three assumptions in your map that you are least confident about. Chapter 38 asks you to reread this.

2.19 † Add platform/docs/source-systems.md. For kestrel_app, answer all six questions from §2.2 as far as you can from the schema in 00-outline.md and Chapter 1. For the three you cannot answer from the material, write down exactly who you would ask and what you would ask them — the phrasing matters, and "can you tell me about the orders table" gets you nothing useful.

Reflection

A. The chapter claims senior engineers locate the stage before choosing a tool, and junior engineers reach for the tool they know best. Think about the last technical problem you debugged. Which did you do? What would locating first have changed?

B. Of the six undercurrents, which one is weakest in an organization you have worked in or studied? What was the visible symptom, and was it correctly attributed?