Exercises: The Lakehouse

Part D is the most hands-on set in Part II. Several exercises ask you to make something fail on purpose, because the failure messages are what you will be searching for at 3am.

Difficulty: ⭐ warm-up · ⭐⭐ standard · ⭐⭐⭐ deeper. Solutions: daggered (†) and odd-numbered problems are in appendices/answers-to-selected.md.

Part A — Warm-ups ⭐

10.1 † Name the five guarantees a table format adds, and for each, the Chapter 9 failure it prevents.

10.2 Name four things a table format does not give you.

10.3 † What two kinds of action appear in a Delta commit? Give the commit contents for a compaction that replaces 34,560 files with 4.

10.4 Why are file statistics stored in the transaction log rather than only in Parquet footers? Quantify the benefit for a table with 14 million files.

10.5 † What does dataChange: false mean, and what breaks without it?

10.6 What single object-storage primitive do atomic commits rest on? What changed about S3 in 2024?

10.7 † Classify each as safe or unsafe: add a nullable column · add a required column · int → long · long → int · rename with column mapping · rename without it · change a partition column.

10.8 State the three ongoing maintenance obligations of a lakehouse table and what happens if each is neglected.

Part B — Standard ⭐⭐

10.9 Two writers both read version 47 and both attempt to commit version 48. Walk through what happens in each case: (a) both appended to different date partitions; (b) both deleted rows matching the same predicate; (c) one appended and one ran OPTIMIZE on the partition being appended to.

10.10 † §10.3's ⚠️ callout describes a commit that times out and is retried, producing duplicate rows. Write the sequence as a timeline. Then implement the "check before retrying" defense in pseudocode, and identify the remaining race it does not close.

10.11 Redo §10.5's maintenance cost model for a larger table: 4 TB/year, hourly OPTIMIZE on the trailing partition, 8 nodes for 20 minutes each run. Compute annual maintenance cost, annual storage cost, and the ratio. Compare to Kestrel's 6.4×, and explain why it moved.

10.12 † mergeSchema = true is recommended in bronze and forbidden elsewhere. Write the incident that occurs when it is enabled in silver: what the upstream does, what the table looks like afterwards, when it is noticed, and by whom.

10.13 Design the erasure runbook for Kestrel, given merge-on-read and deletion vectors. It must cover: the DELETE statements across all three layers, the OPTIMIZE and VACUUM sequence, the verification query that proves completion, and the retention setting that makes the whole sequence finish inside a stated deadline. State the deadline you are designing for.

10.14 † Time travel is bounded by vacuum retention. Kestrel uses 7 days. Construct the scenario in which 7 days is not enough, estimate how often it occurs, and propose a retention value with justification. What does a longer retention cost?

10.15 For each Kestrel table, choose copy-on-write or merge-on-read and justify it: bronze.events, silver.orders (a CDC target), gold.fct_order_item, gold.dim_customer (Type 2).

Part C — Deeper ⭐⭐⭐

10.16 §10.3 says optimistic concurrency fails badly when writers collide constantly, and that the fix is disjoint partition ownership rather than retry tuning. Design a write architecture for Kestrel's CDC pipeline that guarantees disjoint ownership across six Kafka partitions writing to one Delta table. What does your design cost, and what does it forbid?

10.17 † §10.9 makes a principled exception to the raw-payload rule by extracting customer_id. Write the policy that governs future exception requests: what qualifies, who approves, and what is recorded. Then apply it to three plausible requests — extracting session_id for partitioning, event_type for filtering, and country_code for a residency requirement — and decide each.

10.18 Compare the Delta transaction log and the Iceberg metadata tree in detail. For each, trace what a reader does to resolve the current file set, and identify which scales better as the number of files grows and why. Cite the specifications rather than asserting.

10.19 † §10.10 says a lakehouse gives no multi-table transactions. Kestrel needs gold.fct_order_item and gold.dim_customer to be mutually consistent for the 6am dashboard. Design three approaches that tolerate the absence of a cross-table transaction, and state what each gives up.

Part D — The Kestrel Platform ⭐⭐⭐

10.20 — Increment 10: bronze as Delta.

(a) Convert bronze/events to a Delta table using the schema and configuration in §10.9. Write alongside the existing Parquet directory rather than replacing it, then switch readers over.

(b) Concurrent append test. Two processes appending to different event_date partitions simultaneously. Both must succeed. Print dt.history() and confirm two commits.

(c) Conflicting write test. Two processes both running DELETE ... WHERE customer_id = 8841. One must fail. Record the exact error text in platform/docs/known-errors.md. That string is what you will search for at 3am.

(d) Run OPTIMIZE, then dt.history(). Confirm dataChange: false on the optimize commit, and write one sentence explaining why a streaming reader needs that flag.

(e) The erasure trap. Delete one customer's rows. Then verify, with a file listing, that the files containing those rows still exist. Run OPTIMIZE, list again. Run VACUUM RETAIN 168 HOURS, list again. Record at which step the files actually disappear, and write the full sequence into your runbook.

10.21 † Write platform/storage/delta_maintenance.py: a scheduled job that, for each Delta table, reports the current version, file count, average file size, number of commits since the last checkpoint, and total size of tombstoned-but-not-vacuumed files. It must flag any table where average file size is under 32 MB or where tombstoned files exceed 25% of the table.

Then add the part that makes it useful: a --dry-run mode that prints the OPTIMIZE and VACUUM commands it would run, so a human can approve them before a job that rewrites terabytes executes unattended.

Reflection

A. §10.5 argues the maintenance section is the one most introductions skip. Why do you think that is? What other technologies have you adopted whose ongoing cost was not in the introduction?

B. The 🏭 callout describes a ninety-second restore replacing a four-hour rebuild, and notes that the value is decoupling restoring service from fixing the cause. Where else in your experience would that decoupling have changed an incident?