Exercises: Data Lakes

Several of these are arithmetic you should be able to do in your head by the end of Part II, because you will do a version of it every time someone proposes a partitioning scheme.

Difficulty: ⭐ warm-up · ⭐⭐ standard · ⭐⭐⭐ deeper. Solutions: daggered (†) and odd-numbered problems are in appendices/answers-to-selected.md. Costs use the frozen basis in Appendix J.

Part A — Warm-ups ⭐

9.1 † Name the four ways object storage differs from a filesystem, and give one consequence of each.

9.2 Why is "renaming a directory" in S3 not atomic? Name the failure a reader experiences during one.

9.3 † Describe the five steps a Parquet reader takes, in order. Which two correspond to projection pushdown and predicate pushdown?

9.4 Why is the Parquet footer at the end rather than the beginning? Name one operational consequence.

9.5 † Give four reasons to use separate buckets per layer. Which one decides it?

9.6 What does "landing raw" mean precisely? Why is the payload kept as a string rather than parsed?

9.7 † Why is bronze partitioned by ingest date and silver by event date?

9.8 State the three compaction thresholds from §9.6 and the three properties of a good compaction job.

Part B — Standard ⭐⭐

9.9 Complete the 🧪 Try It from §9.2 against your Kestrel Parquet files. Report: the largest column on disk and why, the number of row groups, and — the important one — how the per-row-group product_id min/max ranges differ between the sorted and unsorted files. Explain what that difference means for a query filtering on product_id.

9.10 † Redo the §9.5 partition-grain table for Kestrel's order_items: 6,480,000 rows a year, roughly 90 bytes per row as Parquet. Compute partitions and bytes-per-partition for yearly, monthly, daily, and hourly grains. Which grain would you choose, and what makes this answer different from the clickstream's?

9.11 A colleague proposes partitioning the clickstream by event_type (7 values) in addition to event_date. Compute the resulting partition count and average size. Then decide, and state the condition under which the answer would flip.

9.12 † Compute the small-files cost for a different commit interval. A consumer commits every two minutes across 12 partitions. (a) Files per day and per year. (b) Average file size, given 341 GB/year. (c) Request cost for a full-year scan. (d) Request latency at 100-way parallelism, at 30 ms per GET. (e) Does this need compaction by the §9.6 thresholds? Show your reasoning.

9.13 Write the compaction job from §9.6 for real, against your local MinIO. It must: compact one partition, sort by a column you justify, write to a temporary prefix, verify the row count matches before swapping, and be safe to run twice. Test the "run twice" case explicitly.

9.14 † §9.4 argues for keeping the payload as an unparsed string, and §9.4's 🔐 callout notes that tokenizing at landing contradicts that rule. Write the decision as an ADR (Chapter 3 §3.7) for a company that may not store raw email addresses: what you land, what you tokenize, where the mapping lives, and what you give up.

9.15 Take the seven practices in §9.7. For each, write the specific artifact or automation that would enforce it at Kestrel — not the principle, the thing you would build. Then rank them by (value ÷ effort) for a four-person team.

Part C — Deeper ⭐⭐⭐

9.16 §9.7's 🏭 callout describes 96 of 200 datasets being archived with nobody noticing. Design the audit that produces that finding. Specify: how you determine ownership, how you determine whether anything reads a dataset (and what your method misses), and how you would handle the case where the only reader is a quarterly job that has not run yet.

9.17 † The _SUCCESS marker is described as "the poor-man's fix" for partial-partition reads. Enumerate the cases where it fails: list at least four ways a reader can still see an inconsistent partition despite the marker. Then explain which of them a transaction log (Chapter 10) fixes and which it does not.

9.18 Design a lake layout for a business unlike Kestrel — a hospital, a logistics company, an ad-tech platform. State the volumes, the query patterns, and the retention obligations, then derive the buckets, prefixes, partitioning, and lifecycle policies. Justify every choice against §9.3 and §9.5, and name the one you are least confident about.

9.19 † §9.5 says to partition by the low-cardinality column and use sort order for the high-cardinality one. Quantify that claim: for Kestrel's clickstream sorted by session_id within a daily partition, estimate how many row groups a query for one session_id would need to read, and compare to the unsorted case. State your assumptions.

Part D — The Kestrel Platform ⭐⭐⭐

9.20 — Increment 9: the lake.

(a) Extend minio-init to create the four buckets' prefix structure from §9.9.

(b) Write platform/ingest/batch/land_parquet.py: read orders from kestrel_app for a date range, write to s3://kestrel-bronze/orders/v1/ingest_date=<today>/ with the §9.4 envelope — _ingested_at, _source, _schema_version, payload preserved.

(c) The part that matters. Write the _SUCCESS marker last, after every part file is closed. Then write a reader that checks for it and refuses to read an incomplete partition. Test the failure path: kill the writer partway through, run the reader, and confirm it declines rather than returning partial data. Record what the reader printed.

(d) Write platform/storage/lifecycle.json with the four buckets' rules and apply it. Verify by listing the rules back — do not trust the apply.

9.21 † Add platform/storage/lake_audit.py, which walks the buckets and reports, per dataset: object count, total size, average file size, largest and smallest object, partition count, and whether every partition has a _SUCCESS marker.

Then run it and record the output in platform/docs/lake-inventory.md. Flag any dataset whose average file size is under 32 MB. This is the report Chapter 33 extends into cost attribution, and the flag is what tells you where compaction is owed.

Reflection

A. §9.7 argues the seven practices are invisible for eighteen months and then arrive as ambiguity rather than failure. What is the equivalent in a codebase you know — a discipline whose absence is free at first and compounds silently?

B. The scratch bucket has a seven-day expiry "with no exceptions," and people have lost work to it. Is that the right rule? What is the alternative, and what does it become after three years?