Chapter 9 — Key Takeaways (Data Lakes)

The page for designing a lake, or for diagnosing one that has gone wrong.

Object storage is not a filesystem

Inside a Parquet file

PAR1 │ RowGroup 0 [ col chunk → pages, + min/max/null stats ] │ RowGroup 1 … │
     │ FOOTER: schema · row-group stats · column byte offsets │ len │ PAR1

Reader sequence: last 8 bytes → footer → skip row groups by statistics (predicate pushdown) → range-request only named column chunks (projection pushdown) → decode.

Footer at the end so the writer can stream without knowing statistics in advance. ⚠️ A truncated Parquet file is entirely unreadable — it fails cleanly rather than silently returning partial data.

Targets: row group 128 MB · file 128 MB – 1 GB, 256 MB default · zstd.

Layout

Separate buckets per layer — because access control is per bucket (the deciding reason), plus lifecycle policy, blast radius, and cost attribution.

s3://kestrel-bronze/orders/v1/ingest_date=2025-11-28/hour=03/part-00000.parquet
                    └─dataset └─schema version  └─Hive partition       └─ + _SUCCESS

1. Dataset name first · 2. Schema version in the path (free to add, impossible to retrofit) ·
2. Hive-style key=value · 4. Partition columns in descending selectivity · 5. _SUCCESS marker written last

📐 Bronze by ingest date; silver and gold by event date. Partitioning bronze by event date means a late arrival must modify an old partition — a non-atomic rewrite while people are reading. The write path gets the simple option.

Landing raw

Raw = exactly what the source sent + receipt metadata. Payload stays a STRING.

{"_ingested_at": "...", "_source": "kafka:...", "_partition": 7,
 "_offset": 412883901, "_schema_version": "v1", "payload": "{...}"}

Why unparsed: parsing at landing forces a decision about an unparseable record, and every answer loses information — drop it (gone), null it (original lost), fail the batch (availability).

Why keep it: reprocessing · investigation ("what did the source actually say?") · unanticipated questions · audit. At Kestrel: $7.84/month, so cost is not the argument.

🔐 The real argument against forever-retention is erasure, not cost. Three answers: a table format with row-level deletes (Ch. 10) · tokenize at landing (contradicts "unparsed", and right when the alternative is not landing at all) · short bronze retention. Decide before you land.

Partitioning

Kestrel clickstream, 341 GB/year, 66.7 bytes/event:

⚠️ Never partition by a high-cardinality column. customer_id → 1.9M partitions → 1,900 LIST calls before reading a byte, $0.76/query in requests, and a catalog that degrades.

Partition by the low-cardinality column you filter on; use SORT ORDER within the partition for the high-cardinality one.

The small-files problem

30-second commits × 12 partitions:

$$\frac{86{,}400}{30} \times 12 = 34{,}560 \text{ files/day} = 12.6\text{M/year at 27 KB}$$

Compact when: avg file < 32 MB · files/partition > 1,000. Leave alone at 128 MB+.

Three properties of a good compaction job: 1. It sorts — free at write time, improves compression and pruning. Often the larger win. 2. It writes then swaps. Never delete first. 3. It runs on a lag — yesterday's partition, not today's.

Two-tier is the standard answer: small files for freshness, nightly compaction for scans.

⚠️ There is no atomic multi-object operation

Every protocol on plain object storage has a window. Make it small, detectable, and loud:

• Marker last — protects against partial writes, not partial replacements
• Version the prefix + flip a pointer object — the flip is one object write, which IS atomic. This is the pattern that works, and it is a hand-rolled worse version of a transaction log.
• Verify the row count before the flip. A compaction that loses rows and commits destroys its own input.
• Never delete before writing. Old files cost storage; a partial partition costs correctness.

Lake vs. swamp — seven practices

All seven are invisible for eighteen months. The cost arrives as ambiguity, not failure.

Make ownership a precondition for existence, enforced by policy. A catalog nobody is required to fill in stays empty. One audit: 112 of 200 datasets had no answer to "who owns this"; 96 were archived and nobody noticed.

"Nothing is reading it" is a claim about your observation window. 30 days of logs cannot see a quarterly job. Archive with access revoked and let the failure reclaim it — converts an unanswerable question into a cheap, self-resolving one.

Querying a lake — ascending guarantees

The four numbers to know about any object-storage dataset

Object count · total size · average file size · partition count.

Three lines of Python, and they diagnose a class of problem no query profile surfaces. Superlinear scaling in a scan means the cost is per-something-else — count, and find out which.