Chapter 10 — Key Takeaways (The Lakehouse)

The page for adopting a table format, and for the day one gets slow.

The five guarantees, and the failure each prevents

It is NOT: a query engine · a catalog · faster by itself · multi-table transactions.

The transaction log

_delta_log/
  00000000000000000047.json          ← one atomic commit: add + remove actions
  00000000000000000050.checkpoint.parquet   ← full state, every 10 commits
  _last_checkpoint

A commit is add + remove actions. A compaction is "remove these 34,560, add these 4," atomically.

Statistics live in the log, so pruning happens without opening a data file — one log read instead of 14.7 million Parquet footers.

dataChange: false on a compaction lets a streaming reader ignore it. Without it, every compaction looks like new data downstream.

Read path: find checkpoint → read it → apply subsequent JSONs → prune by statistics → read survivors. No prefix listing at all.

Atomic commits

The primitive: put-if-absent. Create version $N+1$ only if it does not exist.

🧭 S3 added conditional writes in 2024, removing the need for a DynamoDB log store or commit service. Pre-2024 advice says otherwise. Third time in this book a foundational assumption changed under existing practice — when advice seems oddly elaborate, check its date.

A: read v47 → write files → create 48.json ✓
B: read v47 → write files → create 48.json ✗ conflict
                            └─ disjoint partitions? → retry as 49 ✓
                            └─ overlapping files?  → FAIL

Optimistic concurrency works when collisions are rare. Twenty jobs merging into one table spend their time retrying. The fix is disjoint partition ownership, not retry tuning.

⚠️ A table format gives you atomicity, NOT idempotency. A commit that times out may have succeeded; a blind retry appends the same rows twice. Use txnAppId/txnVersion, replaceWhere, or check before retrying. Chapter 4 §4.5 stands unchanged.

Schema

Enable column mapping at creation. Turning it on later is a protocol upgrade older readers cannot handle — exactly the change you do not want to make under time pressure.

⚠️ mergeSchema = true disables the guarantee you adopted the format for. An upstream typo silently creates a column. On in bronze, off everywhere else.

Maintenance — the section introductions skip

Three settings you must know: checkpointInterval (10) · logRetentionDuration (30d) · deletedFileRetentionDuration (7d). ⚠️ Log cleanup requires a checkpoint to have covered the commits. If checkpointing falls behind, cleanup stops while every setting looks correct.

💸 At Kestrel: ~$600/year of maintenance against $94/year of storage — 6.4×. That is the honest cost of the guarantees.

⚠️ Never lower VACUUM retention during an incident. It deletes files a long query has pinned, failing it with an unhelpful file-not-found. It also bounds your time travel.

Time travel

SELECT * FROM t VERSION AS OF 47;
SELECT * FROM t TIMESTAMP AS OF '2025-11-27 06:00:00';
RESTORE TABLE t TO VERSION AS OF 117;      -- seconds, not a rebuild

Good for: reproducing a report · debugging a change as a diff not a debate · restoring from a bad write (90 seconds vs. 4 hours).

NOT: a backup (same bucket, same blast radius, bounded by retention) · an audit log.

DESCRIBE HISTORY is the FIRST command in a data-correctness incident. Four answers from metadata in under a second, and numOutputRows down the version list is a free volume monitor.

Retention must exceed your realistic detection time — including weekends and holidays.

Deletes and updates

Deletion vectors — a bitmap of removed row positions — are the modern refinement.

⚠️ The erasure trap. With merge-on-read, deleted rows are logically gone and physically present until OPTIMIZE, and still present until VACUUM. If your obligation is physical erasure by a deadline, DELETE alone does not satisfy it. Put the full sequence in the runbook.

Delta vs. Iceberg vs. Hudi

Migration between them is weeks, not quarters — the data files are Parquet either way.

What a lakehouse still does NOT give you

Multi-table transactions · low-latency point lookups · high-concurrency small writes · enforced referential integrity · automatic performance · uniform maturity across engines (verify for your engine and version).

The rule to carry

A table format is an operational commitment, not a file format choice. Adopt compaction, vacuum, and log maintenance in the same sprint as the format. Without them you get a worse outcome than plain Parquet — 4 s → 47 s over six months, 12× the storage, and a failure mode no query profile surfaces.

Time a lakehouse query in four phases, because tools measure only the last:

1. resolve snapshot   ← invisible to query profiles
2. prune files
3. read data          ← the only phase a profile shows
4. compute