Chapter 10: The Lakehouse

"A table is not a set of files. A table is a set of files plus an agreement about which ones count right now."

Overview

Chapter 9 ended with a list of things a data lake cannot do, and every item on it has the same root cause: there is no agreement about which files constitute the table at any given moment.

Two writers to one prefix interleave silently. A reader listing mid-write sees a partial result and does not error. A compaction job that deletes before writing leaves a window in which the partition is empty. A row-level delete requires rewriting a file with no way to swap it in safely. None of these are storage bugs — object storage is doing exactly what it promises. They are the consequence of using a file listing as a table definition.

A table format — Delta Lake, Apache Iceberg, or Apache Hudi — fixes this with one idea: maintain a log of which files belong to the table, and make appending to that log the atomic operation. Readers consult the log rather than listing the prefix. Writers append to the log rather than hoping.

That single change buys ACID transactions, schema enforcement, time travel, efficient row-level deletes, and safe concurrent writers, over storage that costs $0.023 per gigabyte-month and that any engine can read. It is the most consequential idea in Part II.

It also costs something, and this chapter is more insistent than most treatments about what. A lakehouse table requires ongoing maintenance that a Parquet directory does not — compaction, log cleanup, and vacuum — and a team that adopts the format without budgeting the maintenance ends up with tables that get slower every week for reasons nobody can explain. §10.5 is that section, and it is the one experienced practitioners say is missing from most introductions.

This is also the most actively changing material in the book. Delta and Iceberg are converging, interoperability is improving fast, and specific claims here may be stale. The mechanisms — a log, optimistic concurrency, snapshots, tombstones — are stable and are what the chapter emphasizes.

In this chapter, you will learn to:

• State the five guarantees a table format adds, and the specific failure each prevents.
• Describe how a transaction log turns files into a table, and trace one commit through it.
• Explain atomic commits over object storage and what happens when two writers collide.
• Distinguish schema enforcement from evolution, and predict which changes are safe.
• Budget the maintenance, and explain what a neglected table looks like.
• Use time travel for the three things it is good for, and recognize the two it is not.
• Choose between copy-on-write and merge-on-read.
• Compare Delta, Iceberg, and Hudi on adoption-deciding properties.

Who needs this chapter: everyone building on object storage. It is on the Platform path and the Streaming path. On Quick Start it can be skimmed — but read §10.1 and §10.5.

10.1 What a Table Format Actually Adds

Five guarantees. Each one maps to a failure from Chapter 9.

1. Atomic commits

The failure it prevents: a reader sees a partial write. Chapter 9 §9.1's non-atomic rename, and Case Study 1's attempts 1 and 2.

With a transaction log, a set of file additions and removals becomes visible all at once or not at all. There is no moment at which half the files are part of the table.

2. Snapshot isolation

The failure it prevents: a reader's result changes underneath it while a query is running. On a plain lake, a long query that lists a prefix and then reads files can read files that were deleted after the listing — producing an error — or miss files added after it, producing a silently incomplete answer.

Each reader pins a snapshot — a specific log version — and sees a consistent set of files for the whole query, regardless of what writers do meanwhile.

3. Schema enforcement

The failure it prevents: a writer appends a file whose schema does not match, and the table becomes unreadable or silently wrong. On a plain lake, writing a Parquet file with quantity as a string into a directory where it is an integer produces a table that some engines refuse to read and others read with nulls.

The format rejects the write.

4. Time travel

The failure it prevents: you cannot answer "what did this table say last Tuesday." Which matters for reproducing a report, debugging a pipeline change, and recovering from a bad write.

5. Efficient row-level deletes and updates

The failure it prevents: an erasure request requires rewriting every file that might contain a person's data, non-atomically, with no transaction. Chapter 3's Case Study 2 eliminated a plain lake on exactly this.

DELETE FROM bronze.events WHERE customer_id = ? becomes a supported operation that takes minutes.

What it does not add

Worth stating early, because expectations run ahead of reality:

• It is not a query engine. Delta and Iceberg are storage layers. Spark, Trino, DuckDB, Snowflake, or BigQuery still does the computing.
• It is not a catalog, though both integrate with one. Something still has to map a table name to a location.
• It does not make queries faster by itself. Layout does — and the format gives you the tools (compaction, Z-ordering) to control layout. Adopting the format and skipping the maintenance makes things slower, which is §10.5.
• It does not give you multi-table transactions. Committing to two tables atomically is generally not supported. Design around it.

📐 Design Decision — Table format, or just a warehouse?

If the guarantees above are what you want, a warehouse (Chapter 8) provides all five and requires no maintenance from you. So why take on the operational burden?

The case for a warehouse: simpler, fewer moving parts, the vendor handles maintenance, better out-of-the-box performance, and mature access control. For a team whose data fits and whose workloads are SQL, this is the right answer and this book says so.

The case for a table format, which is Kestrel's:

• Storage cost at volume. 4.19 TB/year of raw clickstream is materially cheaper on object storage than in warehouse storage.
• Engine independence. Spark, DuckDB, and a warehouse can all read the same bytes. Chapter 8's Case Study 2 used this to make the vendor decision an 8% decision instead of a 100% one.
• Semi-structured data, which warehouses handle as second-class citizens.
• The raw layer. You want bronze cheap, permissive, and outside the warehouse.

What it costs: compaction, vacuum, and log maintenance as standing jobs; a younger ecosystem; and two storage systems to operate rather than one. Chapter 3's ADR-001 records exactly this as the negative consequence, and eighteen months later the team's review found the maintenance burden worse than predicted and the dual access model easier.

10.2 The Transaction Log

The mechanism. Delta Lake's is the easiest to read, so this section uses it; Iceberg's differs in structure and not in principle.

What is on disk

s3://kestrel-bronze/events/
├── _delta_log/
│   ├── 00000000000000000000.json          ← commit 0: create table
│   ├── 00000000000000000001.json          ← commit 1: append 4 files
│   ├── 00000000000000000002.json          ← commit 2: append 4 files
│   ├── ...
│   ├── 00000000000000000010.checkpoint.parquet   ← every 10 commits
│   └── _last_checkpoint
├── event_date=2025-11-27/
│   ├── part-00000-a3f8...parquet
│   └── part-00001-b921...parquet
└── event_date=2025-11-28/
    └── part-00000-c740...parquet

The data files are ordinary Parquet — readable by anything. The _delta_log/ directory is the whole difference.

What is in a commit

Each numbered JSON file is one atomic transaction, containing actions:

{"commitInfo":{"timestamp":1764300862481,"operation":"WRITE",
               "operationParameters":{"mode":"Append"},
               "operationMetrics":{"numFiles":"4","numOutputRows":"14038201"}}}
{"add":{"path":"event_date=2025-11-28/part-00000-c740.parquet",
        "partitionValues":{"event_date":"2025-11-28"},
        "size":247891204,"modificationTime":1764300862000,"dataChange":true,
        "stats":"{\"numRecords\":3509550,
                  \"minValues\":{\"event_ts\":\"2025-11-28T00:00:00Z\",
                                 \"session_id\":\"0000a1\"},
                  \"maxValues\":{\"event_ts\":\"2025-11-28T05:59:59Z\",
                                 \"session_id\":\"3f2b9c\"},
                  \"nullCount\":{\"customer_id\":1284003}}"}}
{"remove":{"path":"event_date=2025-11-28/part-00000-old.parquet",
           "deletionTimestamp":1764300862481,"dataChange":true}}

Three things worth noticing, because each is load-bearing:

add and remove are both actions in one commit. A compaction is "remove these 34,560 files, add these 4" — one atomic transaction. Chapter 9's Case Study 1 spent three attempts building a worse version of this by hand.

Statistics are in the log, not only in the Parquet footers. That means the engine can prune files without opening any of them — the min/max of every column of every file is available from one log read. This is a significant speedup over plain Parquet, where pruning requires reading 14.7 million footers.

dataChange: true distinguishes commits that change data from ones that only reorganize it. A compaction sets dataChange: false, which lets a streaming reader ignore it — otherwise every compaction would look like new data to a downstream consumer and be reprocessed.

Reading the table

1. List _delta_log/, find _last_checkpoint.
2. Read the checkpoint (a Parquet file containing the full state as of that version).
3. Read the JSON commits after the checkpoint and apply them.
4. You now have the exact set of files in the table, with statistics, without listing the data prefixes at all.
5. Prune files by statistics, then read only the survivors.

Step 4 is why lakehouse tables scale where directory listing does not. A table with 14 million files has a log you can read in seconds; listing its prefix is 14,000 paginated API calls.

Checkpoints exist because replaying ten thousand JSON commits is slow. Every ten commits by default, the full state is written as one Parquet file. Reading is then "one checkpoint plus at most nine JSONs."

🔎 Read the Plan — Read the log yourself

The log is JSON. Reading it is the fastest way to make this concrete, and it is a genuinely useful debugging skill — when a table behaves strangely, the log says exactly what happened and when.

```bash

the most recent commit

aws s3 cp s3://kestrel-bronze/events/_delta_log/00000000000000000047.json - \ | python -m json.tool ```

Or from Python:

python from deltalake import DeltaTable dt = DeltaTable("s3://kestrel-bronze/events") print(dt.version()) # current version print(dt.history(5)) # last 5 operations, with metrics print(len(dt.files())) # files in the CURRENT snapshot

Three questions the history answers immediately that are otherwise archaeology:

1. When did this table last change, and what operation was it?
2. How many files does the current version have, and how many did it have ten versions ago? A steadily climbing count with no compaction is §10.5's problem.
3. Was the last write an append, a merge, or an optimize? Which tells you whether a change in row count is expected.

10.3 Atomic Commits Over Object Storage

The hard part, and the reason this is not trivially easy.

Committing means creating the next log file — version $N+1$. Two writers both at version $N$ both want to create ...0000N+1.json. Exactly one must win.

The primitive: put-if-absent

The whole design rests on a single operation: create this object only if it does not already exist. If two writers attempt it, one succeeds and one gets a conflict error. That is a single-object atomic operation, which object stores can provide.

• Google Cloud Storage and Azure Blob Storage have offered conditional writes (precondition on generation or ETag) for years.
• S3 did not, historically, which is why Delta on S3 needed either a single writer, DynamoDB as a locking layer, or a commit coordination service. S3 added conditional writes (If-None-Match: *) in 2024, which removed that requirement.

🧭 Version Note — S3 conditional writes changed the multi-writer story

Before 2024, "multiple writers to a Delta table on S3" required an external coordinator — Delta's S3DynamoDBLogStore, Databricks' managed commit service, or the discipline of a single writer. An enormous amount of documentation, blog content, and StackOverflow advice describes that constraint as though it were permanent.

S3 now supports conditional writes, so the put-if-absent primitive is available natively and the extra coordination layer is no longer needed for this purpose.

Two consequences, and the second is the general one:

1. Check your library version and your configuration. An older Delta or delta-rs may still default to the older behavior, and advice written before 2024 will tell you a coordinator is mandatory.
2. This is the third time in this book that a foundational assumption changed underneath a body of practice — S3 consistency in 2020 (Chapter 4 §4.4), and now this. When advice seems oddly elaborate, check its date. The elaborate part is often a workaround for something that has since been fixed.

Optimistic concurrency

Writers do not lock. They proceed optimistically and reconcile at commit time:

  Writer A                          Writer B
  ────────                          ────────
  read version 47                   read version 47
  compute changes                   compute changes
  write data files                  write data files
  try to create 48.json ──── ✓      try to create 48.json ──── ✗ conflict
                                    │
                                    read 48.json — what did A do?
                                    │
                                    ├─ A touched different partitions
                                    │  → RETRY: create 49.json ✓
                                    │
                                    └─ A modified files I also modified
                                       → FAIL: raise a concurrency error

In words: both writers work independently and race to create the next log file. The loser reads the winner's commit and decides whether the two changes conflict. Appends to different partitions do not conflict and the loser simply retries; overlapping modifications do conflict and the loser fails.

The design implication that matters: optimistic concurrency works well when writers rarely collide and badly when they collide constantly. Two jobs appending to different date partitions never conflict. Twenty jobs merging into the same table will spend most of their time retrying — and the answer is to restructure so that writers own disjoint partitions, not to tune the retry policy.

⚠️ Failure Mode — The commit that succeeded twice

A subtler failure than a conflict, and it catches people who assume the transaction log makes everything idempotent.

A writer writes its data files, then attempts the commit. The commit request times out — the classic partial failure from Chapter 4 §4.1: it may or may not have succeeded and the writer cannot tell.

If the writer retries and the first commit did succeed, it now creates version $N+2$ containing the same add actions — the same rows, appended twice. The table is internally consistent and the data is wrong. This is Chapter 1's duplicate-rows incident, arriving through the transaction log rather than around it.

The defenses:

• Transaction identifiers. Delta supports txnAppId and txnVersion in the commit; a commit with an application id and version already present is ignored. This is the format-level answer and it is what streaming writers use.
• Idempotent write patterns. replaceWhere or a partition overwrite is idempotent by construction — running it twice produces the same state. An append is not.
• Check before retrying. On a timeout, read the current version and inspect whether your commit landed before retrying blindly.

A table format gives you atomicity, not idempotency. Those are different properties and the distinction costs people real incidents. Chapter 4 §4.5's rule stands unchanged: at-least-once plus idempotent writes.

10.4 Schema Enforcement and Evolution

Enforcement

By default, a write whose schema does not match the table's is rejected.

# The table has quantity: INTEGER
df_with_string_quantity.write.format("delta").mode("append").save(path)
# AnalysisException: Failed to merge incompatible data types IntegerType and StringType

That failure is the feature. On a plain Parquet directory the write succeeds and the table becomes unreadable by strict engines and silently null-producing on lenient ones — a Chapter 2 §2.2-class problem discovered weeks later.

Evolution

Schemas do change, and the format distinguishes safe changes from unsafe ones.

Column mapping is worth understanding because it is the difference between a rename being free and being a full rewrite. Without it, a Parquet file's columns are identified by name, so renaming cust_id to customer_id orphans every existing file. With it, columns carry a stable identifier and the name is metadata — so renames and drops become metadata operations.

Enable column mapping when you create the table. Turning it on later is possible and is a protocol upgrade that older readers cannot handle, which is exactly the kind of change you do not want to make under time pressure.

The pattern that gets people into trouble

mergeSchema = true tells the writer to add any new columns it sees rather than rejecting them.

df.write.format("delta").option("mergeSchema", "true").mode("append").save(path)

Convenient, and it disables the guarantee you adopted the format for. An upstream typo — custmer_id instead of customer_id — silently creates a new column, and now the table has both, each half-populated, and nothing failed.

This book's position: mergeSchema on in bronze, off everywhere else. Bronze's job is to accept what arrives (Chapter 9 §9.4); silver's job is to enforce a contract. Turning it on in silver converts a loud, immediate failure into a subtle one discovered in a dashboard — Chapter 2's Case Study 1 again.

10.5 Maintenance: The Section Most Introductions Skip

A lakehouse table is not a place you put files. It is a system with ongoing obligations, and a team that adopts the format without budgeting them ends up with tables that get slower every week for reasons nobody can explain.

Three obligations.

1. Compaction

Same problem as Chapter 9 §9.6, and the format makes it a command instead of a hand-rolled protocol:

OPTIMIZE bronze.events WHERE event_date >= '2025-11-01';

-- with a sort order, which is where most of the benefit is
OPTIMIZE bronze.events ZORDER BY (session_id, customer_id);

Z-ordering interleaves multiple columns' bits so that rows near each other in several dimensions land in the same file — useful when you filter on more than one high-cardinality column. It is more expensive to compute than a plain sort and worth it when the access pattern genuinely uses several columns.

Cadence: nightly for actively written tables, and only on partitions no longer being written.

Cost: it reads and rewrites the data. Compacting a 341 GB table is 341 GB read and written, and that is a real compute bill you should attribute rather than absorb.

2. Log cleanup and checkpoints

Every commit adds a JSON file. Checkpoints are written every ten commits by default, and old JSONs are cleaned up after the log retention period.

A streaming writer committing every 30 seconds produces 2,880 commits a day. Without checkpoints and cleanup, the log itself becomes the problem: reading the table means replaying hundreds of thousands of JSON files.

Both Delta and Iceberg do this automatically and their defaults may not suit you. Check delta.logRetentionDuration (default 30 days) and delta.checkpointInterval, and know what yours are — this is exactly the kind of default that is load-bearing and unexamined, as Chapter 3's Case Study 1 found with a changelog topic.

3. Vacuum

remove actions tombstone files; the files remain on disk so that time travel works. VACUUM deletes files no longer referenced by any retained version.

VACUUM bronze.events RETAIN 168 HOURS;   -- 7 days

Without vacuum, storage grows without bound. Every compaction doubles the physical footprint of the compacted partitions until the old files age out.

⚠️ Failure Mode — VACUUM with a short retention, during a long query

VACUUM ... RETAIN 0 HOURS deletes every file not in the current version. It is the command people reach for when storage is growing and they want it back now.

A query that started before the vacuum has pinned an older snapshot and is still reading those files. The vacuum deletes them mid-read. The query fails with a file-not-found error that names a Parquet path and explains nothing about why.

Delta refuses RETAIN below 168 hours (7 days) by default, and there is a configuration flag to override the check. The flag exists for genuine reasons and it is reached for during storage incidents, which is precisely when a long analytical query is most likely to be running.

Three rules:

1. Set retention longer than your longest-running query, with margin. Seven days is the default for a reason.
2. Never lower it during an incident. That is when you have the least information about what is running.
3. Retention also bounds your time travel. Vacuuming to 7 days means you cannot query the table as of last month, regardless of what the log says — §10.6.

💸 Cost Check — The maintenance nobody budgets

Kestrel's bronze events table: 341 GB/year, streaming writer committing every 30 seconds.

Nightly OPTIMIZE of yesterday's partition — 934 MB read and written. Small, and it runs 365 times a year: 341 GB read and 341 GB written annually just to maintain layout.

On a small compute cluster, say 4 nodes for 10 minutes a night at the frozen $2.400/node-hour:

$$4 \times \frac{10}{60} \times \$2.400 \times 365 = \$584 \text{ per year}$$

Storage during the retention window. After compaction, old and new files coexist for 7 days. With one week of compacted partitions duplicated:

$$7 \times 934 \text{ MB} = 6.5 \text{ GB extra} \times \$0.023 = \$0.15/\text{month}$$

Request costs for log reads: every reader reads the checkpoint and recent JSONs. At a few thousand queries a month this is cents.

Total maintenance: roughly $600/year against $94/year of storage for the table itself.

The maintenance costs more than six times the storage. That is the number to internalize, and it is the honest cost of the guarantees in §10.1. It is still much cheaper than the alternatives — and a team that budgets $94 and is surprised by $600 has been misled by an introduction that skipped this section.

10.6 Time Travel

Query the table as of a past version or timestamp:

SELECT * FROM bronze.events VERSION AS OF 47;
SELECT * FROM bronze.events TIMESTAMP AS OF '2025-11-27 06:00:00';

DeltaTable("s3://kestrel-bronze/events", version=47).to_pandas()

The three things it is genuinely good for

1. Reproducing a report. "Yesterday's dashboard said $2.4M and today it says $2.3M." Query the table as it was when the dashboard ran, and you can tell whether the pipeline changed or the data did. Without time travel this is unanswerable, and it is one of the most common awkward questions a data engineer receives.

2. Debugging a pipeline change. Run the new transformation against the exact input the old one saw. This converts "it produces different numbers" from a debate into a diff.

3. Recovering from a bad write. A restore is a metadata operation:

RESTORE TABLE gold.fct_order_item TO VERSION AS OF 118;

Seconds, not a rebuild. This is the single most reassuring property of the format the first time you need it.

The two things it is not

It is not a backup. Time travel is bounded by vacuum retention — typically seven days — and it lives in the same bucket with the same permissions and the same blast radius. A bucket deletion, a credential compromise, or a region failure takes the history with it. You still need backups, and teams reliably conflate the two.

It is not an audit log. It records that a version changed, not who changed it or why in business terms. commitInfo carries the operation and its metrics, which is genuinely useful, and it is not "which user requested this deletion under which ticket." Chapter 30 covers audit properly.

🏭 From the Pipeline — The restore that took ninety seconds

A team deployed a transformation change at 22:00 with a bug in a WHERE clause. The nightly job ran, wrote gold.fct_order_item, and dropped roughly 40% of rows — orders with a null promotion_key, which the new predicate excluded.

The 06:00 dashboard was wrong. The on-call engineer was paged at 06:41.

Before the lakehouse migration, the recovery would have been: identify the bad write, reconstruct the correct data from silver, rebuild the table, and verify. Their runbook estimated four hours.

What actually happened:

```sql DESCRIBE HISTORY gold.fct_order_item LIMIT 3; -- version 118, 22:14, WRITE, numOutputRows 4,032,499 <- the bad one -- version 117, 21:03, WRITE, numOutputRows 6,483,117 <- the good one

RESTORE TABLE gold.fct_order_item TO VERSION AS OF 117; ```

Ninety seconds. The dashboard was correct before 07:00, and the SLA was met.

The bug still had to be fixed, and it was fixed during working hours by people who had slept. The value of time travel is not that it fixes bugs. It is that it decouples restoring service from fixing the cause, which is the single most useful property an on-call engineer can have.

One caveat the team recorded: the restore was possible because the bad version was within the vacuum retention window. A version older than retention is gone. Chapter 26 §26.5 makes "check the retention window covers your realistic detection time" a runbook item.

10.7 Row-Level Deletes and Updates

Parquet files are immutable, so changing one row means rewriting a file. Two strategies exist, and the choice is the main performance decision a lakehouse table presents.

Copy-on-write

Rewrite the affected files immediately. The table always contains exactly the current data.

Write cost: high — a single-row delete rewrites an entire file, potentially 256 MB to change 200 bytes. Read cost: minimal — readers read current files and nothing else.

Right for: read-heavy tables with infrequent updates. Kestrel's gold layer.

Merge-on-read

Write a small delete file or deletion vector recording which rows are logically removed, without touching the data file. Readers apply it at query time.

Write cost: low — a delete is a small file. Read cost: higher — every read applies outstanding deletes, and it degrades as they accumulate until compaction reconciles them.

Right for: write-heavy tables with frequent small updates. CDC targets (Chapter 14), streaming upserts.

The comparison

Deletion vectors are the modern refinement: instead of a delete file listing removed rows, a compact bitmap marks removed row positions within a file. Much smaller, faster to apply, and now the default in recent Delta versions for many operations.

The privacy case

The operation Chapter 3's Case Study 2 chose the architecture for:

DELETE FROM bronze.events  WHERE customer_id = 8841;
DELETE FROM silver.events  WHERE customer_id = 8841;
DELETE FROM gold.fct_order_item WHERE customer_key IN (
    SELECT customer_key FROM gold.dim_customer WHERE customer_id = 8841);

Minutes, transactional, and — the part that matters for compliance — verifiable: the log records that it happened, and a subsequent query returns nothing.

One trap that catches teams during their first erasure request. With merge-on-read, the deleted rows are logically gone and physically still present until compaction rewrites the files, and still present after that until VACUUM removes the tombstoned files. If your obligation is physical erasure within a deadline, DELETE is not sufficient — you must also OPTIMIZE and VACUUM with a retention shorter than your deadline. Write that sequence into the runbook. This book is not legal advice; confirm with counsel what your obligation actually requires.

10.8 Delta, Iceberg, and Hudi

Three formats. The differences are narrowing fast, which is itself the most useful thing to know.

Delta Lake. Originated at Databricks, open-sourced, now under the Linux Foundation. Transaction log as ordered JSON plus Parquet checkpoints. Strongest Spark integration; excellent Python support via delta-rs, which needs no JVM at all — the reason this book uses it locally.

Apache Iceberg. Originated at Netflix, an Apache project from the start. Metadata is a tree: a metadata file points at a manifest list, which points at manifests, which point at data files. More layers, and it scales to very large tables well and supports hidden partitioning — the partition transform is recorded in metadata, so queries do not need to reference partition columns explicitly. Broadest engine support; increasingly the neutral choice.

Apache Hudi. Originated at Uber. Built around upserts and incremental pulls from the start, with strong record-level indexing. The best fit for heavy CDC-style workloads and the smallest ecosystem of the three.

Iceberg's partition evolution is the one genuine capability gap. Changing a partitioning scheme in Delta means rewriting the table; Iceberg records the change in metadata and applies the old scheme to old data and the new one to new data. If you expect to get partitioning wrong — and Chapter 4's Case Study 1 suggests you might — that is a real advantage.

🎓 Interview Angle — "Delta or Iceberg?"

A question designed to see whether you have opinions or preferences. The strong answer refuses the binary and names conditions:

"They're converging, so I'd decide on ecosystem fit rather than features. If the shop is Spark-and-Databricks-centric, Delta is the path of least resistance and delta-rs makes the Python story genuinely good. If there are several engines — Trino, Flink, Snowflake, plus Spark — Iceberg has broader native support and is becoming the neutral choice.

The one capability gap I'd actually weigh is partition evolution. Iceberg lets you change the partitioning scheme without rewriting the table; in Delta that's a rewrite. If I thought there was a real chance of getting partitioning wrong — and at a company whose data shape is still changing, there is — that's worth something.

And whichever we chose, the migration cost between them is weeks rather than quarters, because the data files are Parquet either way. The metadata layer is what differs. So I'd pick, write down what would make me switch, and not spend a month on the decision."

The last paragraph is what distinguishes it — Chapter 3 §3.1's reversibility framing, applied.

10.9 Kestrel's Bronze as Delta

The increment, and the decisions in it.

# platform/storage/create_bronze_events.py
from deltalake import DeltaTable, write_deltalake
import pyarrow as pa

SCHEMA = pa.schema([
    ("_ingested_at",    pa.timestamp("us", tz="UTC")),
    ("_source",         pa.string()),
    ("_partition",      pa.int32()),
    ("_offset",         pa.int64()),
    ("_schema_version", pa.string()),
    ("payload",         pa.string()),        # raw, unparsed. Chapter 9 §9.4
    ("event_date",      pa.date32()),        # partition column
    ("customer_id",     pa.int64()),         # extracted ONLY for deletion.
                                             # Chapter 10 §10.7 -- an erasure
                                             # request needs a predicate, and
                                             # you cannot filter on a field
                                             # inside an unparsed string.
])

write_deltalake(
    "s3://kestrel-bronze/events",
    data=pa.table([], schema=SCHEMA),
    partition_by=["event_date"],
    mode="error",
    configuration={
        # Column mapping ON at creation. Turning it on later is a protocol
        # upgrade older readers cannot handle -- exactly the change you do not
        # want to make under time pressure.
        "delta.columnMapping.mode": "name",
        # Retention longer than the longest realistic query, and long enough
        # that a Monday-morning discovery of a Friday-night bug is still
        # recoverable. Chapter 10 §10.5 and §10.6.
        "delta.logRetentionDuration": "interval 30 days",
        "delta.deletedFileRetentionDuration": "interval 7 days",
        # Deletion vectors: erasure requests are cheap, at the cost of
        # compaction urgency. Chapter 10 §10.7.
        "delta.enableDeletionVectors": "true",
    },
)

The one decision worth arguing about is customer_id as a real column beside the unparsed payload. It contradicts Chapter 9 §9.4's "payload stays a string" rule.

It is there because an erasure request needs a predicate, and you cannot write WHERE customer_id = ? against a field inside a JSON string without parsing every row of the table. The alternative — a full scan and rewrite per request — is exactly what the architecture was chosen to avoid.

So the rule becomes: land raw, and extract only the columns required for partitioning and for deletion. That is a principled exception rather than a slide back into parsing at landing, and it is worth writing down as such — because the next request will be to extract "just one more" field.

🧱 Kestrel Platform — Increment 10: bronze as Delta

(a) Convert bronze/events from a Parquet directory to a Delta table. Both formats can coexist — write the Delta table alongside and switch readers over — which is the safe migration.

(b) Write a concurrent-append test. Two processes appending to different event_date partitions simultaneously. Both should succeed, with one retrying. Print dt.history() afterwards and confirm you see two commits.

(c) Write a conflicting write test. Two processes both running DELETE FROM bronze.events WHERE customer_id = 8841. One should fail with a concurrency error. Record the exact error text — that string is what you will search for at 3am, and knowing it in advance is worth ten minutes now.

(d) Run OPTIMIZE, then dt.history(). Note that the optimize commit has dataChange: false — and explain in a sentence why a streaming reader needs that flag.

(e) Delete one customer's rows, then verify: the rows are logically gone, and the files containing them still exist until OPTIMIZE and VACUUM run. This is the §10.7 trap, and seeing it once is what makes you write the full sequence into the runbook.

🧪 Try It — make two writers collide, on purpose

The concurrency guarantee is the hardest one to believe without seeing it fail, and it takes about ten minutes to see both outcomes.

bash cd part-02-storage/chapter-10-the-lakehouse/code python delta_maintenance.py --all --dry-run

Then, in two terminals:

1. Two appends to different partitions, simultaneously. Both must succeed, and dt.history() must show two commits. This is the case people assume will conflict and does not — the log records disjoint file sets and neither invalidates the other.

2. Two DELETE ... WHERE customer_id = 8841, simultaneously. One must fail.

text ConcurrentDeleteReadException: This transaction attempted to delete one or more files that were deleted (for example ...) by a concurrent update.

Copy that string into platform/docs/known-errors.md, verbatim. It is what you will paste into a search box at 3 a.m., and the exception's name is the fastest route to what actually happened.

3. Now the one that teaches the most: run OPTIMIZE while an append is in flight. It should succeed, because OPTIMIZE commits with dataChange: false and does not conflict with a data write. Confirm it in dt.history() — and then say, in one sentence, why a streaming reader needs that flag (§10.5).

4. And the one that surprises everyone: delete a customer's rows, then list the files.

text after DELETE the files are STILL THERE. The log tombstoned them. after OPTIMIZE new files exist; the old ones are still there. after VACUUM NOW they are gone.

Record which step actually removes the bytes, and put the sequence in your runbook. Chapter 31's erasure obligation is not satisfied by a DELETE, and this is the ten minutes in which that stops being an abstraction.

🔐 Privacy & Governance — time travel is a deliberate delay on your own deletion

Every guarantee in this chapter has a privacy cost and one of them is direct.

Time travel works by not deleting. A DELETE writes a tombstone; the files stay until VACUUM runs past the retention. So a retention of 30 days means an erasure request is not complete for 30 days, and a retention of 400 days means it is not complete for more than a year.

text retention time-travel window erasure completes after ──────────────────────────────────────────────────────── 7 days a week 7 days 30 days a month 30 days 400 days a year and a bit 400 days <-- defensible? probably not

The trade is real in both directions and it is worth writing into an ADR rather than leaving as a default: a short retention makes month-end investigations impossible (Exercise 10.14) and a long one extends every deletion obligation by the same amount.

Kestrel's resolution, and it generalises: 30 days of time travel, plus a zero-copy clone taken on the first of each month and retained for thirteen. The clone is a small, enumerable set of objects that an erasure job can be pointed at explicitly, rather than an unbounded tail of tombstoned files that a VACUUM will get to eventually.

Two operational details that are easy to get wrong:

VACUUM with a retention shorter than your longest reader deletes files out from under a running query. The default guard exists for a reason; disabling it to "finish the deletion" is how a deletion becomes an incident.

And a clone is not a copy — until the source changes. A zero-copy clone shares files, so deleting from the source does not delete from the clone, and the clone is therefore a place personal data survives an erasure. It must be in the manifest (Chapter 31), and it is exactly the kind of object a hand-written manifest omits.

📏 Scale Note — what gets slow, and at what size

A transaction log is a file, and a file that grows without bound becomes the bottleneck. Four things degrade at predictable points, and none of them is the data volume.

text quantity comfortable awkward the symptom ───────────────────────────────────────────────────────────────────────── commits since a checkpoint < 100 > 1,000 slow reader STARTUP: every commit replayed files in the table < 100,000 > 1,000,000 slow planning; the manifest is enormous partitions < 10,000 > 100,000 listing and pruning both degrade tombstoned-but-not-vacuumed bytes < 25% > 50% you are paying for data nobody can read

Row 1 is the one that surprises people, because it has nothing to do with size: a table written to every minute accumulates 1,440 commits a day, and a reader that starts cold must replay all of them unless a checkpoint exists. The fix is automatic on every modern implementation and it is a setting, which means it can be wrong.

Row 2 is the small-file problem (Chapter 9) with a metadata layer on top, and the metadata makes it worse rather than better: now every file has a log entry too.

Row 3 is why high-cardinality partitioning is even more destructive here than in a plain lake. A plain lake with a million prefixes is slow to list; a table format with a million partitions is slow to list and has a million partition entries in every manifest it reads.

The maintenance that addresses all four is the same three commands, and Exercise 10.21's job is the version that tells you when to run them:

```bash OPTIMIZE

# rows 2 and 3 VACUUM RETAIN 720 HOURS # row 4

checkpointing: automatic; verify the interval # row 1

```

Run the reporting job weekly and the maintenance when it flags, rather than the reverse. A scheduled OPTIMIZE on a table that does not need one is a full rewrite of terabytes for nothing, which is the most expensive way to be tidy.

🔁 Idempotency Check — MERGE is idempotent; MERGE on the wrong key is not

A table format gives you an atomic MERGE, and atomicity is not idempotency. Running the same merge twice is safe only if the match condition identifies the same rows both times.

```sql -- SAFE to run twice: the key is stable and unique in the source MERGE INTO silver.orders t USING staged s ON t.order_id = s.order_id WHEN MATCHED THEN UPDATE SET ... WHEN NOT MATCHED THEN INSERT ...;

-- NOT safe: the source has two rows per order_id, and which one wins -- depends on read order (ch 18 section 18.7) -- -> the first run and the second run can produce DIFFERENT tables ```

Two conditions have to hold, and only the first is usually checked:

The key must be unique in the source. Most engines raise an error when a target row matches multiple source rows — which is the good outcome — and some silently pick one. Deduplicate before the merge, with a total ORDER BY, and assert it.

And the merge must not depend on the target's current state in a way that accumulates. SET quantity = t.quantity + s.quantity is an accumulator, not an upsert, and running it twice doubles the increment. It looks like a merge and behaves like an append.

The test, as always:

sql -- run the merge, snapshot, run it again, diff both directions SELECT * FROM v1 EXCEPT SELECT * FROM v2; SELECT * FROM v2 EXCEPT SELECT * FROM v1;

And a lakehouse-specific bonus: dt.history() shows you both runs, with their operation metrics. A second merge that reports numTargetRowsUpdated greater than zero over unchanged input is the signal, and it is available without diffing anything.

10.10 What a Lakehouse Still Does Not Give You

Honest limits, because expectations run ahead of the technology.

Multi-table transactions. You cannot atomically commit to fct_order_item and dim_customer together. Design so that consumers tolerate momentary inconsistency, or serialize the writes and accept a window.

Low-latency point lookups. Still a scan-oriented format. WHERE order_id = 88214 reads at least one whole file. Use an operational store (Chapter 12) for that access pattern.

High-concurrency small writes. Every commit is a log file. Committing a thousand times a second is not viable; batch first.

Enforced referential integrity. Same as a warehouse (Chapter 8 §8.1) — no foreign keys, no cascade. Chapter 23's tests.

Automatic performance. The format gives you the tools. Layout is still your job, and neglecting it makes things worse than a plain Parquet directory, because you now have both the small files and a very long log.

Maturity in every engine. Support varies by engine and version, and "supports Iceberg" can mean read-only, or read-write, or write-with-caveats. Verify for your specific engine and version before designing around it.

10.11 Summary

A table format adds five guarantees to files in object storage, each preventing a specific Chapter 9 failure: atomic commits (no partial writes visible), snapshot isolation (a reader's result does not change underneath it), schema enforcement (a mismatched write is rejected rather than corrupting the table), time travel, and efficient row-level deletes. It is not a query engine, not a catalog, does not make queries faster by itself, and does not give you multi-table transactions.

The mechanism is a transaction log: numbered commit files containing add and remove actions, with per-file statistics in the log itself — so pruning happens without opening any data file, and a 14-million-file table has a log you read in seconds instead of 14,000 paginated LIST calls. Checkpoints every ten commits keep replay bounded. dataChange: false on a compaction is what stops a streaming consumer reprocessing reorganized data.

Atomic commits rest on one primitive: put-if-absent. Writers proceed optimistically and race to create version $N+1$; the loser reads the winner's commit and either retries (disjoint partitions) or fails (overlapping modifications). Optimistic concurrency works when collisions are rare — the answer to constant conflicts is disjoint partition ownership, not retry tuning. And S3 gained conditional writes in 2024, which obsoleted a large body of advice about needing an external coordinator — the third time in this book a foundational assumption changed under existing practice.

⚠️ A table format gives you atomicity, not idempotency. A commit that times out may have succeeded; retrying blindly appends the same rows twice. Use transaction identifiers, idempotent write patterns like replaceWhere, or check before retrying.

Schema enforcement rejects mismatched writes, and evolution distinguishes safe from unsafe: adding a nullable column and widening a type are safe; adding a required column, narrowing a type, and changing a partition column are not; renames and drops are metadata-only only with column mapping, which you should enable at creation because turning it on later is a protocol upgrade. mergeSchema = true is convenient and disables the guarantee you adopted the format for — on in bronze, off everywhere else.

Maintenance is the section most introductions skip and it costs more than the storage. Three standing obligations: OPTIMIZE (with a sort order, where most of the benefit is), log cleanup and checkpoints (a 30-second streaming writer produces 2,880 commits a day), and VACUUM. At Kestrel: roughly $600/year of maintenance against $94/year of storage for the same table. And never lower vacuum retention during an incident — it deletes files that a long-running query has pinned, and it also bounds your time travel.

Time travel is genuinely good for three things: reproducing a report, debugging a pipeline change as a diff rather than a debate, and restoring from a bad write in ninety seconds instead of four hours — which decouples restoring service from fixing the cause, the most useful property an on-call engineer can have. It is not a backup (same bucket, same blast radius, bounded by retention) and not an audit log.

Copy-on-write for read-heavy tables, merge-on-read for write-heavy ones, with deletion vectors as the modern refinement. And the erasure trap: with merge-on-read, deleted rows are logically gone and physically present until OPTIMIZE and then VACUUM — if your obligation is physical erasure within a deadline, DELETE alone does not satisfy it. Put the full sequence in the runbook.

Delta, Iceberg, and Hudi are converging. Decide on ecosystem fit; the one genuine capability gap is Iceberg's partition evolution, which matters if you might get partitioning wrong — and you might. Migration between them is weeks rather than quarters because the data files are Parquet either way.

Kestrel lands customer_id as a real column beside the unparsed payload, a principled exception to the raw-payload rule, because an erasure request needs a predicate. Land raw; extract only what partitioning and deletion require — and write that rule down, because the next request will be for just one more field.

What's next

Chapter 11 is file formats and serialization: CSV, JSON, Parquet, Avro, and ORC, compared by measurement rather than assertion. It is where the 12.3× compression ratio this book has referenced since Chapter 1 finally gets computed, where the row-versus-column choice gets decided for the streaming path as well as the analytical one, and where Kestrel's format decision is made from numbers rather than from preference.