Further Reading: The Lakehouse

Sources are tagged Tier 1 (confident it exists, recommended without reservation) or Tier 2 (real and worth seeking, but confirm the current edition, version, or URL yourself).

This is the fastest-moving material in the book. The papers below explain mechanisms, which are stable. The documentation explains current behavior, which is not — check the version selector on every page, and check the date on every blog post.

The papers

• Michael Armbrust et al., "Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores" (2020), VLDB. How the transaction log actually works: the ordered commit files, checkpoints, optimistic concurrency, and — the part worth reading twice — why the naive approaches to atomic commits over object storage fail. §10.2 and §10.3 are this paper compressed. About twelve pages. Tier 1.

• Michael Armbrust et al., "Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics" (2021), CIDR. The paper that named the category and made the argument that warehouse guarantees can be delivered over open formats. Written by people with a commercial interest in the conclusion, and still the clearest statement of the technical case. Tier 1.

• Ryan Blue and Daniel Weeks' Iceberg papers and talks (Netflix). Iceberg's design rationale, particularly hidden partitioning and partition evolution — the one genuine capability gap in §10.8. The talks are more accessible than the specification and cover the why, which the specification does not. Tier 2 — conference material, scattered.

• The Apache Iceberg specification (iceberg.apache.org/spec/). Read alongside the Delta paper. The two solve the same problem with different metadata structures — Iceberg's manifest tree versus Delta's ordered log — and the comparison teaches you what is essential to the problem and what is one design's choice. Exercise 10.18 is this comparison. Tier 2 — the spec is versioned; v2 and v3 differ meaningfully.

• Vinoth Chandar et al. on Apache Hudi. Hudi's record-level indexing and incremental query model are genuinely different from the other two and are the reason it wins on upsert-heavy workloads. Worth reading if you have a CDC-dominant platform. Tier 2.

The documentation you will actually use

• Delta Lake documentation: "Table utility commands" (OPTIMIZE, VACUUM, RESTORE, DESCRIBE HISTORY), "Table properties," and "Concurrency control." The three pages that matter operationally. The concurrency control page in particular lists exactly which operation pairs conflict, which is the reference for §10.3 and for Exercise 10.9. Tier 2 — versioned.

• The delta-rs documentation and repository. The Rust implementation with Python bindings, which is what makes Delta usable without a JVM and what this book uses locally. Its feature coverage lags the Spark implementation; check the feature matrix before assuming an operation is available. Tier 2 — moving quickly.

• Iceberg's "Maintenance" documentation — expiring snapshots, removing orphan files, rewriting manifests, and compacting data files. The Iceberg equivalent of §10.5, and it is more explicit about the obligations than most Delta material is, which makes it worth reading even if you use Delta. Tier 2.

• Delta Lake's "Best practices" page and the equivalent Databricks optimization guidance. Read critically: much of it assumes Databricks-managed maintenance (auto-optimize, auto-compaction, predictive optimization), which you do not have on open-source Delta. The advice is good and its assumptions are not always stated — which is precisely how this chapter's second case study happened. Tier 2.

On the maintenance problem specifically

• Any of the "why is my Delta/Iceberg table slow" write-ups from practitioners. These are the literature for §10.5, and they are almost entirely blog posts rather than papers, because maintenance is an operational subject rather than a research one. Search for "delta lake small files," "iceberg orphan files," and "delta log checkpoint" and read three from different organizations. The consistency of the failure across companies is itself the finding. Tier 2 — blogs, and the quality varies.

• Chapter 9 §9.6 of this book, and its Case Study 1. The small-files problem in its pre-table-format form. Reading the two together shows what the format changes (a command instead of a hand-rolled pointer protocol) and what it does not (you still have to run it). Tier 1.

On the wider trajectory

• The Delta Lake UniForm / Iceberg compatibility work, and the various "one format to read them all" efforts. The direction of travel is toward a single set of Parquet files readable as either format via generated metadata. If it succeeds, §10.8's comparison becomes largely irrelevant, which would be a good outcome and is worth watching. Tier 2 — actively changing; verify current state.

• The Apache XTable project (formerly OneTable). Translates metadata between Delta, Iceberg, and Hudi without rewriting data files. The clearest evidence that the industry regards the format choice as an accident of history rather than a durable technical distinction. Tier 2.

• Warehouse vendors' Iceberg support announcements (Snowflake, BigQuery, Redshift). The significant trend of the mid-2020s: warehouses reading and writing open table formats directly, which erodes the storage-layer lock-in that Chapter 8 §8.5's exit-cost discussion is about. Tier 2 — vendor announcements, and "supports Iceberg" means different things in each.

If you only read one thing

Read the Delta Lake VLDB paper (2020), sections 3 and 4 — the storage format and the transaction protocol. About six pages of the twelve.

It is the clearest available explanation of how you build a transaction over a store that offers only single-object atomicity, and once you have it, Iceberg's design reads as a variation rather than as a separate thing to learn. It also makes §10.3's optimistic concurrency behavior predictable rather than mysterious, which matters the first time two of your writers collide.

Then, before you put a table format into production, read Iceberg's "Maintenance" page even if you are using Delta — it states the ongoing obligations more directly than anything else in this list, and §10.5 exists because most introductions do not.