Further Reading: Data Lakes

Sources are tagged Tier 1 (confident it exists, recommended without reservation) or Tier 2 (real and worth seeking, but confirm the current edition, version, or URL yourself).

The data lake has less good literature behind it than the warehouse, because it is less a technology than an absence of one. The best sources are format specifications and cloud provider documentation, plus a small amount of writing about the organizational half — which is the half that actually determines outcomes.

On the format

• The Apache Parquet documentation and format specification (parquet.apache.org/docs/). Read "File Format" for the row group / column chunk / page structure in §9.2, and "Nested Encoding" for repetition and definition levels if you work with nested data. Short, and it makes every optimization in Chapter 8 legible. Tier 2 — the docs site has been reorganized more than once; the specification itself is stable.

• The parquet-format repository's README and LogicalTypes.md on GitHub. More precise than the docs site, and LogicalTypes.md is the place to settle arguments about how a timestamp or a decimal is actually stored — which matters when two engines disagree about the same file. Tier 2.

• Julien Le Dem and Nong Li, "Parquet: Columnar Storage for Hadoop" (2013) and the associated talks. The design rationale from the format's creators. Historical, and useful for understanding which decisions were made for Hadoop-era constraints that no longer apply. Tier 2 — conference material.

• The Apache Arrow documentation, particularly on the Arrow/Parquet relationship. Arrow is the in-memory columnar format that most engines decode Parquet into. Worth understanding because it explains why some operations are nearly free (zero-copy) and others are not. Tier 1.

On object storage

• AWS S3 documentation: "Best practices design patterns: optimizing Amazon S3 performance," and "Managing your storage lifecycle." The first covers request rates, prefix parallelism, and the performance characteristics behind §9.6's arithmetic. The second is §9.9's lifecycle policies. Read for mechanism; the specific numbers change. Tier 2 — cloud documentation moves, and this book's prices are frozen at the time of writing.

• The AWS announcement of strong read-after-write consistency (December 2020). Worth reading once, specifically so that you can recognize pre-2021 advice about S3 consistency workarounds and discard it. It is also a good example of the more general lesson in Chapter 4 §4.4: a system property you designed around can change, and nothing tells you. Tier 2.

• The Google Cloud Storage and Azure Blob Storage documentation on consistency and performance. Read the consistency sections if you use either — the models differ from S3's in ways that matter, and assuming S3 semantics on another store is a real source of bugs. Tier 2.

On layout, partitioning, and compaction

• The Apache Hive documentation on partitioning, which is where key=value path partitioning comes from and why every engine understands it. Hive itself is largely historical; its partitioning convention is not. Tier 2.

• The Delta Lake and Apache Iceberg documentation on OPTIMIZE / compaction and on ZORDER / sort ordering. Read these even before Chapter 10, because they are the productized versions of §9.6's compaction job and they document the thresholds and trade-offs more precisely than this chapter has room for. Iceberg's "Maintenance" page and Delta's "Optimizations" page are the two. Tier 2 — versioned.

• The DuckDB documentation on read_parquet, parquet_metadata, and Hive partitioning. DuckDB is an unusually good tool for inspecting Parquet — parquet_metadata() and parquet_file_metadata() expose row group statistics directly, which is what makes this chapter's 🧪 Try It possible in ten lines. Tier 1.

On the swamp problem

• The original "data lake" coinage — James Dixon's 2010 blog post — and the "data swamp" counter-coinage that followed within a few years. Worth reading Dixon's post for what he actually proposed, which is narrower and more sensible than what the term came to mean: a single-source, raw-retained store, not an unmanaged dumping ground for everything. Most criticism of "data lakes" is criticism of something Dixon did not propose. Tier 2 — a personal blog post from 2010.

• Zhamak Dehghani's data mesh writing (covered properly in Chapter 35). Her diagnosis of why centralized lakes decay — that a central team cannot own the semantics of data it did not generate — is the organizational analysis behind §9.7's seven practices, whether or not you accept her prescription. Tier 2.

• The DAMA-DMBOK (Data Management Body of Knowledge), 2nd edition. Dry, comprehensive, and the standard reference for data governance as a discipline. Chapters on data quality, metadata management, and data governance cover §9.7's practices in far more institutional detail than this book. Use it as a reference for vocabulary and completeness rather than reading it through. Tier 2 — check the edition.

On the privacy trigger

• GDPR Articles 12 and 17, and your jurisdiction's equivalent. Article 17 is the right to erasure; Article 12 sets the one-month response window that appears in this chapter's second case study. As in Chapter 2, the actual articles are shorter and clearer than most summaries of them. Not legal advice; confirm with counsel. Tier 1.

If you only read one thing

Read the Parquet file format specification's "File Format" section — perhaps fifteen minutes — and then open a real Parquet file with pyarrow and print its row group statistics, as §9.2's 🧪 Try It asks.

The combination of reading the structure and then seeing the min/max values of your own data in it is what makes predicate pushdown stop being a phrase and start being a thing you can predict. Once you can look at a file and say "this will prune well" or "this will not," most of Part II's performance material becomes obvious rather than memorized.