Chapter 9: Data Lakes

"A data lake is a data swamp with a discipline attached. The technology is identical."

Overview

A data lake is files in object storage, in open formats, with a query engine pointed at them. That is the whole definition, and its simplicity is both the appeal and the danger.

The appeal is real. Object storage is the cheapest durable storage available — $0.023 per gigabyte per month at the frozen rate, eleven nines of durability, effectively unlimited. Open formats mean any engine can read your data, forever, with no vendor between you and it. And there is no schema to define before you can store something, which matters enormously when the thing you are storing is a clickstream whose shape changes weekly.

The danger is that a data lake guarantees nothing, and every guarantee a database was quietly providing becomes your job. No transactions. No schema enforcement. No consistency across files. No updates or deletes without rewriting. No metadata. No record of what any of it means.

Chapter 3 §3.4 stated this as a table row — "ACID transactions: No" — and this chapter is where it becomes concrete, because the failure modes are specific and every one of them has a name.

The chapter is organized around a claim: most data lake failures are not technology failures. They are the absence of practices that a database used to enforce on your behalf, and which nobody noticed were being enforced. Layout, naming, partitioning, compaction, and documentation are all things a warehouse does for you and a lake does not. §9.7 is the list.

In this chapter, you will learn to:

• Explain the four ways object storage differs from a filesystem, and what each costs you.
• Describe the internal structure of a Parquet file and connect each part to a specific query optimization.
• Design a bucket and prefix layout that serves access control, lifecycle policy, and pruning at the same time.
• State precisely what landing raw means, and defend keeping data you have already cleaned.
• Choose a partitioning scheme and predict the partition sizes it produces.
• Quantify the small-files problem and design a compaction job that fixes it.
• Name the practices that separate a lake from a swamp.

Who needs this chapter: everyone; it is on the Quick Start path. §9.6 in particular is the chapter that prevents a class of problem that is much cheaper to avoid than to fix.

9.1 Object Storage Is Not a Filesystem

The single most common source of data lake bugs is treating S3 like a disk. It looks like one — there are paths with slashes in them, and libraries let you "open" a "file" — and it behaves differently in four ways that matter.

1. There are no directories

s3://kestrel-bronze/events/event_date=2025-11-28/part-0001.parquet looks like a path with directories. It is not. It is a flat key-value store: the entire string after the bucket name is a single opaque key, and the slashes are just characters.

Consequences that surprise people:

• You cannot rename a "directory." There is nothing to rename. Moving a prefix means copying every object to a new key and deleting the old one — $N$ copies and $N$ deletes, at request-cost and time proportional to object count.
• Listing is a query, not a lookup. LIST with a prefix scans the key space, returns 1,000 keys per call, and is paginated. Listing a prefix with a million objects is a thousand API calls.
• An empty "directory" does not exist. There is no such thing.

2. Objects are immutable

You cannot append to an object or modify a byte in the middle. You can only replace the whole object.

This is the reason lakehouse table formats exist (Chapter 10). Updating one row in a Parquet file means rewriting the entire file, and doing that safely while other readers are reading requires a protocol that object storage does not provide.

3. Operations are billed and rate-limited

Every GET, PUT, and LIST is an API call with a price — $0.0004 and $0.005 per thousand at the frozen rates — and a rate limit. This is the mechanism behind the small-files problem in §9.6, and it is why a "free" storage layer can generate a meaningful bill from a workload that stores almost nothing.

4. Consistency has changed, and old advice persists

S3 has been strongly read-after-write consistent for all operations since December 2020, as Chapter 4 §4.4's 🧭 note explains. Before that it was eventually consistent for overwrites and deletes, and an entire body of engineering practice grew up around working around it — much of which is still near the top of search results.

Other object stores have their own models. Verify rather than assume, and note the date on any advice about this.

⚠️ Failure Mode — The rename that was a copy

A pipeline wrote output to a staging prefix and then "moved" it to the final location:

```python

Looks like a rename. Is not a rename.

for obj in s3.list_objects_v2(Bucket=b, Prefix="staging/2025-11-28/")["Contents"]: dest = obj["Key"].replace("staging/", "gold/") s3.copy_object(Bucket=b, CopySource={"Bucket": b, "Key": obj["Key"]}, Key=dest) s3.delete_object(Bucket=b, Key=obj["Key"]) ```

With 34,560 objects (the daily count from Chapter 2 §2.4), this is 34,560 copies and 34,560 deletes. Three consequences, in ascending order of severity:

It is slow. Sequentially, at ~30 ms per operation, roughly 35 minutes.

It costs money. 34,560 COPY at $0.005/1,000 plus 34,560 DELETE — a few tens of cents per run, which is $100+/year for a nightly job doing something that should be free.

It is not atomic, and that is the real problem. A reader listing the gold/ prefix partway through sees a partial result and does not error. A failure partway through leaves objects in both locations, and the retry copies the already-copied ones again.

The fixes: write directly to the final location and use a manifest or a marker object to signal completeness; or use a table format (Chapter 10) whose atomic commit makes the whole set of files visible at once. The second is the real answer — this is precisely the problem Delta and Iceberg were built to solve.

9.2 Inside a Parquet File

Parquet is the default file format for analytical data, and understanding its structure explains every optimization in Chapter 8.

  ┌───────────────────────────────────────────────────┐
  │ PAR1                                   magic bytes│
  ├───────────────────────────────────────────────────┤
  │ Row Group 0            (~128 MB of rows)          │
  │  ┌──────────────────────────────────────────────┐ │
  │  │ Column chunk: order_id                       │ │
  │  │   Page 0  [values][def levels][rep levels]   │ │
  │  │   Page 1  ...                                │ │
  │  │   → statistics: min=88214 max=104882 nulls=0 │ │
  │  ├──────────────────────────────────────────────┤ │
  │  │ Column chunk: placed_at                      │ │
  │  │   → statistics: min=2025-11-28 max=...       │ │
  │  ├──────────────────────────────────────────────┤ │
  │  │ Column chunk: net_revenue_cents      ...     │ │
  │  └──────────────────────────────────────────────┘ │
  ├───────────────────────────────────────────────────┤
  │ Row Group 1 ...                                   │
  ├───────────────────────────────────────────────────┤
  │ FOOTER: schema · row group metadata · statistics  │
  │         · column chunk byte offsets               │
  ├───────────────────────────────────────────────────┤
  │ footer length (4 bytes) │ PAR1                    │
  └───────────────────────────────────────────────────┘

In words: a Parquet file is a sequence of row groups, each containing a column chunk per column, each chunk made of pages. The footer at the end holds the schema, the byte offset of every column chunk, and the min/max/null statistics for each.

How a reader uses that structure

This sequence is worth knowing precisely, because every optimization in Chapter 8 is one of these steps:

1. Read the last 8 bytes to find the footer length and confirm the magic bytes.
2. Read the footer. Now the reader knows the schema, every column chunk's location, and every row group's statistics — without reading any data.
3. Skip row groups whose statistics exclude them. WHERE placed_at >= '2025-11-28' against a row group whose max is 2025-11-27 — skipped, unread. Predicate pushdown.
4. Read only the column chunks named. The footer's byte offsets make this a targeted range request. Projection pushdown.
5. Decompress and decode pages, applying the encodings from Chapter 8 §8.3.

The footer being at the end is deliberate and load-bearing. It means the writer can stream data without knowing the statistics in advance, and it means a reader can fetch the footer with one small range request before deciding what else to fetch. It also means a truncated Parquet file is entirely unreadable — not partially readable — which is a real operational property: an interrupted write leaves a file that fails cleanly rather than one that silently returns partial data.

Practical properties

🧪 Try It — Read a Parquet footer

Ten minutes, and it makes the structure concrete in a way that reading about it does not.

```python import pyarrow.parquet as pq

f = pq.ParquetFile("_out/oi.parquet") print(f.metadata) # rows, row groups, created_by print(f.schema_arrow)

rg = f.metadata.row_group(0) for i in range(rg.num_columns): col = rg.column(i) s = col.statistics print(f"{col.path_in_schema:24} " f"{col.total_compressed_size/1e6:7.2f} MB " f"min={s.min if s else '-'} max={s.max if s else '-'}") ```

Then answer three things:

1. Which column is largest on disk, and why? (Chapter 8 §8.3's compression table predicts it.)
2. How many row groups? If it is one, the file is too small for pruning to help — a single row group means the statistics span everything.
3. Do the statistics on your sorted file differ usefully from the unsorted one? Compare the min/max ranges of product_id across row groups in both. On the unsorted file every row group spans nearly the whole range, which is exactly why nothing can be skipped.

Question 3 is the point of the exercise. It is Chapter 8 §8.2's dependency, visible in bytes.

9.3 Laying Out a Lake

Layout is the decision people make by accident and pay for continuously. Get it right at the start; it is a rename-a-million-objects problem afterwards.

Buckets

Use separate buckets per layer, not prefixes within one bucket:

kestrel-bronze      raw, append-only, source-shaped
kestrel-silver      cleaned, deduplicated, typed, conformed
kestrel-gold        dim_* and fct_*, business-defined
kestrel-scratch     temporary, aggressive expiry

Four reasons, and the first is the one that decides it:

Access control is per bucket. A read-only role scoped to kestrel-gold is a bucket policy. The same restriction inside one bucket is a prefix-condition policy — expressible, more fragile, and easier to get subtly wrong.

Lifecycle policies are per bucket or prefix. Bronze wants tiering to infrequent access after 90 days; gold does not. Separate buckets make these independent.

Blast radius. An accidental recursive delete is confined to one layer.

Cost attribution. Bucket-level metrics tell you what each layer costs, without tagging work.

Prefixes

Within a bucket, a consistent structure:

s3://kestrel-bronze/
  orders/                          ← the source table or dataset
    v1/                            ← schema version (Chapter 17)
      ingest_date=2025-11-28/      ← Hive-style partition
        hour=03/
          part-00000-a3f8...parquet
          part-00001-b921...parquet
          _SUCCESS                 ← completeness marker

Five rules, each preventing a specific problem:

1. Dataset name first. Everything about one dataset shares a prefix, so lifecycle policies and access control can target it.

2. Schema version in the path. When a breaking change arrives (Chapter 17), v2/ sits beside v1/ and both remain readable. This one costs nothing to add and is impossible to retrofit.

3. Hive-style partitioning — key=value in the path. Every engine understands it, and the partition column is recoverable from the path without opening a file.

4. Partition columns in descending order of selectivity. Date first, because almost every query filters on it.

5. A completeness marker. An empty _SUCCESS object written last. Without it, a reader cannot distinguish "this partition is complete" from "this partition is being written right now." This is the same problem as §9.1's non-atomic rename, and the marker is the poor-man's fix — Chapter 10's transaction log is the real one.

📐 Design Decision — Partition by ingest date or by event date?

A genuine fork, and Kestrel makes different choices in different layers.

Partition bronze by ingest date. The writer knows when it wrote something; it does not necessarily know the event time of every record in a batch. Partitioning by ingest date makes the write path simple, idempotent (rewrite one partition), and immune to late-arriving data — a Tuesday event arriving Thursday lands in Thursday's partition and nothing is corrupted.

Partition silver and gold by event date. The business question is "what happened on Tuesday," not "what did we receive on Thursday" (Chapter 4 §4.6).

The transformation from bronze to silver re-buckets by event time, and that is where the late-arrival policy lives — Kestrel's three-day window, reprocessed nightly.

What this costs: the bronze-to-silver step must reprocess a trailing window rather than only the newest partition, which is more compute every night. What the alternative costs: partitioning bronze by event date means a late arrival must modify an old partition, which on plain object storage means rewriting it — non-atomically, while people are reading it. That is the worse problem, and it is why the write path gets the simple option.

9.4 Landing Raw

Chapter 2 §2.2 said landed data should be preserved, not cleaned. This section is what that means precisely, because "raw" is used loosely and the looseness costs you.

Raw means: exactly what the source sent, plus metadata about the receipt. Not parsed into columns, not type-cast, not deduplicated, not filtered, not corrected.

# bronze record for a clickstream event
{
  "_ingested_at":  "2025-11-28T03:14:22.481Z",   # when WE received it
  "_source":       "kafka:kestrel.clickstream.v1",
  "_partition":    7,
  "_offset":       412883901,
  "_schema_version": "v1",
  "payload":       "{\"event_id\":\"...\",\"event_ts\":\"...\", ... }"
}

The payload stays a string. Not parsed JSON — a string.

That looks pedantic and it is the whole point. If you parse at landing, you must decide what to do with a record that does not parse, and every available answer loses information: drop it and it is gone; null the fields and you have lost the original; fail the batch and you have lost availability for one bad record. Storing the bytes defers the decision to a layer where you can afford to make it wrong.

Why keep raw when you have cleaned data

Four reasons, and each has a corresponding question you cannot otherwise answer:

Reprocessing. You will find a bug in a transformation. "Can we fix the last six months?" — only if the input still exists.

Investigation. The first question in every incident is "what did the source actually say?" Chapter 4's Case Study 2 was recoverable only because the source retained history; a bug against a seven-day Kafka topic would have lost six weeks permanently.

New questions. In 2027 someone asks about a field you discarded in 2025 because nothing used it.

Audit. "Prove this number came from the source." A lineage claim you cannot demonstrate is an assertion.

What it costs, honestly: Kestrel's bronze clickstream is 4.19 TB/year raw or 341 GB as Parquet, which is $96.37 or $7.84 a month (Chapter 1 §1.5). At that price the question is uninteresting. At 1,000× Kestrel it is a real budget line and the answer becomes a short bronze retention with a long-retained transformed layer — Chapter 3 §3.3's 📏 note.

🔐 Privacy & Governance — Raw retention meets the right to erasure

The strongest argument against keeping raw data forever is not cost. It is that raw data contains personal data you have committed to deleting on request.

An erasure request must reach every copy — including Parquet files in bronze with no index, which is exactly the case Chapter 3's Case Study 2 used to eliminate a plain data lake as an option.

Three approaches, in ascending order of how much they cost you:

1. Use a table format that supports row-level deletes (Chapter 10). Kestrel's answer. DELETE FROM bronze.events WHERE customer_id = ? becomes minutes rather than days.
2. Tokenize at landing. Replace direct identifiers with tokens, keep the mapping in a separate permissioned store, and delete the mapping to render the raw data non-identifying. This is a form of parsing at landing, so it contradicts the rule above — and it is the right trade when the alternative is not being permitted to land at all.
3. Short bronze retention. Keep raw for 30–90 days rather than forever. Cheap and it forfeits reprocessing beyond the window.

Decide this before you land, not after. Chapter 31 has the mechanics; the architectural point is that this choice belongs in Part II. This book is not legal advice; confirm your obligations with counsel.

9.5 Partitioning in Object Storage

Partitioning in a lake is physical: the partition value is in the object key, so an engine can skip whole prefixes without reading anything. It is the strongest pruning available and the easiest to get wrong.

Choosing the scheme

Three rules:

Partition by what you filter on. Almost always a date. Kestrel filters by date in essentially every query.

Choose a grain that produces substantial partitions. The arithmetic, for Kestrel's clickstream at 14,000,000 events/day and 66.7 bytes per event as Parquet (341 GB/year ÷ 5.11B events):

Daily is right here. 934 MB per partition splits into three or four well-sized Parquet files. Hourly gives 38.9 MB partitions — usable, and you are now managing 8,760 prefixes for a modest pruning gain. By minute is the small-files problem with extra metadata.

Do not partition by high-cardinality columns. Partitioning by customer_id at Kestrel gives 1.9 million partitions averaging under 2 KB each. This is the single most destructive layout mistake available and it happens because the reasoning — "we filter by customer, so partition by customer" — sounds right.

Nested partitioning

event_date=2025-11-28/hour=03/ gives coarse pruning by date and finer pruning within a day.

Use it only when queries genuinely filter at both levels. A nested partition scheme where the second level is never filtered is pure overhead: more prefixes, more listing, smaller files, no benefit.

💸 Cost Check — What over-partitioning costs, three ways

Partitioning Kestrel's clickstream by customer_id instead of event_date: 1.9 million partitions, 5.11 billion events, averaging 2,689 events per customer per year and — at one file per partition per day — potentially hundreds of millions of tiny objects.

Take a conservative version: one file per customer per year, 1.9 million files averaging 179 KB.

1. Listing. LIST returns 1,000 keys per call. Enumerating the dataset is 1,900 API calls before reading a byte, and every query that must list does this.

2. Reading. A query for one day scans every partition, because event_date is not in the path: 1.9 million GET requests at $0.0004/1,000 = $0.76 per query in requests alone, plus 341 GB scanned instead of 934 MB.

3. Metadata. Every engine that catalogs this table stores 1.9 million partition entries. Hive metastores and Glue catalogs both degrade badly at this scale, and partition discovery — which some engines do on every query — becomes the dominant cost.

Against event_date partitioning: 365 partitions, ~1,100 files, one day's query reads 934 MB and issues about three GET requests.

The reasoning that produces the mistake is "we filter by customer, so partition by customer." The correction: partition by the low-cardinality column you filter on; use sort order and file statistics for the high-cardinality one. Sorting by customer_id within a date partition gives you most of the skipping benefit with none of the metadata explosion.

9.6 The Small Files Problem

The most common data lake performance problem, and the most preventable.

Why it happens

Streaming ingestion writes on a time or size trigger. A consumer committing every 30 seconds across 12 partitions writes:

$$\frac{86{,}400}{30} \times 12 = 34{,}560 \text{ files/day} = 12{,}614{,}400 \text{/year}$$

at an average of 27 KB each. Chapter 2 §2.4 introduced this; here is what it actually costs.

What it costs

Request overhead. A full-year scan issues at least one GET per file:

$$12{,}614{,}400 \times \frac{\$0.0004}{1{,}000} = \$5.05 \text{ per scan, in requests alone}$$

Latency, which is worse. Each GET has ~20–50 ms of latency. Even at 100-way parallelism, 12.6 million requests is:

$$\frac{12{,}614{,}400 \times 0.03}{100} = 3{,}784 \text{ seconds} = 63 \text{ minutes of pure request latency}$$

against roughly 30 seconds to read the same bytes from 1,332 well-sized files.

Metadata overhead. Every file has a Parquet footer to read and, in a catalog, an entry to track.

Compression loss. Compression works on patterns within a file. A 27 KB file has almost no patterns to exploit, so the same data in small files is substantially larger as well as slower.

No pruning. A 27 KB file is one row group, so its statistics span everything in it. Fine — but you must read the footer of all 12.6 million files to find that out.

Compaction

The fix is a scheduled job that rewrites many small files into few large ones.

"""Compact one partition of Kestrel's bronze events.

Idempotent by construction: it writes to a NEW prefix and swaps, so a failure
part-way through leaves the original untouched. The obvious version -- delete
the small files, then write the big ones -- has a window in which the partition
is empty or partial, and readers do not error, they just get fewer rows.
"""
import duckdb

def compact(partition: str, target_mb: int = 256) -> None:
    con = duckdb.connect()
    src = f"s3://kestrel-bronze/events/v1/{partition}/*.parquet"
    tmp = f"s3://kestrel-bronze/_compacting/events/v1/{partition}/"

    n_rows, n_bytes = con.execute(
        "SELECT COUNT(*), SUM(_file_size) FROM parquet_metadata(?)", [src]
    ).fetchone()
    n_files = max(1, round(n_bytes / (target_mb * 1_000_000)))

    con.execute(f"""
        COPY (
            SELECT * FROM read_parquet('{src}')
            -- Sorting is not decoration. It is what makes the statistics in
            -- the new files SELECTIVE, which is what makes pruning possible
            -- on later reads. Chapter 8 section 8.2.
            ORDER BY session_id, event_ts
        ) TO '{tmp}' (
            FORMAT PARQUET, COMPRESSION zstd,
            PARTITION_BY (), FILE_SIZE_BYTES {target_mb * 1_000_000}
        )
    """)
    swap_prefix_atomically(tmp, f"events/v1/{partition}/")   # Ch. 10 does this properly

Three properties of a good compaction job:

It sorts — usually. Compaction is a free opportunity to establish sort order, which improves both compression (Chapter 8 §8.3) and pruning (§8.2). A compaction job that does not sort has left most of its value on the table.

The exception, measured in Chapter 11 §11.6: if the data already arrives in a beneficial order — and append-only event data arrives in timestamp order, which delta-encodes almost perfectly — sorting by something else destroys it. Sort by the order queries filter on, which for event data is usually the order it already has. Measure before assuming.

It writes then swaps. Never delete first. The swap is the unsafe part on plain object storage, which is Chapter 10's subject.

It runs on a lag. Compact partitions that are no longer being written — typically yesterday's, not today's. Compacting an actively-written partition races the writer.

📏 Scale Note — When to compact, and the two-tier pattern

Compaction is not free: it reads and rewrites the data, so it costs compute and GET/PUT requests. The thresholds worth knowing:

• Average file size under 32 MB: compact. The overhead dominates.
• Files per partition above ~1,000: compact regardless of size.
• Files already 128 MB+: leave them alone. You are spending compute for nothing.

At Kestrel's volume the practical pattern is two-tier: the streaming consumer writes small files for freshness (data is queryable within 30 seconds), and a nightly job compacts yesterday's partition into 256 MB files.

That gives you fresh data and efficient historical scans, at the cost of one scheduled job and a brief period where a partition exists in two forms. It is the standard answer and it is worth reaching for by default rather than deriving each time.

9.7 From Lake to Swamp

Everything in this chapter so far is technology. This section is the practices, and it is why the chapter opened by claiming that most lake failures are not technology failures.

A data swamp is a data lake in which nobody can find anything, nobody knows what anything means, and nobody trusts what they find. The technology is identical. The difference is seven practices, each replacing something a database used to enforce for you.

The failure is that each of these is invisible for the first eighteen months. Nothing breaks when you skip the catalog. Nothing breaks when the retention policy is a document nobody enforces. The cost arrives later and arrives as ambiguity — five overlapping datasets, none authoritative; a deletion request that cannot be honored; a table nobody dares delete because nobody knows who reads it.

🏭 From the Pipeline — Two hundred datasets, four owners

A company's lake had grown to about 200 distinct datasets over three years. An audit asked three questions of each.

Question	Answerable for
Who owns this?	34 of 200
What does it mean — is there a schema or documentation?	51 of 200
Is anything reading it?	88 of 200

Nothing was answerable for 112 of them. Not "the answer was bad" — there was no answer, and nobody could produce one.

The remediation was not technical. It was a deadline: every dataset must have an owner and a one-paragraph description by a date, or it is moved to an archive prefix and access is revoked. Fourteen weeks later, 71 datasets had been claimed and documented, 96 were archived without complaint, and 33 turned out to be duplicates of one another.

96 of 200 datasets were archived and nobody noticed. That is roughly half the lake, costing storage, appearing in every listing, and confusing every new engineer, for nothing.

The general practice: make ownership a requirement for existence, with a deadline and a consequence. A catalog nobody is required to fill in stays empty. Chapter 30 covers the tooling; the tooling is the easy half.

9.8 Querying a Lake

Three ways, in ascending order of guarantee.

Direct file access. read_parquet('s3://.../*.parquet') in DuckDB, or Spark's reader. No catalog, no metadata, no consistency. Fine for exploration and for one-off analysis; a liability as an interface, because every consumer must know the path convention and nothing tells them when it changes.

A catalog over files. AWS Glue, a Hive metastore, or Unity Catalog maps a table name to a set of paths with a schema. Engines see tables rather than paths, partitions are registered, and pruning uses the catalog rather than listing. This is the minimum viable interface for a lake other people consume.

A table format. Delta, Iceberg, or Hudi. Adds a transaction log giving atomic commits, schema enforcement, time travel, and row-level deletes. Chapter 10.

🎓 Interview Angle — "What's the difference between a data lake and a data warehouse?"

Common opener, and most candidates answer with storage cost and flexibility. Answer with guarantees, which is the framing from Chapter 3 §3.4:

"A warehouse gives you schema on write, ACID transactions, and a query optimizer with statistics, at a higher storage cost and with a poor fit for semi-structured data. A lake gives you the cheapest possible storage, any data shape, and total engine independence — and guarantees essentially nothing. No transactions, no schema enforcement, no consistency across files.

The practical consequence is that a directory of Parquet files is not a table. Two concurrent writers to one prefix interleave silently, and a reader that lists mid-write gets a partial result and does not error. That's the specific gap a lakehouse table format fills.

So in practice I'd usually want both: cheap open storage for raw and intermediate data, and a warehouse or a table format for the layer people actually query."

The "a directory of Parquet files is not a table" line does a lot of work, because it demonstrates you have thought about failure rather than about features.

Naming, versioning, and the path you cannot change

An object key is a contract with every reader, and unlike a schema it has no evolution mechanism. Renaming a prefix is N copies and N deletes (§9.1); updating every job, notebook, and dashboard that referenced it is worse.

So the layout is a decision with the reversal cost of a schema change and the ceremony of a directory name, which is why it is made carelessly.

The five components of a good key

s3://kestrel-bronze/orders/v1/ingest_date=2026-11-27/part-00000.parquet
     └──── 1 ────┘ └─2─┘└3┘ └────────── 4 ─────────┘ └────── 5 ─────┘

1  bucket    the layer. Access control and lifecycle attach here (§9.5).
2  dataset   a stable name. NOT the source table's name -- that changes.
3  version   the SHAPE's version. v2 is a new prefix, written in parallel.
4  partition key=value, Hive style, one partition column, low cardinality.
5  file      generated. Never meaningful. Readers must not parse it.

Component 3 is the one people omit and the one that saves the migration. A breaking change to the landed shape — a different envelope, a different partition column, a semantic change — writes to v2/ while v1/ continues, readers move at their own pace, and v1/ is deleted when a read audit says nobody has touched it (Exercise 25.18). That is Chapter 17's expand-contract, at the storage layer, and it costs one path segment.

Component 5 has a rule that is violated constantly: readers must not parse filenames. A job that extracts a date from orders_20261127.parquet has coupled itself to the writer's naming, and the coupling is invisible until somebody changes a separator.

Two conventions worth adopting on day one

key=value partitions, always, even with one partition column. Hive-style partitioning is what lets an engine prune without being told the layout; a bare 2026-11-27/ directory requires every reader to be configured. The extra nine characters buy automatic pruning in every engine in this book.

And lower-case, hyphen-or-underscore, no spaces, no dates in the dataset name. orders is a dataset; orders_2026 is a dataset that will be wrong in January and will still exist in 2031.

What to write beside the data

s3://kestrel-bronze/orders/v1/
├── _README.md          who writes this, what it is, which chapter, who to ask
├── _schema/v1.json     the shape as landed, dated
└── ingest_date=.../
    ├── part-*.parquet
    └── _SUCCESS

A _README.md in a prefix costs nothing and answers the question that otherwise costs an hour. Chapter 30's catalog is a better version of this and it does not exist yet in Chapter 9; a file beside the data is the version that is available immediately and never goes stale silently, because it is next to the thing it describes.

🧭 Version Note — S3 became strongly consistent, and a body of practice became folklore

Until December 2020, S3 offered eventual consistency for overwrites and for list-after-write. A file you had just written might not appear in a listing; an object you had just overwritten might return its previous version. An entire generation of lake tooling was designed around this.

text what people built why still needed? ──────────────────────────────────────────────────────────────────────────── S3Guard / EMRFS consistent view listings were stale NO -- removed a retry loop around a read after a write the read might 404 NO time.sleep(5) after a write same NO a "consistency reconciliation" job to find missing files NO writing to a temp key and copying to get atomicity STILL YES *

All of it now reads as prudent engineering and most of it is dead code, and the last row is the exception that matters: strong consistency is not atomicity. A multi-object write is still not atomic, which is why _SUCCESS markers and transaction logs (Chapter 10) exist and why they are unaffected by this change.

Two things to take from this beyond the S3 fact.

Compensating for a problem that no longer exists is harmless, which is why it survives. The sleep(5) still works. Nothing fails. It is removed only when somebody asks why it is there, and the comment above it — if there is one — says what it does rather than why.

And the signal that it had changed was a blog post. No system told anyone. Exercise 4.19's conclusion applies: write down the assumption with a date and a source, not the workaround, because an assumption can be rechecked and a ritual cannot.

🔁 Idempotency Check — the three ways to write a partition twice

A lake write is not one operation, and only one of the three shapes below is safe to repeat.

```text APPEND to a prefix part-00000.parquet, then part-00001.parquet on the second run -> the partition now contains the data TWICE. No error. -> this is Chapter 1's incident, in object storage

OVERWRITE a prefix delete every key under the prefix, then write -> safe to repeat, and there is a WINDOW during which the partition is empty or partial (§9.7)

REPLACE via a manifest or a table format write to new keys, then flip a pointer atomically -> safe to repeat AND no visible window -> this is Chapter 10, and it is why Chapter 10 exists ```

The middle row is what most hand-rolled pipelines do, and it is correct in the sense that the final state is right. What it is not is atomic — a reader arriving mid-write sees a partition that is missing most of its files, and _SUCCESS does not help because the marker from the previous successful run is still there (§9.7's second failure case).

The rule to apply until Chapter 10 gives you a better mechanism:

text 1. write to a STAGING prefix keyed by the run id 2. verify: file count, row count, and a checksum if you have one 3. delete the target prefix 4. copy or move from staging 5. write _SUCCESS LAST

Step 2 is the one that is usually skipped and the one that makes the rest worth doing. Without it, steps 3 and 4 will faithfully replace good data with a truncated write. The verification is what converts "overwrite" from a hope into an operation.

And the run id in step 1 is what makes a retry safe: two concurrent runs of the same interval write to different staging prefixes and the second one's step 3–4 is a no-op with respect to the first, rather than an interleaving.

🔎 Read the Plan — the lake's plan is a file listing

In a warehouse the plan tells you what will be scanned. In a lake you can see it directly, and looking is faster than reasoning.

```bash

what will this query actually open?

aws s3 ls --recursive --summarize \ s3://kestrel-silver/order_lines/order_date=2026-11-27/ ```

```text 2026-11-28 03:14 134217728 .../part-00000.parquet 2026-11-28 03:14 134217728 .../part-00001.parquet 2026-11-28 03:14 41984102 .../part-00002.parquet 2026-11-28 03:14 0 .../_SUCCESS

Total Objects: 4 Total Size: 310419558 ```

Four numbers, and each answers a question the engine's plan would only imply.

Object count. Three data files for a day is right. Three thousand is the small-file problem (§9.6), and you can see it without running anything.

Average size. 103 MB here, comfortably inside the 128 MB–1 GB range. A dataset whose average is under 32 MB is flagged by Exercise 9.21's audit and this is the manual version of that check.

The _SUCCESS marker's presence and its timestamp. Present, and written after the parts — check the ordering, because a marker written first is a marker that means nothing (§9.7).

And the modification times, which should cluster. Files written minutes apart are one run; files written hours apart are two runs against the same prefix, which is either an append (not idempotent) or a concurrent write (worse).

Then the engine-side confirmation, which tells you whether the pruning you designed is happening:

```python df.explain("formatted")

PartitionFilters: [(order_date = 2026-11-27)] <- one prefix opened

ReadSchema: struct <- two columns of many

```

The listing says what exists; the plan says what will be touched. Reading both takes thirty seconds and it resolves most "why is this query slow" questions in a lake before any tuning begins.

🏭 From the Pipeline — the prefix that was renamed on a Friday

A dataset had been landing at s3://kestrel-bronze/events/ for two years. Somebody renamed it to s3://kestrel-bronze/clickstream/ to match the naming convention that had been adopted since.

The rename was a copy of 341 GB and a delete, which took four hours and cost about $9 in requests. That part went fine.

What did not go fine was the seventeen readers. Eleven were dbt sources and were updated in the same pull request. Four were notebooks. One was a scheduled export to a partner. One was a Terraform lifecycle rule which now applied to a prefix that no longer existed, so the new prefix had no expiry at all.

The partner export failed silently for nine days, because it caught the exception, logged it, and continued — and the partner did not complain until their monthly reconciliation.

The lifecycle rule was found four months later, during a cost review, by which point the new prefix held 113 GB of data that should have expired.

Three things would each have prevented most of this.

A version segment in the key (§9.9). events/v1/ and clickstream/v1/ written in parallel, with readers moving at their own pace and events/v1/ deleted when a read audit said nobody had touched it. The rename was a breaking change dressed as a tidying.

A read audit before the change, not after. Exercise 25.18's query over 400 days would have found fifteen of the seventeen; the notebooks are the two it would have missed, and the notebooks are also the two whose failure was loud.

And a _README.md in the prefix naming the owner. The partner export's owner was a person who had left, which is why nobody was notified and why the exception handler had been written to swallow failures in the first place.

The transferable rule: an object key is a public interface, and renaming one is a deprecation exercise rather than an edit. Chapter 17's expand-contract applies to paths exactly as it applies to schemas, and it is cheaper here because a prefix costs nothing to keep.

🧭 Version Note — S3 Express, Tables, and what a new storage class does to this chapter

Object storage stopped being one thing. The advice in this chapter assumes S3 Standard; the newer classes change one variable each, and the variable they change is worth knowing.

text class / feature changes what it does NOT change ──────────────────────────────────────────────────────────────────────── Express One Zone single-digit ms latency, durability across AZs, higher price per GB and the small-file problem is still a REQUEST problem S3 Tables / a managed table format the layout decisions in managed Iceberg with maintenance included section 9.5. Grain is still yours to choose. Intelligent-Tiering automatic class movement the minimum-duration charges, which still apply per tier

The Express row is the one to reason about carefully. Lower per-request latency reduces the pain of many small files and does not reduce their cost — requests are still billed per request, and at Exercise 9.12's 6.3 million GETs the bill is the same shape. A faster store makes a bad layout tolerable, which is how bad layouts survive.

The managed-table row is the one that changes a decision. Chapter 10 argues for a table format and Exercise 10.21 builds the maintenance job; a managed offering does that maintenance for you at a price. Whether that is worth it is Chapter 5 §5.8's evaluation, and the honest answer for a four-person team is usually yes.

What none of them changes is anything in §9.1. There are still no directories, objects are still immutable, requests are still billed, and there is still no atomic multi-object write. Every conclusion in this chapter follows from those four, which is why the chapter is written from them rather than from a product's feature list.

9.9 Kestrel's Lake

s3://kestrel-bronze/                       lifecycle: Standard 90d → IA → 2y expiry
  orders/v1/ingest_date=YYYY-MM-DD/               from CDC (Ch. 14)
  order_items/v1/ingest_date=YYYY-MM-DD/
  events/v1/ingest_date=YYYY-MM-DD/hour=HH/       from Kafka (Ch. 15)
  carrier/v1/fetch_date=YYYY-MM-DD/               from the API (Ch. 16)

s3://kestrel-silver/                       lifecycle: Standard, no expiry
  orders/event_date=YYYY-MM-DD/                   deduplicated, typed, conformed
  order_items/event_date=YYYY-MM-DD/
  events/event_date=YYYY-MM-DD/
  sessions/session_date=YYYY-MM-DD/

s3://kestrel-gold/                         lifecycle: Standard, no expiry
  fct_order_item/date_key=YYYYMMDD/
  fct_session/session_date=YYYY-MM-DD/
  dim_customer/                                   Type 2, no partitioning
  dim_product/

s3://kestrel-scratch/                      lifecycle: 7-day expiry, no exceptions
  _compacting/ · _tmp/ · exploration/

Five decisions in that layout, each traceable to a section of this chapter:

Bronze partitioned by ingest_date; silver and gold by event date. §9.3's 📐 callout.

Schema version in the bronze path only. Bronze accepts whatever arrives, so it needs to distinguish versions. Silver enforces a schema, so v2 bronze data is transformed into the same silver table or fails loudly. §9.3, rule 2.

hour= nesting on events only. Events are 14 million a day; orders are 6,575. Nesting the smaller datasets would produce sub-megabyte partitions for no benefit. §9.5.

Scratch has a 7-day expiry with no exceptions. This is the rule that prevents scratch from becoming a fifth layer. It is enforced by a lifecycle policy rather than a convention, and people have lost work to it — which is the correct outcome, because the alternative is a scratch bucket with three years of nobody's data in it.

Bronze tiers to infrequent access after 90 days. Reprocessing happens within days of ingestion in practice; retrieval cost beyond that is rare enough to be worth the storage saving.

🧱 Kestrel Platform — Increment 9: the lake

(a) Create the layout. Extend platform/infra/docker-compose.yml's minio-init to create the prefix structure with a placeholder object in each, so the shape is visible in the console.

(b) Land your first Parquet. Write platform/ingest/batch/land_parquet.py: read from kestrel_app, write to s3://kestrel-bronze/orders/v1/ingest_date=<today>/, with the raw-record envelope from §9.4 — _ingested_at, _source, _schema_version, and the payload preserved.

(c) Write the _SUCCESS marker last. After all part files are closed. Then write the reader that checks for it and refuses to read an incomplete partition. Test the failure path: kill the writer partway through and confirm the reader declines rather than returning partial data.

(d) Write platform/storage/lifecycle.json with the four buckets' lifecycle rules, and apply it. Then verify by listing the rules back rather than trusting the apply.

Part (c) is the one that matters. An incomplete partition that reads as complete is the lake's characteristic silent failure, and the marker is the cheapest defense until Chapter 10 replaces it with something real.

9.10 Summary

Object storage is not a filesystem, and four differences cause most lake bugs. There are no directories — a key is one opaque string, so "renaming a directory" is $N$ copies and $N$ deletes, slow, billed, and not atomic. Objects are immutable, which is why updating a row means rewriting a file and why lakehouse table formats exist. Every operation is billed and rate-limited. And S3's consistency model changed in 2020, so pre-2021 workaround advice is obsolete and still ranks well.

A Parquet file is row groups of column chunks of pages, with a footer at the end carrying the schema, byte offsets, and per-row-group statistics. A reader takes the footer first, skips row groups whose statistics exclude them (predicate pushdown), then range-requests only the column chunks it needs (projection pushdown). The footer's position means a truncated file fails cleanly rather than returning partial data — a real operational property.

Separate buckets per layer, because access control and lifecycle policies are per bucket and blast radius is per bucket. Within a bucket: dataset name, schema version in the path (free to add, impossible to retrofit), Hive-style partitions in descending selectivity, and a _SUCCESS marker written last.

Partition bronze by ingest date and silver/gold by event date. The writer knows when it wrote something; partitioning bronze by event time means a late arrival must modify an old partition, which on plain object storage means a non-atomic rewrite while people are reading.

Raw means exactly what the source sent, plus receipt metadata — payload as a string, unparsed. Parsing at landing forces you to decide what to do with an unparseable record, and every answer loses information. Keep raw for reprocessing, investigation, unanticipated questions, and audit; at Kestrel it costs $7.84 a month, which makes the question uninteresting. The strongest argument against keeping it forever is not cost but erasure obligations, and the three answers are a table format with row-level deletes, tokenization at landing, or a short retention window — decided before you land, not after.

Partition by the low-cardinality column you filter on. Daily for Kestrel's clickstream: 365 partitions of 934 MB. Never by a high-cardinality column — customer_id gives 1.9 million partitions, 1,900 LIST calls before reading a byte, $0.76 per query in requests, and a catalog that degrades. Use sort order within the partition for the high-cardinality dimension instead.

The small-files problem costs far more in latency than in requests. 12.6 million files a year at 27 KB: $5.05 per full scan in requests, and 63 minutes of pure request latency at 100-way parallelism against 30 seconds for the same bytes in 1,332 files — plus worse compression, because a 27 KB file has no patterns to exploit. Compact when average file size is under 32 MB or files per partition exceed 1,000. A good compaction job sorts (free pruning and compression), writes then swaps (never deletes first), and runs on a lag. The standard pattern is two-tier: small files for freshness, nightly compaction for scans.

A swamp is a lake without seven practices, each replacing something a database used to enforce: layout convention, versioned schemas, a catalog, quality checks at the boundary, enforced retention, compaction, and layered access control. All seven are invisible for eighteen months, and the cost arrives as ambiguity rather than as failure. Make ownership a requirement for existence, with a deadline and a consequence — one audit found 112 of 200 datasets with no answer to who owns this, and archiving 96 of them was noticed by nobody.

What's next

Chapter 10 is the lakehouse — Delta Lake and Apache Iceberg — and it is the answer to the specific problems this chapter kept deferring: the non-atomic rename, the _SUCCESS marker that is a poor-man's transaction, the concurrent writers that corrupt a prefix silently, and the row-level delete that a privacy request requires. A transaction log over object storage turns a directory of Parquet files into something that is actually a table.