Answers to Selected Exercises

Worked answers to the daggered (†) and odd-numbered problems from every chapter's exercises.md. Try the problem before you read the answer; the value is in the attempt, and several of these answers are only legible once you have been wrong in the specific way the problem invites.

This file is generated by scripts/assemble.py from _scratch/answers/part-NN.md. Edit the fragment, not this file.

Part I — Foundations

Chapter 1 — What Is Data Engineering?

1.1 A software service fails loudly and immediately: the request errors, the status page goes red, and the person who broke it usually finds out within minutes. A data pipeline fails quietly and plausibly: it emits a number, the number is wrong, and nothing about the number announces that it is wrong.

The second is more expensive to detect because there is no exception to catch. Detection requires someone to compare the output against an independent expectation — a reconciliation, an assertion, a person who knows what the number should be — and if nobody has built that comparison, the failure is found by a human noticing that something looks odd, which happens on a timescale of weeks. Kestrel's duplicate-rows incident ran for 31 days with every DAG green.

1.3

AOV          = $182,000,000 / 2,400,000 orders
             = $75.8333... → $75.83

orders/day   = 2,400,000 / 365
             = 6,575.34 → 6,575

Two things worth noticing in your own working. The AOV does not round to a nice number, which is what tells you it was derived rather than chosen — $75.83 × 2,400,000 = $181,992,000, which is $8,000 short of $182.0M, and that gap is the rounding, not an error.

And 6,575 is an average, not a typical day. Black Friday is 41,300 orders, 6.28× that figure. Any capacity number derived from 6,575 is a number about the median day and says nothing about the day that breaks you (§1.5, and Exercise 1.11).

1.5 Idempotency: running the operation twice leaves the system in the same state as running it once.

# idempotent
mkdir -p /tmp/kestrel          # already exists? fine.
cp source.csv /tmp/dest.csv    # same bytes, same result

# NOT idempotent
echo "row" >> data.csv         # appends every time
mv source.csv /tmp/            # second run fails; source is gone

The >> example is the one to keep in mind, because it is exactly the shape of a pipeline bug: an append-mode write that produces correct-looking data on the first run and inflated data on the second. Kestrel's 11.4% revenue inflation was a >> in disguise.

1.7

The test is §1.4's, and it is worth stating as a question rather than a table: who finds out why the number is wrong? (a) and (d) are availability failures with an obvious owner. (b) and (c) are correctness failures, and they are the ones with contested ownership — which is why the chapter argues that the answer must be written down before the incident rather than during it.

1.9

Where the colleague is right. Gzip on JSON is genuinely good: the chapter's own measurement puts gzipped JSON Lines at 1.31 GB a day against 11.48 GB raw, an 8.8× reduction — not far off Parquet's 12.3×. And JSON is easier to debug: zcat | head beats every Parquet tool for the "what is actually in this file" question at 2 a.m. The size argument is close, and the debuggability argument is simply correct.

The cost they have not considered is not storage. It is scan. Kestrel's storage is 2.1% of the bill (Chapter 33); the format decision is a compute decision that happens to affect storage. A two-column aggregate over one day reads ~1.08 GB of gzipped JSON — every byte, decompressed and parsed, to reach two fields — against ~9.4 MB of Parquet. That is roughly 100× the bytes read for the same answer, on every query, forever. One query is negligible; twenty-two dashboards refreshing hourly is not.

The measurement that settles it: write one day in both formats, then run the three queries your team actually runs and record bytes scanned and wall clock, not file size. If the workload turns out to be whole-record reads, the colleague wins. If it is column subsets, they lose by two orders of magnitude. Do not argue it on compression, because on compression they are nearly right.

1.11 This is a question about assumptions, and the arithmetic is the easy part.

The naive answer: if capacity is sized for the average and runtime scales linearly with volume, a 6.28× day takes 6.28 × 22 = 138 minutes, so the job finishes about 116 minutes late.

Three assumptions that answer must name, and each one changes the number:

1. Is 22 minutes the runtime at capacity, or with headroom? "Sized for the average" usually means sized with some slack. If the job normally runs at 60% utilisation, the multiplier applies to the utilisation, not the runtime, and the answer changes materially.

2. Is 6.28× the right multiplier? It is a daily figure. 41,300 orders spread over a day is 6.28× the average day; the instantaneous peak on Black Friday is higher still, because order arrival is not uniform. A job sized against a daily multiple is under-sized against the hour that matters.

3. Does runtime actually scale linearly? It does not, usually. Above a memory threshold a shuffle spills to disk and the curve turns superlinear (Chapter 8 §8.8) — so linear degradation is the optimistic assumption, and the honest answer says so.

And the answer that is better than any number: if the job runs on a schedule shorter than 138 minutes, "how far behind" has no finite answer. Each run starts before the last one finished, lag accumulates, and the pipeline does not recover until traffic drops. That is the failure mode worth describing, and it is invisible in a single-run calculation.

1.13 The distinguishing line is the PartitionFilters entry on the scan node: present in one plan, empty in the other.

What it means: a non-empty PartitionFilters tells you the engine pushed the date predicate down to the file listing and will open only the partitions that can match. An empty one means the predicate survived — probably wrapped in a CAST or a function, which makes it opaque to the partition matcher — so the engine lists and reads every partition and applies the filter afterwards. Same result, 4.2 TB instead of 34 GB, and a $3,840 bill instead of $74.88.

1.15 The strongest case for distributed-by-default, made honestly:

First, the cost of migration is real and asymmetric. Rewriting a single-node pipeline as a distributed one is not a configuration change; it is a rewrite of the parts that assumed shared memory, plus a new operational surface. A team that starts distributed pays a known cost up front instead of an unknown cost later, at a moment they will not choose.

Second, the ceiling is not knowable in advance. "You do not have big data" is true until an acquisition, a new market, or a product that emits ten times the events. Sizing for today's volume is sizing for a number you have no control over.

Third, the skills argument runs the other way from the usual telling. Distributed frameworks are where the industry's tooling, hiring pool, and documentation are. A single-node stack is simpler and also more idiosyncratic, and an idiosyncratic system with one expert is its own risk (Chapter 5).

Why it fails at Kestrel, specifically: the working set is 340 GB of Postgres and 341 GB of Parquet a year, which fits comfortably on one machine and will for years at the current growth rate; the team is four engineers, so a second operational surface is 25% of the team's attention; and the latency requirement is a 6am daily deadline, not seconds. None of the three arguments above survives those numbers.

What would have to change: a working set past roughly a few terabytes per query, or a concurrency requirement that one machine cannot serve, or a team large enough that the operational surface is somebody's actual job. Notice that "the data got bigger" alone is not on that list — storage growing is cheap; it is scan-per-query growing that forces the decision.

1.17 An example: the source system is right and the definition is wrong.

fct_order_line reconciles to kestrel_app.order_lines to the row and to the cent. Every value matches. And the revenue figure on the executive dashboard is still wrong, because it counts gift-card lines as revenue — a gift card sold is a liability, not a sale, and it is recognised when redeemed. The data matches the source perfectly; the measure does not match what the business means by revenue. (Chapter 38's R3 is exactly this rule.)

Who owns it: nobody, which is the point of the exercise. The data engineer's standard was met. The analytics engineer implemented the definition they were given. The definition itself has no owner, and that is the gap Chapter 30 exists to close: a metric with a named owner and a written definition would have made this a decision instead of an accident.

A second, sharper example: a source system whose updated_at is set at transaction start (§2.3). The extract matches the source exactly at the moment it ran. Rows are still missing. The extract is correct and the result is incomplete, which is a category of failure that "matches the source" cannot detect.

1.19 A good day-one open-questions list is specific enough to be answerable and honest enough to be uncomfortable. Five of the right shape:

## Open questions — 2026-01-15

1. Does `orders` need CDC, or is a nightly watermark extract sufficient?
   Blocked on: whether `updated_at` is set at commit or at statement start.
   Decided in: Ch. 13-14.
2. Where is the silver/gold boundary? Specifically: does `net_revenue_cents`
   live in silver (typed, per line) or gold (business rules applied)?
   Decided in: Ch. 34.
3. Who owns the definition of "active customer"? There are at least two in
   use today and I have not found a written one.
   Decided in: Ch. 30 — and this one is a person, not a design.
4. How long do we retain raw clickstream, and who signs off on deleting it?
   Decided in: Ch. 3 (ADR-003) and Ch. 31.
5. What is the acceptance test that says the platform is correct? "It runs"
   is not one.
   Decided in: Ch. 1 §1.7 — and this is the one I should answer first.

Two properties make the list worth rereading. Each question names where it gets answered, so the list is a plan rather than a worry log. And each one is phrased so that a future answer either does or does not settle it — "do we need CDC?" settles; "think about ingestion" does not.

Chapter 2 — The Data Engineering Lifecycle

2.1

2.3 The six undercurrents, each with a failure:

Security. The extract runs as a superuser because that was what worked on the first day, and the credential is in the repository.

Data management. Nobody can say which table is authoritative for revenue; there are four candidates and three of them are stale.

DataOps. A model is changed on a Friday, deployed by hand, and there is no way to tell what production is running.

Data architecture. Each team picked a format independently; joining across them requires four different readers and two of them disagree about time zones.

Orchestration. The transform runs at 02:00 because the extract "usually finishes by 01:45," and one night it did not.

Software engineering. The pipeline is a 900-line script with no tests, and the only way to know whether a change is safe is to run it in production.

The pattern worth naming: none of the six is a stage. Each one is a property that has to hold at every stage, and each of the failures above happens at a different stage — which is precisely why they are drawn as currents running underneath.

2.5 Four hypotheses, ordered by cost to eliminate — cheapest first, because the point of an ordering is to spend the least time reaching the answer.

1. SERVING     Is the dashboard querying the right table and the right date?
               SELECT max(revenue_date), count(*) FROM gold.daily_revenue;
2. TRANSFORM   Did the model build, and did it build for yesterday?
               (orchestrator UI, or: SELECT max(_dbt_run_at) FROM ...)
3. INGESTION   Did the extract land yesterday's rows?
               SELECT count(*), max(placed_at) FROM bronze.orders_raw
                WHERE ingest_date = current_date;
4. GENERATION  Did the source produce anything yesterday?
               SELECT count(*) FROM kestrel_app.orders
                WHERE placed_at::date = current_date - 1;

Work upstream, not downstream. The instinct is to start at generation because that is where data comes from; the discipline is to start at serving because that is where the complaint came from, and because "the dashboard is pointed at last week's table" is both the cheapest hypothesis and, in practice, a common one.

And note what step 4 is really for. If the source genuinely produced nothing, the pipeline is working correctly and the incident belongs to somebody else. Establishing that quickly is worth as much as finding a bug.

2.7

At-most-once: every message is delivered zero or one times — loss is possible, duplication is not. At-least-once: every message is delivered one or more times — duplication is possible, loss is not. Exactly-once: every message has exactly one effect.

This book recommends at-least-once delivery plus idempotent writes, because exactly-once delivery across two systems is not achievable without distributed transactions that nobody runs in a data pipeline, whereas at-least-once is cheap and duplication is a solved problem at the write — a merge on a key, a partition replacement, a dedup on read. You move the guarantee from the transport, where it is expensive and fragile, to the write, where it is a MERGE statement.

2.9 Two overlapping long transactions, as a timeline:

10:00:00  T1 BEGIN.  writes order A with updated_at = 10:00:00
10:02:00  T2 BEGIN.  writes order B with updated_at = 10:02:00
10:05:00  T3 (short) inserts order C, updated_at = 10:05:00, COMMITS
10:06:00  EXTRACT RUNS.  sees C only.  watermark := 10:05:00
10:09:00  T2 COMMITS.   order B becomes visible, updated_at = 10:02:00
10:14:00  T1 COMMITS.   order A becomes visible, updated_at = 10:00:00
11:06:00  EXTRACT RUNS.  WHERE updated_at > 10:05:00  ->  A and B invisible

Two rows lost, permanently, and no run ever fails. The mechanism is that updated_at orders rows by transaction start while visibility is ordered by commit, and the watermark is a high-water mark in the wrong ordering.

Evaluating the three fixes against this specific case:

Overlap the window by 15 minutes. Works here — both transactions committed within 14 minutes of their timestamps. It does not always work, and that is the answer the exercise is after: the overlap must exceed the longest possible transaction duration, which is a property of the source application that nobody has measured and that changes when someone ships a slow migration. An overlap window is a bet on a number you do not control.

Read the change log instead of the table (CDC). Works unconditionally, because the log is ordered by commit. This is the correct fix and it costs a connector, a replication slot, and Chapter 14.

Use a transaction-visible sequence (pg_current_snapshot(), txid, or an LSN). Works, and is cheaper than CDC. The catch is that it requires the source to expose one and the extract to store it instead of a timestamp — so it is only available if you own the source or can ask nicely.

2.11

The documentation sentence (the one that goes in the model's description):

fct_order_item has one row per order line. Adding any join that can match more than one row per order line changes the grain and inflates every additive measure.

The dbt test — marked where I am guessing at syntax I have not learned yet:

models:
  - name: fct_order_item
    description: "One row per order line. Grain: order_item_id."
    columns:
      - name: order_item_id
        tests: [unique, not_null]     # GUESS: exact test names
    tests:
      # GUESS: this is a package test, not built in
      - dbt_utils.unique_combination_of_columns:
          combination_of_columns: [order_id, line_number]

The row-count assertion, which is the one that survives being wrong about the YAML:

-- fails if the join fanned out
SELECT count(*) AS after_join, (SELECT count(*) FROM silver.order_items) AS before
  FROM fct_order_item
HAVING count(*) <> (SELECT count(*) FROM silver.order_items);
-- expect zero rows

Marking your guesses is the point of the exercise. An answer that silently invents plausible YAML is worse than one that says "I think it is unique_combination_of_columns and I would check."

2.13

How many objects rebuild: 22, one per dashboard, plus anything downstream of them.

How many could silently disagree: all 22. The definition lives in 22 places, so a change applied to 21 of them leaves one dashboard reporting the old measure — correctly, from its own point of view, with no error anywhere.

How you would find out: today, you would not. Somebody would notice two dashboards disagreeing, eventually, in a meeting. The maintenance cost is therefore not the 22 rebuilds — those are an afternoon — it is that the failure mode of the 22nd is silence.

one shared gold model, 22 dashboards on top:   1 place to change,  0 can disagree
22 pre-aggregated tables:                     22 places to change, 22 can disagree

The concrete fix: one gold.daily_revenue model owning the definition, with dashboards reading it. The boundary belongs at the point where the measure is defined, and pre-aggregating per dashboard moves it upstream of that point, which is §2.6's argument.

2.15 The argument that the lifecycle is an artifact of the batch era:

In a fully event-driven system, generation is ingestion — the application emits an event to a log, and there is no separate act of fetching. Storage is the log itself, so storage and ingestion collapse too. Transformation happens continuously in stream processors whose output is another log, so "transform" is no longer a stage between storage and serving but a set of always-on subscriptions. And serving is a materialised view maintained incrementally, so it is not downstream of transform in time — it is the same computation, read from a different end. On that account the five stages are five batches, and the boundaries between them are the boundaries between jobs, which an event-driven system does not have.

What survives the argument, and it is most of it. The stages are not descriptions of jobs; they are descriptions of questions with different owners, and every one of those questions still gets asked. An event-driven platform still has to know what the producer promises (generation), still has to decide retention and format for the log (storage), still has to get events across a trust boundary and know it got all of them (ingestion), still has to decide what a session means (transformation), and still has to decide who reads the projection and what breaks if it is wrong (serving).

What genuinely changes is coupling in time, not the set of concerns. Chapter 36 makes the same point from the other side: event-driven architecture moves the boundaries, and every boundary it removes it replaces with a schema, a consumer group, and a replay policy that somebody owns.

2.17 The incident: a reverse-ETL job pushes customer_lifetime_value into Kestrel's support tool, where agents see it beside every ticket. A bug in the join duplicates order rows, so CLV is inflated for a subset of customers — and for a smaller subset, a mis-keyed join attaches the wrong customer's CLV.

What the customer sees: nothing directly, and that is the problem. What they experience is an agent treating them differently — a retention offer they should not have got, or worse, a refusal because their inflated value flagged them for a different queue. In the mis-keyed case, a support agent is looking at a stranger's purchasing behaviour while talking to a real person.

Blast radius, compared to a dashboard. A wrong dashboard is read by a handful of internal people who have context and can say "that looks off." A wrong field in an operational tool is acted on, by people with no context, at the speed the tool is used, and the actions leave the building. A dashboard error is retracted with an email; a support interaction is not retractable at all.

The control I would add and would not bother with for a dashboard: a blocking pre-write assertion with a hard stop — the job compares the new values against the previous run and refuses to write if more than X% of records changed by more than Y%, and refuses outright on any null or out-of-range value. Plus an audit trail of what was written when, so that "what did the agent see on Tuesday" has an answer.

The general principle worth stating: reverse ETL turns an analytics artifact into a production dependency, and it should inherit production's controls, not analytics'. Chapter 27's staged deploy applies; a dashboard's "we'll fix it in the morning" does not.

2.19 For kestrel_app, the six questions from §2.2:

The three that require a conversation are 2, 3, and 6, and the exercise's real point is the phrasing. "Can you tell me about the orders table" invites a schema dump you already have. "When an order is cancelled, does the row stay?" is a yes/no question that a busy person can answer in one line, and it is the answer that determines whether your extract loses data.

Chapter 3 — Data Architecture Principles

3.1 An architectural decision is one whose reversal cost is high enough that you should write down why you made it.

Ranked by reversal cost, highest first:

1. cloud provider                 years, and a rewrite of everything operational
2. dimensional grain              every downstream model and dashboard rebuilds
3. partitioning scheme on the
   largest table                  a full rewrite of the data, plus every query re-tuned
4. orchestrator                   a rewrite of every DAG; the transforms survive
5. compression codec              a rewrite job, and nothing else changes

The interesting pair is 2 and 3. Grain outranks partitioning because a partitioning change is mechanical — expensive, but nobody has to decide anything — whereas changing grain invalidates every measure computed on top of it and every number anyone has quoted from it. Reversal cost is not compute cost; it is how many other decisions have to be re-made.

3.3 Because storage and compute became cheap and separable, so the reason to transform before loading — that you could not afford to store what you had not yet reduced — stopped applying.

3.5

All four "no"s have the same cause: a plain lake is a directory of files, and a directory has no transaction log. All four "yes"s have the same cause: adding one. That is worth stating plainly, because the lakehouse is often presented as four features when it is one mechanism.

3.7 The three most often omitted: the alternatives considered, the consequences (including the bad ones), and what would reverse this decision.

The most valuable is the last. Alternatives and consequences are usually recoverable — a knowledgeable reader can reconstruct them. A reversal condition cannot be reconstructed, because it encodes what the author knew about the future at the time, and it is the only section that turns an ADR from a historical record into an operational trigger: "if bronze exceeds 8 TB, revisit this" is a monitoring query. "If requirements change" is not.

3.9 The three follow-up questions:

1. "What action would you take on a stockout that you cannot take today?"
2. "How quickly can that action happen — how long from decision to effect in the warehouse?"
3. "How wrong can the number be before the action is wrong?"

The likely answers: "we'd reorder or reallocate between warehouses"; "a reallocation takes about four hours to pick and ship, and a reorder is days"; "we wouldn't act on a swing of under a hundred units."

What I would build: a 15-minute micro-batch, not a stream. The action has a four-hour horizon, so latency below about an hour buys nothing, and 15 minutes gives a comfortable margin at a fraction of the cost and operational surface of a streaming pipeline (§3.2, and Chapter 29's warning about what a stream costs to operate).

If the answer is group three — nobody acts on it, they just like watching it:

"We can have this at 15-minute freshness on the existing batch stack, which we can ship next sprint and can operate with the team we have. True real-time would mean a streaming pipeline and about a quarter of an engineer's ongoing time, so I'd like to start with 15 minutes and revisit if the lag actually blocks a decision — and if it does, that'll be a much easier case to make with a specific example."

Notice what that response does not do: it does not say no, and it does not deliver a lecture about latency. It offers a thing next sprint and names the condition under which the expensive version gets built.

3.11

# ADR-003 — Bronze clickstream retention: 2 years

Status: accepted · Date: 2026-01-20 · Deciders: data platform, with legal

## Context
Raw clickstream is 4.19 TB/year of JSON (11.48 GB/day). It is the only
faithful record of what the producer sent, and it is what every rebuild
of silver and gold reads from (Ch. 34).

## Decision
Retain raw bronze clickstream for 24 months, then delete. Parquet derived
from it is retained for 24 months on the same schedule.

## Alternatives
- 90 days. $23.76/month vs $192.74/month for two years -- a saving of
  $168.98/month, $2,027.76/year.
- Indefinite. Unbounded cost and an unbounded privacy surface (Ch. 31).
- 90 days raw + 2 years Parquet. Cheaper, and it loses the fidelity
  argument that is the whole reason bronze exists.

## Consequences
+ Any silver/gold model can be rebuilt over two years of history.
+ Year-over-year analysis is possible without a special archive.
- $2,028/year more than 90 days.
- Two years of raw personal data is two years of erasure obligation
  (Ch. 31), and erasure from bronze is the expensive kind.

## What would reverse this
1. Raw bronze clickstream exceeds 12 TB.
     SELECT sum(size_bytes) FROM s3_inventory
      WHERE prefix LIKE 'bronze/clickstream/%';   -- alert above 12e12
2. Twelve months pass with no query reading a partition older than
   180 days.
     SELECT max(current_date - partition_date) FROM query_log
      WHERE table = 'bronze.clickstream' AND query_date > current_date - 365;
3. Legal counsel sets a shorter statutory maximum.

Review: 2027-01-20.

The arithmetic, shown:

2 years:   4,190 GB x 2       =  8,380 GB x $0.023 = $192.74 / month
90 days:   4,190 GB x 90/365  =  1,033 GB x $0.023 =  $23.76 / month
                                                     ─────────
difference                                           $168.98 / month
                                                   $2,027.76 / year

And the sentence the ADR is really for: $2,028 a year is 0.55% of Kestrel's platform bill. The decision is not close on cost; it is close on the privacy consequence, which is why that line is in the consequences section rather than left implicit.

3.13 The case that a hybrid is a compromise:

You now operate two storage systems, two query engines, two access-control models, two cost meters, and two sets of on-call knowledge — with four data engineers. Every table near the boundary raises a question with no clean answer: does this model live in the lakehouse or the warehouse? Data is copied across the boundary, so there are two copies of gold facts and a synchronisation job that can fail; lineage tools see one half; and the "which system is authoritative" conversation recurs forever. A single warehouse would do everything Kestrel currently needs, and a compromise that doubles the operational surface to avoid a cost that is 2.1% of the bill is a bad trade.

The Kestrel property that makes the argument fail: the 4.19 TB/year of raw semi-structured clickstream, which the warehouse would charge warehouse rates to store and which has no schema stable enough to load. It is not the volume, it is the shape — the lake exists because bronze must accept what the producer sent without rejecting it (Chapter 34), and a warehouse is a schema-on-write system by construction.

What would have to change for the argument to succeed: if the clickstream were dropped, or schematised at the producer and stabilised by a contract (Chapter 17), the lake's only remaining job would be cheap bulk storage — and at that point the hybrid genuinely is a compromise and should be collapsed. The reversal condition is therefore not a volume threshold; it is "the clickstream has a stable, enforced schema."

3.15 A protocol that makes two concurrent writers safe without a table format:

1. Each writer takes an exclusive lease on the target prefix from an external store with compare-and-swap (DynamoDB, etcd, a Postgres row). The lease has a TTL and a fencing token that increments on every acquisition.
2. The writer writes to a staging prefix keyed by the token: .../_staging/token=71/.
3. It writes a manifest listing exactly the files that constitute the new state.
4. It atomically publishes the manifest by a compare-and-swap on a pointer object, conditional on the token still being current.
5. A reader reads the pointer, then the manifest, then only the files the manifest lists — never a directory listing.
6. A separate process garbage-collects staging prefixes whose token is stale.

Why every such protocol converges on a transaction log: the pointer in step 4 is a log of length one. The moment you need "what did this table look like yesterday" (time travel), or "apply this change to the previous state" (append, not replace), or two writers touching disjoint partitions concurrently, the single pointer becomes a sequence of atomic appends with a version number — which is a transaction log, arrived at by force.

The hardest part is not the log. It is step 6. Deciding when a staging file is safe to delete requires knowing that no reader still holds an old manifest, which requires either reader registration, or a conservative time bound, or accepting that long-running readers can fail. Every table format has this problem and every one of them solves it with a retention window and a VACUUM you have to remember to run — and getting it wrong deletes files out from under a query, which is the failure that makes people stop trusting the system.

3.17 (Answers vary — this is a worked example of the shape.)

The architecture: a mid-sized SaaS company's pipeline, described in a conference talk. Kafka → Flink → Iceberg → Trino, with a separate Spark cluster for ML feature generation, serving about 80 internal users.

The implicit ADR:

Decided: streaming-first ingestion into a lakehouse, with a separate
         batch engine for ML.
Alternatives probably considered: batch ELT into the warehouse they
         already ran; Kafka -> object storage -> dbt.
Consequences accepted: four systems, a dedicated platform team of six,
         and a latency profile nobody downstream requested.
What would reverse it: not stated. This is the tell.

The decision most likely copied from a larger company: Flink. Nothing described in the talk acted on data faster than hourly — every consumer was a dashboard or a daily model. Flink was chosen because the reference architecture had Flink in it, and the reference architecture came from a company with sub-second requirements and thirty platform engineers.

What it costs: a stateful stream processor is the most expensive thing on that list to operate — checkpointing, state backends, savepoints on every upgrade, and a debugging model that nobody learns in a quarter. Conservatively, one to two engineers of the six, permanently, to deliver latency that no consumer asked for.

3.19

# ADRs at Kestrel

**Write one when** a decision would be expensive to reverse: storage
format, dimensional grain, a tool you will operate, a retention period, a
partitioning scheme, or anything you expect to be asked to justify in a
year.

**Do not write one when** the decision is cheap to reverse or has an
obvious answer. A codec choice, a variable name, or "we'll use the library
everyone else uses" is a comment, not an ADR. **If writing it takes longer
than reversing it, do not write it.**

**How.** Copy `template.md`. Number sequentially, never reuse a number.
Keep it under two pages -- an ADR nobody finishes is an ADR nobody read.
Write it *before* the work, in `proposed` status.

**Review.** One other engineer and one person who will live with the
consequences. Not a committee. Comments go on the pull request; the
discussion is the value and the merged file is the record.

**Superseding.** Never edit an accepted ADR except to change its status.
Write a new one, and add `Superseded by ADR-0NN` to the old one. The wrong
decision, dated and explained, is more useful than a tidy record with no
history.

**The review date.** Every ADR carries one. On that date someone checks
the reversal conditions and either confirms, supersedes, or moves the date
with a reason. **An ADR whose review date has passed silently is a decision
nobody owns any more**, and that is the failure this process exists to
prevent.

Chapter 4 — Distributed Systems for Data Engineers

4.1 Success, failure, and timeout — and the third does not exist on a single machine. A local function call either returns or raises; it does not leave you not knowing whether it ran. Every hard problem in this chapter descends from that third outcome.

4.3 Replication lag is the interval between a write committing on the primary and becoming visible on a replica.

It loses rows in a watermark extract because the extract reads the replica and stores a watermark derived from what it saw: MAX(updated_at) over rows visible on the replica at that moment. Rows committed on the primary before that instant but not yet replicated carry timestamps below the new watermark and were not read. The next run filters updated_at > watermark and skips them permanently — and no run fails, because nothing about the query knows those rows exist.

4.5 What CAP actually says: in the presence of a network partition, a distributed system must choose between remaining available (serving requests that may return stale or conflicting data) and remaining consistent (refusing requests it cannot serve correctly). That is the whole theorem.

The two things people get wrong:

"Pick two of three." You do not choose P. Partitions are a property of networks, not a design option — the choice is only what you do when one happens, so the real menu has two items, not three.

"CA systems exist." A single-node database is not a CA distributed system; it is a system with no partitions to tolerate. And the trade-off does not apply when the network is healthy — a system described as "AP" is perfectly capable of being strongly consistent 99.99% of the time. CAP describes behaviour during a partition, and people quote it as though it described behaviour always.

4.7 At-most-once: delivered zero or one times; loss possible. At-least-once: delivered one or more times; duplication possible. Exactly-once: exactly one effect.

This book never relies on exactly-once because the guarantee stops at the boundary of the system that provides it — the moment a consumer writes to object storage, a warehouse, or Postgres, the write is outside the transaction that made the guarantee, and you are back to at-least-once with extra configuration.

4.10 The timeline:

replication lag = 4 minutes

02:00:00  primary: order X committed, updated_at = 02:00:00
02:01:00  EXTRACT reads the REPLICA. Replica is at 01:57:00, so X is
          not visible. Latest visible row: updated_at = 01:56:40.
02:01:01  extract stores watermark := 01:56:40      <- looks fine
02:04:00  X arrives on the replica, updated_at = 02:00:00

03:01:00  EXTRACT: WHERE updated_at > 01:56:40 ... picks up X.  OK!

That case is fine — and it is why the bug survives. The loss requires the watermark to advance past a row that has not arrived:

02:00:00  primary: order X committed, updated_at = 02:00:00
02:00:30  primary: order Y committed, updated_at = 02:00:30
02:00:31  Y replicates fast (small row, quiet moment). X does not
          (large transaction, still applying).
02:01:00  EXTRACT reads the replica: sees Y, not X.
          watermark := 02:00:30
02:04:00  X arrives, updated_at = 02:00:00
03:01:00  EXTRACT: WHERE updated_at > 02:00:30  ->  X NEVER READ

The condition is out-of-order arrival, which replication makes routine — apply order is not commit order once parallel apply, large transactions, or multiple replicas are involved.

Evaluating the three fixes:

The honest summary: two of the three fixes are mitigations and one is a solution, and the mitigation is usually the right choice anyway — which is a legitimate engineering answer as long as the residual risk is written down rather than forgotten.

4.12

(a) Read the unit of the handling time before you multiply — that is the exercise.

if 300 ms is per EVENT:   L = 2,900 x 0.300 = 870 concurrent handlers
if 300 ms is per BATCH
   of 100 (3 ms/event):   L = 2,900 x 0.003 =   8.7 concurrent handlers

A factor of 100 hangs on a word. Chapter 15 §15.8 uses the second reading, measured; take 870 for the rest of this answer because it is the version that makes (b) and (c) interesting.

(b) A Kafka partition is consumed by at most one consumer in a group, so 12 partitions permit at most 12 consumers. 870 concurrent handlers across 12 partitions means ~73 concurrent handlers inside each consumer process — threads, async tasks, or a worker pool.

The problem that creates is ordering. The per-partition ordering guarantee is the reason you chose a partition key at all, and it holds only if that partition is processed serially. Hand a partition's records to 73 workers and record 500 may commit before record 499. For Kestrel's clickstream that is usually tolerable; for anything where the events are state transitions on the same key it is a correctness bug, and it is one that appears only under load.

The resolution: parallelise within a partition by key. Hash each record's key to one of N in-process workers, so records sharing a key stay serial while different keys run concurrently. You keep per-key ordering, which is the guarantee anyone actually wanted, and you give up per-partition ordering, which nobody did.

(c) If you needed 40 concurrent handlers per partition, the options in order of preference:

Make the handler faster. 300 ms per event is enormous; it is usually a synchronous call to something. Batch the writes, or make the call async, and the requirement disappears. This is almost always the answer.

Key-partitioned in-process concurrency, as in (b) — 40 workers with a key hash.

More partitions. Cheap to do, and it breaks per-key ordering permanently for existing data, because the hash changes which partition a key lands in (Chapter 15 §15.2 and its 🔁 callout).

What not to do: more consumers than partitions. Consumers beyond the partition count sit idle.

4.14

What exactly-once genuinely provides. Kafka's transactional producer plus enable.idempotence is not marketing. Within Kafka, a consume-process-produce loop is genuinely atomic: offsets and output records commit together, so a crash mid-loop leaves neither a duplicate output nor a lost input. For Kafka-to-Kafka work it removes real deduplication code, and the colleague is right that this is a meaningful guarantee.

Where the guarantee stops: at the first write to something that is not Kafka. The consumer writes to PostgreSQL, and PostgreSQL is not a participant in Kafka's transaction. The sequence is: write to Postgres, commit the Kafka offset. Between those two steps the process can die — and it will, because that is what the third outcome in §4.1 means. The row is in Postgres and the offset is not committed, so the next consumer reprocesses it. No Kafka configuration changes this, because the problem is that two systems are committing separately.

What to do instead: keep at-least-once, and make the write idempotent. Key the Postgres write on something stable and let a re-delivery be a no-op:

INSERT INTO silver.events (event_id, ...)
VALUES (%s, ...)
ON CONFLICT (event_id) DO NOTHING;

This is strictly better than transactional exactly-once for this pipeline, and it is worth saying so rather than framing it as a fallback: it is simpler, it survives a Kafka upgrade, it survives a replay from an arbitrary offset, and it keeps working if the sink changes. The cost is one unique constraint. Then test it the way Chapter 4 §4.6 says: replay the same offsets and diff the table.

4.17 Why external systems break exactly-once. The guarantee is implemented by making the offset commit and the output write part of one atomic operation. That is possible when the message log owns both — Kafka can write the output records and the offsets to its own topics in one transaction. An external sink is not in that transaction, so the system must do two commits, and between any two commits there is an interval in which a crash leaves them disagreeing. There is no ordering of the two that removes the interval; there is only a choice of which duplicate-or-loss you prefer.

The two-phase-commit approach that would fix it in principle. A coordinator asks every participant — Kafka and the sink — to prepare: durably write the change and promise to be able to commit it, without making it visible. If all participants vote yes, the coordinator writes a commit decision and tells everyone to commit; if any votes no, everyone aborts. Because prepare is durable, a participant that crashes after voting can, on restart, ask the coordinator what was decided and finish. This genuinely gives atomicity across systems, and it is not a trick.

Why almost nobody uses it in data pipelines:

It requires participation. The sink must implement a prepared state that survives a crash and holds locks until told otherwise. Most data sinks do not — object storage certainly does not, and neither does a typical warehouse load.

It blocks. If the coordinator dies after prepare and before the decision, participants are stuck holding locks with no way to resolve. Recovery requires the coordinator to be highly available, which is another distributed system to operate.

Throughput collapses. Two round trips and durable writes per message, on a pipeline moving 14 million events a day.

And the alternative is very cheap. At-least-once plus a unique key is one constraint and one ON CONFLICT clause. 2PC solves a problem for which a much simpler solution exists, and that, rather than any of the technical objections, is the real reason.

4.19 (Answers vary — a worked example.)

The example: PostgreSQL's serial versus IDENTITY, and more sharply, the behaviour of REPEATABLE READ. A closer match to the S3 case: Linux's O_DIRECT and page-cache semantics, or TCP's silent change in default congestion control. Take a data-adjacent one: Hive's move from directory listing to a metastore-driven file listing, which invalidated a large body of advice about partition layout that assumed listing cost dominated.

What signal told practitioners: in the S3 case, a blog post and a documentation change, both easy to miss. In most cases, nothing. The system got better, code that compensated for the old behaviour kept working (compensating for a problem that no longer exists is harmless), and the compensation stayed in the codebase as an unexplained ritual — time.sleep(5) after a write, a retry loop around a read, a "consistency" reconciliation job that has found nothing in four years.

What the absence of a signal implies about how you write down assumptions. Three things, and the third is the one that matters:

Write the assumption, not the workaround. # S3 is eventually consistent, so retry is an assumption you can later check. A bare retry loop is a ritual.

Date it and name the source. "As of 2019, per " lets a future reader test whether it still holds without re-deriving it.

And put an expiry on it. The reason these survive is that nothing ever prompts a re-read. An assumption with a review date is the only kind that gets reviewed — which is the same argument as Chapter 3's ADR review date, applied one level down.

4.21 The register's shape, with the rows that can be filled on day one:

# Idempotency register

| # | Write operation | Target | Strategy | Key | Ch. |
|---|---|---|---|---|---|
| 1 | land raw orders | bronze.orders_raw | partition replacement | ingest_date | 13 |
| 2 | land raw clickstream | bronze/clickstream/ | partition replacement | event_date | 15 |
| 3 | CDC merge | silver.orders | merge | order_id, lsn + tiebreak | 14 |
| 4 | dedup + type | silver.order_items | delete+insert | order_item_id | 18 |
| 5 | sessionize | silver.sessions | partition replacement | session_date | 18 |
| 6 | build facts | gold.fct_order_line | merge, 90d lookback | order_line_id | 20 |
| 7 | SCD2 dimension | gold.dim_customer | snapshot | customer_id | 20 |
| 8 | daily rollup | gold.daily_revenue | partition replacement | revenue_date | 19 |
| 9 | reverse ETL | support tool | upsert by external id | customer_id | 2 |

Two rules make the register worth keeping.

Every row names a key, and the key must be unique at the grain of the write. A row whose key is "the whole row" is a row with no strategy.

And a blank cell is an open bug, not a gap in the documentation. Chapter 38 asks you to reread this table; if row 3's key still says lsn with no tiebreaker at that point, you have found the defect Chapter 38's Case Study 2 is about — which is precisely why the register is built in Chapter 4, before you know how to fill it in.

Chapter 5 — The Modern Data Stack

5.1

5.3 The heuristic: roughly two operated systems per engineer, above which the team spends more time maintaining than building.

Six engineers and fourteen systems is 2.33 systems per engineer — over the line, and the prediction is a team that feels permanently behind, where every quarter's roadmap is displaced by upgrades and incidents.

Its most obvious weakness: it counts systems, not weight. A managed service you have never opened and a self-hosted Kafka cluster both count as one, and they differ by an order of magnitude. The heuristic is a smell test, not a metric — and its real value is not the threshold but that it forces somebody to write the list down, which almost never exists.

5.5 Orchestration and quality are the two most expensive to lack.

Orchestration, because its absence is not felt as a missing tool — it is felt as cron entries with guessed offsets, no dependency graph, no retry semantics, and no answer to "did yesterday's run succeed." The failure is silent and the workaround degrades continuously.

Quality/observability, because without it the other failures are silent too. Every category in the list can be done badly and survive if something checks the output. Nothing checks the output.

5.7 Because wal_level is not changeable without a database restart, and a restart of a production OLTP database is a scheduled event that requires someone else's approval.

Setting it in Chapter 5 costs nothing — the sandbox is empty, nobody is connected, and the setting is inert until a replication slot exists. Deferring it to Chapter 14 converts a one-line config change into a change-management conversation, at exactly the moment you are trying to demonstrate that CDC works.

The transferable habit: when a setting is cheap now and expensive later, set it now. wal_level, partitioning on a table that will grow, a unique constraint, an event_id column nobody uses yet (§4.14). The cost of being early is a config line; the cost of being late is a maintenance window.

5.10 The specific failure the default produces: depends_on with the default condition (service_started) waits only for the container to start, not for the service inside it to be ready. MinIO's process starts in milliseconds and takes a second or two to accept API calls, so minio-init runs mc mb against a socket that is listening-or-not depending on scheduling, and the bucket creation fails.

Why it usually works on the second run: by then MinIO's image is cached, its volume is warm, and it comes up faster — and often the buckets already exist from a partially-successful first run, so even a failed mc mb leaves the stack looking correct.

Why "usually works on the second run" is dangerous. Three reasons, escalating:

It trains people to retry instead of to look. The failure becomes folklore — "just run it again" — and folklore is not debuggable.

It hides the actual dependency. Nobody learns that minio-init needs MinIO to be ready, so the same bug is reintroduced every time a service is added.

And it fails differently on a different machine. A slower laptop, a loaded CI runner, or a cold image pull changes the timing, so the thing that "works" locally fails in CI — and the failure looks like a CI problem rather than a missing health check. A race condition that resolves in your favour most of the time is worse than one that never works, because the latter gets fixed.

5.12

# platform/infra/.env.example
# Copy to .env and fill in. .env is gitignored and must stay that way.

# --- PostgreSQL (source OLTP) -------------------------------------------
POSTGRES_USER=
POSTGRES_PASSWORD=          # any value; local only
POSTGRES_DB=kestrel_app
POSTGRES_PORT=5432

# --- MinIO (S3-compatible object storage) -------------------------------
MINIO_ROOT_USER=            # >= 3 chars
MINIO_ROOT_PASSWORD=        # >= 8 chars, or MinIO refuses to start
MINIO_API_PORT=9000
MINIO_CONSOLE_PORT=9001

# --- Redpanda (Kafka API) -----------------------------------------------
KAFKA_BROKERS=redpanda:9092

# --- Airflow ------------------------------------------------------------
AIRFLOW_UID=50000           # must match your host uid on Linux; see App. A
AIRFLOW__CORE__FERNET_KEY=  # python -c "from cryptography.fernet import
                            #   Fernet; print(Fernet.generate_key().decode())"

Why .env.example is committed and .env is not, for someone who thinks that is redundant:

They serve different purposes and only one of them is a secret. .env.example is documentation — it is the authoritative list of what the stack requires, it is reviewed when someone adds a variable, and it is how a new person gets running without asking. .env is a credential file — it is machine-specific, it differs between developers, and committing it puts real values in git history, where they persist after deletion and travel with every clone and fork.

The redundancy objection has it backwards: the two files hold the same keys and deliberately different values, and the value is the whole point. The example file exists precisely so that the real one can stay out of the repository without the requirements becoming tribal knowledge.

And there is a test for whether it is working: delete your .env, recreate it from the example alone, and bring the stack up. If you cannot, the example is incomplete — which is a bug in documentation that you just found for free.

5.14

# Memo: proposal to optimise the analytics query path

**Ask:** two engineers, one quarter. Estimated: nine engineer-months.

## The problem, as reported
The `customer_activity` query takes 4 seconds. Analysts describe it as slow.

## What I measured before writing this
- Dataset: 15.5 GB. Fits in memory on the machine we already have.
- Query volume: 340 runs/month, 11 distinct users.
- p50 4.1 s, p99 6.2 s. No timeouts in 90 days.
- Total human time spent waiting: 340 x 4.1 s = ~23 minutes/month.

## Options
A. Do nothing. Cost: 23 minutes/month of waiting, spread over 11 people.
B. Add two indexes and rewrite one subquery. ~3 days. Expected: 4s -> ~2s.
C. Migrate to a distributed engine. ~9 engineer-months, plus a permanent
   operational surface (Ch. 5 section 5.1: we would go from 6 systems to 7
   with 4 engineers).

## Recommendation
B, and revisit if the dataset passes 200 GB or query volume passes
5,000/month.

## The question this memo exists to ask
**What will be true after this project that is not true now, and who will
notice?**

The honest answer for C is: a query that eleven people run fifteen times a
month each will take two seconds less. **Nobody will notice.** If the
answer to that question is "nobody," the project should not start --
and that is the only paragraph of this memo that matters.

5.16 Five things a catalog product does that a Markdown file does not:

The common thread is that four of the five are things a document cannot do at all, because they require observing the system rather than describing it. The Markdown file's weakness is not that it is less detailed. It is that it cannot be wrong out loud — it drifts silently, and a stale catalog entry is indistinguishable from a current one.

Where the balance flips: roughly eight to ten engineers, or the moment more than one team writes to the warehouse. The determining variable is not table count; it is the number of people who can change something without you knowing. With four engineers in one room, that number is small and a document plus a conversation covers it. With two teams, it is unbounded, and the drift rate exceeds anyone's ability to maintain a file by hand.

5.18 (The seven criteria, applied honestly.)

dbt

Airflow

Which would be harder to leave: Airflow, decisively — and the reason is instructive. dbt's lock-in is in tooling around artifacts that remain readable; Airflow's is in logic that only exists inside DAG files. A dbt project can be read by someone who knows SQL. An Airflow deployment of any age contains business rules in Python operators that exist nowhere else.

Does that change my view of adopting it? It changes how I would adopt it, not whether. Orchestration is one of the two categories §5.3 says is most expensive to lack. But knowing the exit cost sits in the DAGs rather than the scheduler tells you where the discipline goes: keep DAGs thin. A DAG should call things, not compute things. Every line of business logic that stays out of an operator is a line you do not have to port, and that rule is worth writing down on day one — because it is free then and expensive in year three.

5.20

# Tool decisions

| Category | Kestrel uses | B/B/W | Reasoning | What we give up |
|---|---|---|---|---|
| Storage | S3 (MinIO locally) | Buy | Cheap, universal, decoupled from compute | Nothing meaningful |
| Compute | DuckDB + Snowflake | Buy | One machine covers most of it; the warehouse covers concurrency | A Spark-shaped escape hatch we would have to build under pressure |
| Ingestion | Python + Debezium | Build | Three sources; a vendor tool costs more than the code | Connector maintenance is ours, forever |
| Transformation | dbt | Buy | Version control, tests, lineage, and everyone can read SQL | Non-SQL transforms have no home |
| Orchestration | Airflow | Buy | Dependencies, retries, backfills, and a managed option exists | Real operational weight; DAG lock-in (see 5.18) |
| Quality | dbt tests + assertions | Build | Adequate at this size; lives with the models | Anomaly detection, profiling, and freshness we did not write |
| Catalog | a Markdown file | **Do without** | Four engineers in one room | Lineage, usage stats, drift detection (see 5.16) |
| BI | one vendor tool | Buy | It is where the consumers already are | Definitions can be written in the BI layer, outside dbt |

## Where we do without, and what would change our mind

**Catalog.** Adopt when the team passes eight engineers, **or** when a second
team gains write access to the warehouse -- whichever comes first. The
second is the real trigger.

**A dedicated quality tool.** Adopt when we have had two incidents in a
quarter that a freshness or volume anomaly check would have caught. Not
before: we would be buying a tool to solve a problem we have not measured.

The column that makes this document worth writing is the last one. A tool list without "what we give up" is a list of things you like, and it cannot be reviewed. A row that gives up nothing is a row someone has not thought about.

Chapter 6 — Data Modeling for Analytics

6.1

1. Select the business PROCESS      -- what event are we measuring?
2. Declare the GRAIN                -- what does one row mean?
3. Identify the DIMENSIONS          -- by what can we slice it?
4. Identify the FACTS               -- what do we measure?

What goes wrong if you do each before its predecessor:

Grain before process: you declare "one row per order line" without having agreed which business event you are measuring, so nobody can say whether a cancelled order belongs in the table. The grain is a sentence about a process; without the process it is a sentence about a table.

Dimensions before grain: you attach a dimension that is finer than the grain — a promotion that applies per line to a table at order grain — and the join fans out. Every fan-out bug is this step done out of order (§6.9).

Facts before dimensions: you choose measures without knowing what you will slice by, and discover that gross_margin_pct cannot be averaged across any of them (see 6.3). Additivity is a property of a measure relative to a set of dimensions, so the question is unanswerable in the wrong order.

6.3

The two that trip people are the semi-additive pair, and they share a shape: a level measured at a point in time. Summing yesterday's and today's on-hand units counts the same physical items twice. The correct aggregation across time is LAST or AVG, never SUM, and no schema can enforce that — only documentation and a suspicious reviewer.

6.5 The four reasons for a surrogate key:

1. It lets the dimension have history. With Type 2, one natural key maps to several rows; the fact must point at which version, and only a surrogate can.
2. It insulates you from source key changes — reused ids, a system migration, a merger.
3. It is compact and uniform. An integer join beats a composite natural key of three varchars.
4. It handles the unknown member. A -1 row for "not yet known" keeps facts joinable without nulls.

Reason 1 is decisive. The others are optimisations you could live without; without a surrogate key, Type 2 history is impossible, and everything else follows from that.

What they cost: an extra join for anyone who knows the natural key and not the surrogate; a lookup step in every load; keys that are meaningless in isolation, so a debugging session gains a hop; and — the underrated one — a fact table whose keys mean nothing without the dimension, which makes the fact table alone unreadable. That is a real loss when someone is reading raw rows at 2 a.m.

6.7 A degenerate dimension is a dimension attribute that lives on the fact table because its dimension would have no other columns — typically a transaction identifier.

Kestrel's example: order_id on fct_order_line. A dim_order would contain order_id and nothing else; every other order attribute is either a dimension of its own (customer, date, channel) or a measure.

Why it is kept: it is the grouping key that reconstructs the transaction. "How many lines per order," "orders containing more than one category," and every reconciliation against the source system (Chapter 38) need it. Dropping it because it "is not a dimension" makes the fact table impossible to tie back to the source, which is the one thing a fact table must always support.

6.10

(a)

rows/day  = 47,000 SKUs x 3 warehouses            =     141,000
rows/year = 141,000 x 365                         =  51,465,000

(b)

51,465,000 rows x 90 bytes = 4,631,850,000 bytes
                           = 4,631,850,000 / 2^30
                           = 4.31 GB per year

4.31 GB x $0.023/GB-month  = $0.099 per month
                           = $1.19 per year

(c) Storing only rows where the quantity changed:

Two things it makes harder:

Every point-in-time question needs a window function instead of a filter. "What was on hand on 2026-06-14" becomes "the last row at or before 2026-06-14, per product per warehouse" — a ROW_NUMBER() over a partition rather than WHERE snapshot_date = '2026-06-14'. Every consumer has to know this, and any consumer who does not will silently get too few rows.

"No row" becomes ambiguous. A missing row now means either unchanged or never stocked or the pipeline failed, and those are indistinguishable without a separate record of which days the job ran. A dense snapshot makes a gap an alarm; a sparse one makes it normal.

The decision: keep the dense daily snapshot. It costs $1.19 a year. The proposal trades a figure that rounds to zero against a permanent complexity tax on every query and a lost failure signal. This is the exercise's whole point — compute the number before you optimise, and here the number ends the discussion in one line.

(The transferable version: sparse snapshots are right when the density is genuinely enormous — billions of rows a day — and even then the missing-row ambiguity has to be solved separately.)

6.12 Diagnosis. A promotion applies to an order or a line, and an order line can carry more than one promotion — a category discount and a coupon, say. Joining fct_order_line to a promotions table on order_line_id therefore produces one row per line per promotion, duplicating net_revenue_cents for every line with two. 31% is the share of revenue on multi-promotion lines, counted twice.

The pattern is §6.9's multi-valued dimension, and the tell is that the total changed when a dimension was added. Adding a dimension must never change a measure; if it does, the join changed the grain.

The fix — a bridge table with allocation weights:

CREATE TABLE bridge_line_promotion AS
SELECT order_line_id,
       promotion_id,
       -- equal allocation. Any rule works; it must sum to 1.0 per line.
       1.0 / COUNT(*) OVER (PARTITION BY order_line_id) AS weight
  FROM stg_line_promotions;

-- revenue by promotion, allocated:
SELECT p.promotion_name,
       SUM(f.net_revenue_cents * b.weight) AS allocated_revenue_cents
  FROM fct_order_line f
  JOIN bridge_line_promotion b USING (order_line_id)
  JOIN dim_promotion         p USING (promotion_id)
 GROUP BY 1;

The test that the weights are right:

-- every line's weights must sum to exactly 1.0
SELECT order_line_id, SUM(weight) AS w
  FROM bridge_line_promotion
 GROUP BY 1
HAVING ABS(SUM(weight) - 1.0) > 1e-9;
-- expect zero rows

And the second test, which people forget: total allocated revenue must equal total revenue.

SELECT (SELECT SUM(net_revenue_cents * weight)
          FROM fct_order_line JOIN bridge_line_promotion USING (order_line_id))
     - (SELECT SUM(net_revenue_cents) FROM fct_order_line
         WHERE order_line_id IN (SELECT order_line_id
                                   FROM bridge_line_promotion)) AS drift;
-- expect 0

Two things worth saying about allocation. Equal weights are a choice, not a fact — allocating a $10 coupon and a 5% category discount equally is defensible and arbitrary, and the choice belongs to finance, not to the modeller. And the allocated total is not comparable to the unallocated one for any single promotion; only the sum is. Write both facts into the model's description, because a consumer who does not know they are looking at allocated revenue will draw a wrong conclusion from a correct number.

6.14 (Answers vary. A worked example: a public library.)

                          dim →  Date  Patron  Item  Branch  Staff  Vendor  Program
process ↓
Checkout                          X      X      X      X       X
Return                            X      X      X      X       X
Hold placed                       X      X      X      X
Acquisition                       X             X      X               X
Fine assessed                     X      X      X      X       X
Program attendance                X      X             X       X               X
Computer session                  X      X             X

The dimension used by the most processes: Date — as it always is, in every bus matrix anyone has ever built, which is why conformed date is the first dimension to get right.

The two processes sharing the fewest dimensions: Acquisition and Computer session. They share Date and Branch and nothing else — no patron, no staff, and Acquisition has no item-in-hand in the sense the others mean.

A comparison the business wants and the matrix says is impossible: "do patrons who attend programmes borrow more?" is answerable (both have Patron). "Does buying more copies of a title reduce hold wait times for it?" is not — Acquisition is at title/vendor grain with no patron, and Hold is at patron/item grain, so the two can only be joined through Item, and Item in Acquisition means a purchase order line while in Hold it means a physical copy. The dimension has the same name and a different grain in the two processes, which is exactly the conforming failure Chapter 2 §2.5 describes — and the matrix surfaces it before anyone writes the query.

6.17 The erasure procedure for a Type 2 dim_customer:

DELETE     nothing. Deleting rows breaks fact-table joins on customer_sk.

NULL /     every attribute that identifies or describes the person, in
REDACT     EVERY version row for that natural key:
             name, email, phone, address lines, postal_code, date_of_birth,
             any free-text field, and any derived attribute that narrows
             identity (precise geo, device id).

PRESERVE   customer_sk, customer_id (as an opaque, non-reversible token
           or a retained key -- see the caveat), valid_from, valid_to,
           is_current, and any coarse attribute retained under a stated
           lawful basis.

SET        an `erased_at` timestamp and `is_erased = true` on every version.

Why not delete: every historical fact row points at a customer_sk. Deleting the dimension rows orphans them, which either breaks the join or silently drops facts from every aggregate — turning a privacy action into a revenue restatement. Nulling the attributes removes the personal data while leaving the referential structure intact.

The part specific to Type 2, and the part people miss: every version must be redacted, not just the current one. A prior version holds the address the person lived at in 2024, which is personal data with a longer memory than the current row. An erasure that updates WHERE is_current is a bug that looks like a fix.

How you verify completeness:

-- 1. no version of an erased customer retains an identifying attribute
SELECT customer_sk FROM dim_customer
 WHERE is_erased
   AND (email IS NOT NULL OR full_name IS NOT NULL OR postal_code IS NOT NULL);
-- expect zero rows, across ALL versions

-- 2. no fact row is orphaned
SELECT count(*) FROM fct_order_line f
  LEFT JOIN dim_customer d USING (customer_sk)
 WHERE d.customer_sk IS NULL;
-- expect 0

-- 3. the row survives in every downstream copy
--    (Ch. 31: the manifest must be GENERATED, not written)

Which choices I would want a lawyer to confirm. Three, and they are genuinely legal rather than technical:

Whether retaining customer_id as a key is permissible. It is a pseudonymous identifier linking to purchase history. Under some readings that is still personal data and must itself be broken; under others, retention for legitimate accounting purposes is lawful. This is the load-bearing question and it is not an engineering one.

Whether transaction records may be retained at all. Tax and consumer-protection law frequently requires retaining order records for years, which can override an erasure request — but the scope of that exemption is jurisdictional.

And whether "erased" must mean removed from backups. The engineering answer is "backups expire in 90 days"; whether that satisfies the obligation is a question with a real answer that varies.

6.19 The argument against storing net_revenue_cents:

A stored derived column can disagree with its own definition, and a computed one cannot. The moment the value is materialised, there are two sources of truth — the stored column and the formula — and they drift the first time the formula changes without a backfill. The failure is silent and selective: rows written before the change hold the old definition, rows after hold the new one, and the table now contains two different measures under one name with no marker distinguishing them.

Nothing analogous can happen with a computed column, because there is only ever one definition and it is applied at read time.

Three ways the drift actually arises, all ordinary: a rule change applied to the model but not backfilled; a partial backfill that failed halfway and was not retried; and an incremental model whose lookback window is shorter than the range the change affects (Chapter 20 §20.7) — which restates recent rows and leaves older ones on the old rule, producing a table that is internally inconsistent at a boundary nobody can see.

The test that closes the gap:

-- recompute from the stored components and compare. Runs on every build.
SELECT order_line_id,
       net_revenue_cents AS stored,
       (gross_cents - discount_cents - refund_cents) AS recomputed
  FROM fct_order_line
 WHERE net_revenue_cents <> (gross_cents - discount_cents - refund_cents);
-- expect zero rows

Two properties make that test worth its cost. It runs over the whole table, not the incremental window — which is the only way it can catch the partial-backfill case, and it is worth the scan. And it fails at the boundary, naming the exact rows where the two definitions disagree, which turns "revenue looks odd" into a row list in one query.

So: store it, and test it. Storage buys query simplicity and protects consumers from reimplementing the formula nine different ways in the BI layer — which is the failure the computed-column purist never sees coming, and the more common one.

6.21

# Definitions — Kestrel

## 1. What counts as an order
**Definition.** An order is a customer-submitted purchase that reached
`status = 'paid'` at least once. Test orders (`is_test`) and orders that
never left `pending` are excluded.
**Rejected.** "Every row in `orders`." It includes abandoned and test
orders, and it is the definition the raw table invites.
**Who must agree.** Finance owns it; commerce engineering must confirm the
`is_test` flag is reliable. (Ch. 38's R1 and R2 are this rule, enforced.)

## 2. `net_revenue_cents`
**Definition.** Gross line amount, minus allocated discounts, minus refunds
attributed to the month the item was SOLD -- not the month refunded. Gift
card sales are excluded; a gift card is a liability until redeemed.
**Rejected.** Netting refunds in the month they occur. Simpler, and it makes
a closed month move.
**Who must agree.** Finance, and it is not negotiable by the data team.

## 3. Which region a customer belongs to
**Definition.** The region of the **shipping address on their most recent
order**, not the billing address and not the signup address.
**Rejected.** Billing address -- stable, and frequently a corporate office
in a different region.
**Who must agree.** Marketing (they segment on it) and Finance (they report
tax by it), and **these two may want different answers**, which is itself
the finding: if so, we need two attributes with two names, not one
contested one.

## 4. Promotions: per order or per line
**Definition.** Per **line**, with order-level promotions allocated across
lines in proportion to line gross amount, weights summing to 1.0.
**Rejected.** Per order. It cannot answer "which categories did this
promotion move," which is the question promotions are analysed for.
**Who must agree.** Merchandising, plus Finance on the allocation rule.

## 5. When a session ends
**Definition.** 30 minutes of inactivity, **or midnight UTC**, whichever
comes first.
**Rejected.** Inactivity alone. Sessions then straddle days and daily
session counts stop summing to the total.
**Who must agree.** Analytics owns it. Written down because it is arbitrary,
and an arbitrary rule that is not written down gets re-litigated annually.

The column that makes this file survive three technology migrations is "rejected." A definition without its alternative reads as the obvious choice, so the next person re-opens it. A definition with the rejected alternative and the reason reads as a decision, and decisions are much harder to accidentally reverse.

Part II — Storage

Chapter 7 — Relational Databases as Sources

7.1 The six consequences of being a guest:

1. You do not get to add indexes — every index costs the owning team write throughput forever.
2. You do not get to change the schema, including adding the updated_at that would make your life easy.
3. Your query competes for the same resources — buffer cache, I/O, CPU, connection slots — and the cache eviction outlasts your query.
4. You are subject to their maintenance — failovers, upgrades, vacuum, index rebuilds, on their schedule.
5. Your read-only query can still cause an incident (§7.3).
6. When something goes wrong it will be attributed to you, because you are the unusual workload.

The one that surprises people is the fifth. "Read-only" reads as "safe," and it is not: a long SELECT holds a snapshot, and a held snapshot prevents vacuum from reclaiming dead tuples across the whole database. A query that changes nothing can bloat a table the application depends on, and the mechanism is entirely invisible from the query's side.

7.3

The middle case is the one worth understanding. A bitmap scan collects matching tuple ids from the index, sorts them, then reads the heap in physical order — turning what would be thousands of random reads into a sequential sweep of the pages that matter. It is the planner saying "the index is useful for finding rows and useless for the order I want to read them in."

7.5 NUMERIC/DECIMAL, or an integer count of the smallest unit (cents).

Choose NUMERIC when the arithmetic involves division, percentages, or currency conversion, or when the value can legitimately have sub-cent precision — a unit price of $0.0325, an FX rate, an allocation weight. Choose integer cents for anything that is a stored amount of money that will be summed: order totals, line amounts, refunds, revenue.

The three costs of integer cents:

Every read site must divide, and every one is a chance to divide wrong — or to forget, and display 2194520200 to a human.

Division is lossy and the loss has to go somewhere. Splitting $10.00 three ways gives 333, 333, 334 — and deciding where the extra cent goes is a business rule that now lives in your code (§6.12's allocation weights are the same problem).

Sub-cent inputs must be rounded at the boundary, and the rounding rule (half-up, half-even, toward zero) is a decision someone has to make and write down. A codebase where different call sites round differently will not reconcile, and the discrepancy will be small enough to be dismissed for years.

7.7 UUID v4 is random, so consecutive inserts land in random positions of the primary key's B-tree. Every insert dirties a different page, the working set of the index becomes the whole index rather than its right-hand edge, and pages split near-randomly — leaving them roughly half full, so the index is larger than it needs to be and less of it fits in cache.

UUID v7 is time-ordered in its high bits. Consecutive inserts are monotonically increasing, so they append to the rightmost leaf page — one hot page, sequential splits, dense pages, and a tiny working set. The same behaviour a BIGSERIAL gives you, with the collision-free-across-systems property that made you want a UUID.

And v7 has a second, quieter benefit: because the key sorts by time, range scans over "recently created" rows become index range scans instead of full scans.

7.10 Index on orders (customer_id, placed_at):

(a) WHERE customer_id = 8841 — fully used. Equality on the leading column; the index seeks directly to that range.

(b) WHERE placed_at >= '2025-11-01' — not usable as a seek. The leading column is unconstrained, so there is no starting point in the B-tree. PostgreSQL may still choose a full index scan if the index is much narrower than the heap and the query is index-only, but that is "reading the whole index instead of the whole table," not pruning.

(c) WHERE customer_id = 8841 AND placed_at >= '2025-11-01' — fully used, and this is the shape the index was built for. Equality on the first column plus a range on the second means the matching rows are one contiguous stretch of the index.

(d) WHERE customer_id IN (8841, 9002) AND placed_at >= ... — used, slightly less efficiently. The planner runs two range scans (a ScalarArrayOp on the index, or a BitmapOr) and unions them. Each sub-scan is as efficient as (c); the cost is one extra descent of the tree plus the merge. Interesting property: this degrades gracefully with the list length and then falls off a cliff — at some size the planner decides a sequential scan is cheaper, and the plan changes shape without warning.

(e) WHERE lower(status) = 'paid' AND customer_id = 8841 — partially used. The index handles customer_id; lower(status) is not in the index and cannot be, so those rows are fetched and filtered afterwards. Wrapping a column in a function makes it invisible to an index — the same mechanism as Chapter 1's CAST(event_ts AS DATE) defeating partition pruning, one storage layer down. The fix, if it matters, is an expression index on lower(status), and you do not get to create it (§7.1).

7.12

UNCHUNKED, 90 minutes
  updates during the extract   1,200,000 /h x 1.5 h  = 1,800,000 dead tuples
  dead space                   1,800,000 x 210 bytes =   378.0 MB

CHUNKED, 15-minute chunks
  updates during ONE chunk     1,200,000 /h x 0.25 h =   300,000 dead tuples
  dead space held at once        300,000 x 210 bytes =    63.0 MB

  ratio                                                        6 : 1

What the chunking does not fix, and this is the point of the exercise:

It does not reduce the total churn. The application still updates 1,800,000 rows during those 90 minutes. What changes is the peak unreclaimable space: between chunks the snapshot is released and autovacuum can do its job, so the high-water mark is 63 MB instead of 378 MB. The 6:1 is a ratio of peak bloat, not of work done, and quoting it as "chunking made the extract six times cheaper" is wrong.

It does not make the extract consistent. Each chunk reads a different snapshot. A row updated between chunk 3 and chunk 4 can be read twice (if it moves forward across the chunk boundary) or not at all (if it moves backward), depending on the ordering key. You have traded a consistency guarantee for a bloat guarantee, and that trade must be a deliberate one — it is acceptable when the extract is followed by a deduplicating merge (Chapter 14 §14.8) and unacceptable when it is not.

It does not reduce load on the source. The same bytes are read, plus per-chunk planning overhead, and the wall clock is slightly longer.

And it does nothing about anybody else's long transaction. A forgotten psql session, a replica with hot_standby_feedback on, or another team's report holds the horizon open regardless of how politely you chunk. Bloat is a property of the oldest snapshot in the system, not of your query.

7.14 What hot_standby_feedback does when on. The standby continuously reports the oldest transaction it needs to the primary, and the primary declines to vacuum tuples still visible to that snapshot. The consequence is that a long query on the replica prevents cleanup on the primary — the replica's workload now creates bloat in the application's database. The benefit is that queries on the standby are never cancelled by a replication conflict, which is what happens when the primary vacuums a row the standby's snapshot still needs.

What it does when off. The primary vacuums on its own schedule, ignoring the standby. Bloat stays a primary-side concern. The cost is that a long-running query on the standby will be cancelled — ERROR: canceling statement due to conflict with recovery — when the primary removes rows that the query's snapshot requires, and there is no way to know in advance which query will be hit.

For a replica used only for analytical extracts, I would set it off, and pair it with a raised max_standby_streaming_delay (so short conflicts wait rather than cancel) plus chunked extracts (§7.7) so that no single snapshot is held long enough to collide.

What you give up either way, stated plainly. With it on, you have moved a failure from your side to theirs: your extract always succeeds, and the application team's table bloats, which violates §7.1's entire premise and is the kind of thing that gets your access revoked. With it off, the failure is yours: extracts get cancelled unpredictably, and you need retry logic and idempotent writes to survive it. The second is the right trade because it keeps the cost with the party who can control it — and because a cancelled extract is loud, while bloat is silent.

7.17 The report:

-- Which keys appear in the JSONB column, how often, and as what type?
WITH kv AS (
    SELECT j.key,
           jsonb_typeof(j.value) AS value_type
      FROM events_raw e,
           LATERAL jsonb_each(e.payload) AS j
     WHERE e.ingested_at >= now() - interval '30 days'
), total AS (SELECT count(*) AS n FROM events_raw
              WHERE ingested_at >= now() - interval '30 days')
SELECT kv.key,
       count(*)                                   AS occurrences,
       round(100.0 * count(*) / max(total.n), 2)  AS pct_of_rows,
       count(DISTINCT kv.value_type)              AS distinct_types,
       string_agg(DISTINCT kv.value_type, ', ')   AS types
  FROM kv CROSS JOIN total
 GROUP BY kv.key
 ORDER BY occurrences DESC;

Read two columns, not one. pct_of_rows says whether the key is really optional. distinct_types says whether it has a type — a key that appears as both string and number is the schemaless trap in its pure form, and it is invisible until a downstream cast fails on 0.4% of rows.

The threshold I would promote at: a key present in more than 95% of rows for more than 30 consecutive days, with exactly one value type.

The defence. 95% rather than 100% because a genuinely required field will still be absent in a handful of malformed rows, and demanding 100% means never promoting anything. Thirty days rather than a snapshot because the failure mode being guarded against is promoting a key that some client version happens to be sending this week. One value type is non-negotiable — a key with two types cannot become a column without a decision about which one wins, and that decision belongs to whoever owns the producer, not to whoever runs the report.

And the honest caveat: the threshold is a heuristic and its real job is to force a conversation. The strongest signal is not in the query at all — it is that someone downstream has written payload->>'x' in three different models. A key extracted three times in SQL is a column, whatever the percentage says.

7.19 Where NUMERIC is genuinely better. Anything where the stored value is not a whole number of cents: a per-unit price in a catalogue with sub-cent precision — Kestrel's wholesale cost basis is quoted to four decimal places, and $0.0325 per gram is not representable in cents. Storing it as cents means rounding at input, and then a 10,000-unit line item is wrong by dollars. Tax rates, FX rates, and allocation weights are the same case. NUMERIC(18,6) stores them exactly and sums them exactly.

Where a float genuinely is acceptable. When the value is a measurement or a statistic, not money: a model score, a similarity distance, a p99 latency, a conversion rate. These are estimates whose last digits are noise, they are never summed to a figure anyone reconciles, and the performance and storage advantages are real. The test is whether anyone will ever compare the total to another system's total. Nobody reconciles an average.

The rule to give a team, written for people who will not read the reasoning:

Money that gets summed is an integer of the smallest unit. Money that gets multiplied or divided is NUMERIC. Floats are for measurements, never for money.

If you are unsure, use NUMERIC. It is slower and it is never wrong.

Two properties make that rule survivable. It has a default — the unsure case has an answer, so nobody has to think at 4 p.m. on a Friday. And the default is the safe one: NUMERIC costs performance and never costs correctness, so following the rule badly still produces a correct system. A rule whose failure mode is a wrong number is not a rule, it is a hope.

7.21 (Implementation exercise — the reporting shape and the expected result.)

The measurement to record:

                      n_dead_tup on orders    n_dead_tup after     peak
                      before                  extract completed    delta
unchunked, 74 min           41,208               1,486,003      1,444,795
chunked, 15 min             41,208                 118,447         77,239

Two things to look for in your own numbers. The chunked delta should be roughly one chunk's worth of churn, not one-sixth of the unchunked figure — autovacuum reclaims between chunks, so the shape is a sawtooth rather than a ramp, and the peak is what matters. And the totals should be similar: the same rows were updated either way (Exercise 7.12). If your chunked total churn is much lower, the write load was not steady and the comparison is not clean.

The --dry-run flag is the part worth building carefully. It should print the exact ranges and row-count estimates without opening a transaction on the source, because the whole point of a dry run against a database you are a guest in is that it costs the owner nothing.

Chapter 8 — Data Warehouses

8.1 Five differences: column-oriented storage rather than row-oriented; the unit of I/O is a column chunk rather than an 8 KB page of whole rows; optimised for scans and aggregations rather than point reads and writes; dozens of large queries rather than thousands of small transactions; usually no B-tree indexes, zone maps instead; updates in bulk rather than in place; and compute scales independently of storage.

The one that surprises people is the absence of indexes. Every instinct from Chapter 7 says that a table without an index is a table you cannot query, and in a warehouse it is the normal case — a columnar scan with zone-map skipping is cheap enough that indexed lookup is rarely the win.

Why it matters for Chapter 23: because it removes the mechanism people unconsciously rely on for data quality. In an OLTP database a unique index is both a performance structure and a correctness constraint, and it fails loudly on a duplicate. Warehouses generally do not enforce uniqueness, primary keys, or foreign keys — several accept the DDL and ignore it. Nothing stops a duplicate row from being written, which is why Chapter 23 has to assert in tests what Chapter 7 got from the schema. Kestrel's 31-day duplicate-rows incident is that gap.

8.3

The pattern worth extracting: every one of the four exploits a property of one column's values — repetition, low cardinality, monotonicity, small range. A row-oriented format cannot exploit any of them, because adjacent bytes belong to different columns with nothing in common. That is the actual reason columnar compresses better, and it is more interesting than "columns compress well."

8.5 What separated storage and compute enables:

1. Independent scaling — add compute without adding storage, and vice versa.
2. Workload isolation — ETL, BI, and data science on separate compute over one copy of the data, so a runaway query cannot slow a dashboard.
3. Elasticity — size for the job, suspend when idle, pay for what ran.
4. One copy of the data — no redistribution, no per-team extract, no reconciliation between copies.

Three costs:

Latency and cost per byte read. Data lives on the network, not on local disk. Caching hides most of it and does not hide a cold query.

Idle compute is now possible. In the coupled model a node you owned was always there; in this one a warehouse can be up and doing nothing, which is a new failure mode that costs real money (§8.5's $70,080, and Chapter 33's $5,040/month).

The cost model becomes non-obvious. The bill is no longer "how many machines" — it is credit- seconds or bytes scanned, and it responds to query shape in ways nobody can predict without measuring. Elasticity is a feature that requires a discipline, and organisations acquire the feature before the discipline.

8.7 The working range is 100–250 MB compressed, per file.

Too small and per-file overhead dominates: 100,000 files of 100 KB load dramatically slower than 100 files of 100 MB for the same bytes. Every file costs a request, a task assignment, and a footer read, and none of that scales with size.

Too large and you lose parallelism, because a single file is generally processed by a single thread. One 40 GB file is one worker while the rest of the cluster idles.

And the shape of the curve is asymmetric, which is the practical point: too-small is a cliff — 100 KB files are catastrophically slow — while too-large degrades gently. When in doubt, err large.

8.10 (Measurement exercise.) The reporting shape, and what to look for:

                          size      query A (filter)   query B (aggregate)
unsorted parquet         112 MB          0.84 s              0.61 s
sorted by placed_at       97 MB          0.19 s              0.58 s
sorted by customer_id    101 MB          0.81 s              0.59 s

What sorting alone bought: on the size axis, about 13% — delta and RLE encodings work better on ordered values (§8.3). On the query axis it bought a factor of four, and only for the query whose predicate matched the sort key.

Did the ranking match? No, and that is the finding. Sorting by customer_id produced a file almost as small as sorting by placed_at and gave no query benefit at all, because the filter was on date. Compression ranking measures how well values cluster; query ranking measures whether the clustering is on the column you filter by. They are different questions and only the second one is on the bill.

If your ranking did match, check whether both queries filter on the sort key — in which case the experiment has not separated the two effects and is worth rerunning with a third query.

8.12 The first move is to notice that 340 TiB is the wrong input.

On-demand is metered on bytes scanned; capacity is metered on slot-time. A query that scans 4 TiB in 20 seconds and one that scans 4 TiB in 40 minutes cost the same on demand and differ by two orders of magnitude on slots. You cannot convert between the two meters without a second measurement.

What I would look up:

1. The current per-slot-hour price for each edition, and which editions
   allow autoscaling.
2. The commitment terms: annual, monthly, or pay-as-you-go autoscale, and
   the discount at each.
3. The baseline slot floor -- the minimum reservation size.
4. Whether idle reserved slots can be shared across projects.

What I would measure, and this is the actual work:

-- the input the decision needs: slot-seconds, not bytes
SELECT DATE(creation_time)                             AS d,
       EXTRACT(HOUR FROM creation_time)                AS hr,
       SUM(total_slot_ms) / 1000 / 3600                AS slot_hours,
       SUM(total_bytes_billed) / POW(1024, 4)          AS tib_billed
  FROM `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
 WHERE creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 90 DAY)
 GROUP BY 1, 2 ORDER BY 1, 2;

The break-even:

on-demand   = 340 TiB x $6.25                  = $2,125 / month

capacity    = (baseline_slots x 730 h x price)
            + (autoscale_slot_hours x price)

capacity wins when the second expression is below $2,125.

And the shape of this workload is the interesting part. 70% of the scan volume in a four-hour nightly window means a flat reservation sized for the peak is idle for 20 hours a day — you would be buying the peak and paying for it 24/7, which is exactly Chapter 3 §3.5's mistake in a new costume. The plausible configurations are therefore a small baseline plus autoscale, or on-demand for the nightly burst and a small reservation for the steady BI load — and BigQuery permits that split by project or by reservation assignment.

One more thing I would check before any of it: whether the 340 TiB is necessary. At $2,125 a month the entire question is worth about $25,000 a year, and §33.6's experience is that a scan-volume review typically finds a double-digit percentage of it is a dashboard refreshing more often than anyone reads it. Reducing the scan makes the pricing question smaller and easier at the same time.

8.14 The incident.

A backfill of fct_order_item for 2024 is run on 2026-03-02 using COPY INTO against files in s3://kestrel-bronze/orders/. It loads correctly. On 2026-03-09 somebody notices the backfill missed a week, and re-runs COPY INTO for the whole range — relying, as everyone does, on Snowflake's load metadata to skip files already loaded.

It skips almost everything and loads a handful of files. Correct, and it looks like the fix worked.

Then in June, a different engineer re-runs the same backfill — same command, same files, part of an unrelated recovery. This time the load metadata for the original March files has expired at 64 days, so COPY INTO no longer knows they were loaded. It loads all of them again.

fct_order_item now contains 2024 twice.

When it is noticed: not immediately, because 2024 is a closed year that nothing dashboards. It is noticed in the following January, when someone runs a year-over-year comparison and 2024 revenue is double. Roughly seven months.

How they diagnose it: SELECT order_item_id, count(*) FROM fct_order_item GROUP BY 1 HAVING count(*) > 1 returns rows, which points at duplication rather than at a transform bug; then COPY_HISTORY shows the same filenames loaded on two dates. The COPY_HISTORY view is the artifact that makes this diagnosable at all, and it retains 14 days by default, so seven months later even that is gone unless someone was archiving it.

The control that prevents it: do not rely on load metadata for idempotency. Use FORCE = TRUE with an explicit, idempotent target:

BEGIN;
  DELETE FROM fct_order_item WHERE date_key BETWEEN :lo AND :hi;
  COPY INTO fct_order_item FROM @stage/orders/ FILES = (...) FORCE = TRUE;
COMMIT;

The principle, which is Chapter 4's: an idempotency guarantee with a time bound you did not choose is not a guarantee, it is a race with a 64-day fuse. Replace it with one you control — a delete-insert on a key you own, inside a transaction. And add the assertion that would have caught it anyway: a uniqueness test on order_item_id in the build (Chapter 23), which turns seven months into one failed run.

8.17 The mechanism. A grain-changing join multiplies rows before the aggregation. A hash join builds a hash table on one side and streams the other through it; when the join fans out, the streamed side produces many more output rows than input rows, and every one of them carries the full width of both sides. Those rows feed a GROUP BY, which needs its own hash table — and that hash table is now sized for the fanned-out cardinality, not the real one.

Memory is allocated per worker from the warehouse's budget. When the aggregation's state exceeds it, the engine spills: first to local SSD (bad), then to remote object storage (much worse — network round-trips per partition of the spill).

So remote spill on a query that should not be memory-hungry is a cardinality signal, not a memory signal. The engine is telling you it is handling far more rows than you think exist. Adding memory makes it complete; it does not make the answer right, and that is why this is a correctness bug wearing a performance costume.

The two diagnostics, in order:

1. Count the rows the join produces, before aggregating.

SELECT count(*) AS after_join,
       (SELECT count(*) FROM silver.order_items) AS expected
  FROM silver.order_items oi
  JOIN dim_promotion p ON p.order_item_id = oi.order_item_id;

Run this first, always. It is one query, it costs almost nothing, and it either confirms or eliminates the hypothesis outright. If after_join > expected, stop — you have found it, and no amount of profile-reading will tell you more.

2. Read the query profile's join node for the output/input row ratio. Snowflake's profile shows rows in and rows out per operator. Find the operator where the row count first exceeds the source table's, and the join immediately below it is the fan-out. This is the diagnostic for the case where step 1 is inconclusive because the fan-out is buried inside a chain of CTEs.

And the fix is never "size up the warehouse," even though that will make the query finish. It is the bridge table and the allocation weights of §6.12 — or, more often, deleting a join that should never have been there.

8.19 The exercise: a pre-flight estimate, checked against a measurement.

Before running ANY query against your local DuckDB Kestrel database:

1. Write down, on paper:
     a. which columns the query touches
     b. the row count of the largest table it reads
     c. estimated bytes = rows x columns x average width, adjusted for the
        partition/date filter
     d. that figure converted to a dollar cost on BOTH meters:
          per-TiB:      bytes / 2^40 x $6.25
          per-node-h:   estimated runtime x nodes x $2.400

2. Run it with:
     EXPLAIN ANALYZE SELECT ...
   and read the scan node's actual row and byte counts.

3. Record: predicted bytes, actual bytes, ratio.

The checkable number is the ratio in step 3, and the exercise is to get it inside 2× on three consecutive queries. Most people are wrong by more than 10× on the first attempt, and the correction is almost always the same one: they estimated the rows the query returns rather than the rows it reads.

Why this closes the gap DuckDB leaves. DuckDB has no bill, so nothing punishes a bad estimate — the query just runs. The habit being built is not arithmetic, it is the pause, and the pause is what transfers to a system where the estimate costs money. Chapter 33 §33.6 is the same exercise against a real meter.

Two refinements worth adding. Do it once for a query you have already shipped — that one is uncomfortable and more useful. And keep the log: predicted | actual | ratio in a file, because the calibration curve over ten queries is the actual output of the exercise.

8.21 The load, and the test that is the point:

-- platform/warehouse/load_gold.sql -- idempotent for ONE date_key
BEGIN;
  DELETE FROM gold.fct_order_item WHERE date_key = :date_key;
  INSERT INTO gold.fct_order_item
  SELECT ... FROM silver.order_items WHERE order_date = :date_key;
COMMIT;

# the nine lines that would have prevented Chapter 1's 31-day incident
run(LOAD_SQL, date_key="2026-11-14")
a = q("SELECT count(*) c, sum(net_revenue_cents) s FROM gold.fct_order_item"
      " WHERE date_key = '2026-11-14'")
run(LOAD_SQL, date_key="2026-11-14")            # the same load, again
b = q("SELECT count(*) c, sum(net_revenue_cents) s FROM gold.fct_order_item"
      " WHERE date_key = '2026-11-14'")
assert a == b, "load is not idempotent: %r != %r" % (a, b)

Three details are load-bearing.

The DELETE and INSERT are in one transaction. Between them the partition is empty; without the transaction a reader sees a day with no revenue, and a crash between them leaves it that way.

The DELETE is scoped by the same predicate the INSERT populates. A delete narrower than the insert leaves duplicates; a delete wider than the insert deletes data the run will not replace. They must be the same expression, and the safest form is one variable used twice.

And the assertion compares a sum, not only a count. A row count catches duplication; it does not catch a row being replaced by a different one. Two measures, because a load fails in two directions — the same principle as §18.7's dedup test.

Chapter 9 — Data Lakes and Object Storage

9.1

9.3 The five steps:

1. Read the FOOTER            schema + every row group's statistics
2. Select ROW GROUPS whose min/max can satisfy the predicate
3. Select the COLUMN CHUNKS the query names
4. Read and decompress the surviving pages
5. Decode (dictionary / RLE / delta / bit-packing) into memory

Step 2 is predicate pushdown. Step 3 is projection pushdown.

And they behave differently, which is the part worth remembering. Projection pushdown always works — the column list is known statically. Predicate pushdown works only if the predicate column correlates with physical order; on a randomly-ordered column, every row group's min/max spans the whole range, nothing is skipped, and the pushdown is real but useless.

9.5 Four reasons for separate buckets per layer:

1. Access control — bronze holds raw personal data; gold is broadly readable. One bucket means one policy surface and a lot of prefix-level exceptions.
2. Lifecycle rules — different retention per layer, expressed once at the bucket rather than as overlapping prefix rules that are hard to reason about.
3. Cost attribution — a per-bucket bill is a per-layer bill, for free.
4. Blast radius — a misconfigured lifecycle rule or a wrong aws s3 rm --recursive is bounded by one layer.

Access control decides it. The other three are conveniences with workarounds; the security boundary is the one you cannot retrofit, because separating buckets later means rewriting every path in every job, and by then there are hundreds.

9.7 Bronze is partitioned by ingest date because that is the only date bronze can know is correct. Bronze accepts what arrived, including late-arriving and out-of-order records; partitioning it by event date would mean deciding where a record belongs, which is a transformation, and bronze does not transform. Ingest date also makes a re-run trivially replaceable — one prefix, one write, one _SUCCESS.

Silver is partitioned by event date because that is what consumers filter on. Every query is "revenue for November," not "records that arrived in November." Silver is where the record is placed in business time, and doing so is one of the things that makes silver silver.

The consequence people miss: the two are not aligned, and a single day of silver is built from several days of bronze. That is why a silver rebuild needs a lookback window (Chapter 20 §20.7) and why "reprocess yesterday" is not one prefix.

9.10

order_items: 6,480,000 rows/year x 90 bytes = 583.2 MB/year

Grain      Partitions/year   Rows each     Bytes each
Yearly              1        6,480,000     583.2 MB
Monthly            12          540,000      48.6 MB
Daily             365           17,753       1.60 MB
Hourly          8,760              740      66.6 KB

(17,753 rows a day is the frozen order-lines-per-day figure, which is a useful check that the arithmetic is right.)

I would choose monthly, and accept 48.6 MB partitions — below the 100–250 MB ideal, but twelve prefixes are nothing to manage and every query filters by a date range that monthly prunes well. Daily would create 365 files of 1.6 MB, which is the small-files problem, self-inflicted, on a table small enough that it never had one.

What makes this different from the clickstream: the clickstream is 341 GB/year and order_items is 583 MB — a factor of 585. The clickstream's daily partitions are 934 MB because the dataset is large, not because "daily" is the right grain for time-series data.

The general rule the comparison establishes: partition grain is a function of bytes per partition, not of the calendar. Copying "we partition daily" from one table to another is how a small table acquires a large table's problems. And below a few hundred megabytes the honest answer is often not to partition at all — sort by placed_at and let row-group statistics do the pruning (§9.5's rule: partition the low-cardinality column, sort the high-cardinality one).

9.12

(a)

commits/hour/partition = 60 / 2       = 30
files/day  = 30 x 24 x 12 partitions  =    8,640
files/year = 8,640 x 365              = 3,153,600

(b)

341 GB/year  =  341,000 MB / 3,153,600 files  =  0.108 MB  =  ~108 KB/file

(c) Assuming 2 GETs per file (footer, then data — most readers issue at least two, and often three):

GETs   = 3,153,600 x 2               = 6,307,200
        6,307,200 / 1,000 x $0.0004  =    $2.52

LISTs  = 3,153,600 / 1,000 keys      =     3,154 requests
        3,154 / 1,000 x $0.005       =     $0.02
                                        ───────
full-year scan, request cost              ~$2.54

(d)

6,307,200 GETs x 30 ms   = 189,216 s of request time
                  / 100  =   1,892 s  =  31.5 MINUTES

...before a single byte is decoded. Compacted to 128 MB files, the same year is 2,664 files, 5,328 GETs, and 1.6 seconds of request latency.

(e) Yes, decisively — and the reason is (d), not (c). §9.6's threshold flags any dataset averaging under 32 MB per file; 108 KB is roughly 300× below it.

The reasoning worth writing down is which argument you use. The cost argument is weak: $2.54 a scan is not a number that gets work prioritised, and a manager is right to ignore it. The latency argument is decisive: 31.5 minutes of pure request overhead on every full scan, which shows up as a query-timeout complaint and gets diagnosed as "the warehouse is slow." And the third argument is the one that eventually forces it: 3.15 million objects is 3.15 million objects to rewrite when a customer asks to be deleted (Chapter 31).

9.14

# ADR-004 — Tokenising email at landing

Status: accepted · Date: 2026-02-11 · Deciders: data platform, security,
                                                 with legal sign-off

## Context
Section 9.4 says bronze keeps the payload as the producer sent it. Our
policy says raw email addresses must not be stored in the analytics
estate at all. These two rules conflict, and one has to give.

## Decision
Land the payload with `email` replaced by a deterministic token at the
ingestion boundary, before the object is written. Everything else in
the payload is untouched.

  token = HMAC-SHA256(email_lowercased_trimmed, key) -> hex, 64 chars

The key lives in the secrets manager, is accessible to the ingestion
service only, and is never written to the lake.

## What we land
- `_payload_raw`: the original JSON with `email` replaced by the token
- `_tokenized_fields`: ["email"] -- explicit, so the deviation is visible
- `_token_key_version`: so a key rotation is diagnosable

## Where the mapping lives
Nowhere in the lake. Re-identification requires the ingestion service's
key and a candidate email to test, which makes reversal a targeted
operation with an audit trail rather than a join.

We do NOT store a token->email table. A lookup table is a re-identification
key with a friendly name, and it would be the highest-value object we own.

## Alternatives
- Drop email entirely. Rejected: it is the only join key to the CRM feed.
- Encrypt reversibly. Rejected: reversible in the lake is not tokenised.
- Land raw and tokenise in silver. Rejected: bronze is the long-retention
  layer, so this stores raw email for two years to avoid an exception.

## What we give up -- and this is the honest section
1. **Bronze is no longer a faithful record.** The rule in section 9.4 has a
   documented exception, and every future "why is bronze different from
   the source" investigation has to know about it.
2. **We cannot re-derive the original.** If the tokenisation had a bug --
   a whitespace or case-folding difference -- the affected rows are
   unrecoverable. This is the real risk and it argues for a very simple
   normalisation, tested hard, changed never.
3. **A key rotation splits the identity space.** Records either side of a
   rotation do not join. `_token_key_version` makes it detectable; it does
   not make it painless.
4. **Deterministic tokens are linkable.** The same person is the same
   token everywhere, which is the point and is also a re-identification
   surface if combined with enough other columns (Ch. 31).

## What would reverse this
1. Legal permits raw email retention under a stated basis.
2. The CRM feed exposes a stable id, removing the need for email as a key.
   -- check quarterly; this is the likeliest exit.

Review: 2027-02-11.

9.17 Four ways a reader still sees an inconsistent partition despite _SUCCESS:

1. The reader does not check it. Most engines do not, by default. A spark.read.parquet(path) reads what it lists, and nothing consults the marker unless you wrote the code that does.

2. Overwrite of an existing partition. The marker from the previous successful write is already there. A re-run that deletes and rewrites the partition passes the check throughout, including during the window when half the files are gone. The marker says "some write finished once," not "this write finished."

3. Eventual visibility of the listing. Even with strongly consistent reads, a LIST is a paginated scan; an object written between page 1 and page 8 of a listing may or may not appear. The marker's presence does not make the listing atomic.

4. Partial failure with a successful driver. A writer whose tasks silently produced fewer files than intended — a swallowed exception, a task that wrote zero rows and reported success — still writes the marker. The marker records that the writer thought it was done.

And a fifth, which is really the general case: the marker is a claim about a directory whose contents nothing prevents anyone else from changing afterwards.

What a transaction log fixes: 1, 2, 3, and 4 — because the reader stops listing the directory altogether. It reads a version pointer, then a manifest naming exactly the files in that version, and reads only those. A half-written new version is invisible because its commit has not been appended yet; an overwrite is a new version, so readers on the old one keep seeing a complete, consistent snapshot.

What it does not fix: a writer that commits a manifest listing files containing wrong data. The log guarantees that readers see a consistent set of files, not that the files are correct. Case 4 survives in weakened form — a job that produced too few rows and committed successfully has committed an atomically-visible, internally consistent, wrong table. That is Chapter 23's problem, and no storage layer solves it.

9.19 Assumptions, stated first because the answer is entirely made of them:

one daily partition          934 MB, ~14,000,000 events
row group size               128 MB   ->  ~8 row groups per day
events per session           ~10 (14M events / ~1.4M sessions)
session_id                   high cardinality, effectively random as a string
the query                    WHERE session_id = 'abc...'  over one day

Sorted by session_id within the partition: one session's ~10 events are contiguous. They fall inside one row group, or across a boundary into two. Each row group's min/max is a narrow, disjoint slice of the id space, so 7 of the 8 row groups are skipped by statistics alone.

row groups read:  1  (occasionally 2)  of 8   ->  ~12.5% of the partition

Unsorted: the ~10 events are scattered uniformly. Two mechanisms both fail:

Statistics are useless. Every row group contains ids spanning nearly the whole space, so every min/max range contains the target. Nothing can be skipped — 8 of 8.

And even if the reader could skip by containment, the probability that a given row group holds none of the 10 events is $(7/8)^{10} = 0.26$, so the expected number of groups actually containing a match is about 5.9 of 8 — which is why "just add more row groups" does not rescue an unsorted layout.

row groups read:  8 of 8  ->  100% of the partition

sorted vs unsorted:  ~8x less data read, for the same answer

The claim in §9.5 is therefore quantified: partitioning gives you 1-in-365; sorting gives you 1-in-8 within what remains. Partitioning is the bigger lever, which is why it goes on the column you filter by most; sorting is the second lever and it is free, which is why it goes on the high-cardinality column that would have made a terrible partition key.

One honest caveat: the 8× assumes the query filters on the sort key. Sorted by session_id, a query filtering on event_type gets nothing from the sort — the same lesson as Exercise 8.10.

9.21 (Implementation.) The report's value is entirely in the flags, so the shape matters:

dataset                  objects   total    avg file   partitions  _SUCCESS
bronze/events            341,206   287 GB     0.86 MB     1,095     1,095/1,095
bronze/orders              4,380    12 GB     2.87 MB     1,095     1,094/1,095   <-- !
silver/events              3,285   341 GB   106.30 MB     1,095     1,095/1,095
gold/fct_order_item          438    18 GB    42.10 MB       365       365/365

FLAGS
  bronze/events    avg file 0.86 MB  -- under 32 MB. Compaction owed.
  bronze/orders    avg file 2.87 MB  -- under 32 MB. Compaction owed.
  bronze/orders    1 partition without _SUCCESS: ingest_date=2026-04-19

The missing _SUCCESS is the more urgent finding, and it is the one people skim past. A partition without a marker is a partition a writer did not finish, which means every query over that date range has been silently reading incomplete data since 19 April. The small-file flags are a cost and latency problem; the marker gap is a correctness problem, and a report that lists them in the same font invites the wrong triage. Print it first, in a separate section.

Chapter 10 — The Lakehouse

10.1

All five come from one mechanism — a transaction log — which is worth stating explicitly, because the lakehouse is usually sold as five features and bought as five features, and the person who understands it as one mechanism can predict which sixth feature it will and will not have.

10.3 Two kinds of action: add (a file becomes part of the table) and remove (a file stops being part of it). Everything else in a commit — metaData, protocol, commitInfo, txn — is bookkeeping around those two.

A compaction replacing 34,560 files with 4:

{"commitInfo": {"operation": "OPTIMIZE", "operationMetrics":
                {"numRemovedFiles": "34560", "numAddedFiles": "4"}}}
{"add":    {"path": "part-00000-....parquet", "size": 268435456,
            "dataChange": false}}
{"add":    {"path": "part-00001-....parquet", "dataChange": false}}
{"add":    {"path": "part-00002-....parquet", "dataChange": false}}
{"add":    {"path": "part-00003-....parquet", "dataChange": false}}
{"remove": {"path": "part-00000-old-....parquet", "dataChange": false,
            "deletionTimestamp": 1775001600000}}
... 34,559 more remove actions ...

34,564 actions in one commit, and it is atomic: readers see 34,560 files or 4, never a mixture.

Two details worth noticing. The removes carry a deletionTimestamp rather than deleting anything — the files still exist on object storage until VACUUM runs, which is Exercise 10.20(e). And a commit of 34,564 actions is a large JSON file, which is why checkpointing exists: without periodic Parquet checkpoints, a reader would replay every commit from version 0.

10.5 dataChange: false means "this commit rearranged bytes without altering the table's logical contents."

What breaks without it: every streaming reader. A structured-streaming consumer follows the log and treats each add as new data to process. A compaction that adds four files holding the same rows as the 34,560 it removed will, without the flag, be interpreted as 34,560 files' worth of new records — and the consumer reprocesses the entire table. Downstream, that is duplicate rows, a re-triggered alert on every row, or a re-sent notification for every order in history.

And it breaks Change Data Feed the same way, which is the quieter version: a CDF consumer sees an enormous batch of changes that did not happen.

The flag is therefore not an optimisation. It is the only thing distinguishing a maintenance operation from a data operation in a log that records both as add and remove — which is the price of having exactly two action types.

10.7

The pattern: safe changes are the ones where every existing byte remains readable under the new schema. That single test predicts every row above, including the two that people argue about (rename, and the direction of an integer widening).

10.10 The timeline:

14:02:10  writer A: build the commit for version 71 (an append of 3 files)
14:02:11  writer A: PUT _delta_log/00000000000000000071.json
14:02:41  writer A: the PUT times out. NO RESPONSE RECEIVED.
          -- but the object store may already hold the commit
14:02:42  writer A: retry. Reads the log: sees version 71 exists...
                    or does not, depending on listing timing.
14:02:43  writer A: writes version 72 with the SAME three add actions
14:02:44  the table now contains those three files TWICE

This is §4.1's third outcome — the timeout — arriving at the storage layer. The write may have succeeded; the client cannot know.

The "check before retrying" defence:

def commit_with_retry(log, version, actions, attempt_id):
    for attempt in range(MAX_RETRIES):
        try:
            log.put_if_absent(version, actions, txn_id=attempt_id)
            return version
        except Timeout:
            # do NOT blindly retry. Look first.
            existing = log.read_commit(version)          # may be absent
            if existing and existing.txn_id == attempt_id:
                return version                # OUR commit landed. Done.
            if existing:
                version += 1                  # someone else's. Rebase.
                continue
            # genuinely absent: safe to retry the same version
        except VersionAlreadyExists:
            version += 1
    raise CommitFailed(version)

The txn_id is what makes it work — it is an application-level idempotency key written into the commit, so "did my write land" becomes a readable fact rather than an inference. This is exactly §16.6's Idempotency-Key, at a different layer.

The race it does not close: between the Timeout and the read_commit, the object store may not yet reflect the write in a listing, even under strong read-after-write consistency for the object itself — a LIST-based log reader can miss a key that a GET of that exact key would return. Reading version 71 by its exact path closes most of it; reading the log by listing does not.

And a residual remains regardless: if the process dies between the timeout and the check, the next run has no attempt_id in memory and cannot distinguish its own orphaned commit from a peer's. The fix for that is to derive txn_id deterministically from the job's logical date and attempt number rather than randomly — the same lesson as §16.6's "generate the key before the first attempt."

10.12 The incident.

The upstream orders producer ships a feature that adds three fields to its payload, one of which is promo_code, a string. The change is backward-compatible by every reasonable standard and nobody is notified, because nobody has to be (Chapter 17 §17.4).

The silver orders table is written by a job configured with mergeSchema = true, inherited by copy-paste from the bronze job. Three columns appear in silver.orders overnight. No error, no alert, and the run is green.

What the table looks like afterwards: it has 34 columns instead of 31. Every row written before tonight has NULL in the three new ones. The silver contract — "silver has an enforced schema" — is now false, and nothing says so.

When it is noticed and by whom: several weeks later, by an analyst who finds promo_code while browsing the table, assumes it is a supported column, and builds a promotions dashboard on it. The dashboard shows a step change on the date the column appeared, because everything before is null. The analyst reports "a data quality issue with promotions."

The diagnosis takes a while because the column looks intentional. There is no ticket, no commit, no schema-change record on the silver side — the change happened in a configuration flag, not in code, so git log on the model shows nothing.

Why the same flag is right in bronze: bronze's job is to accept what arrived, and a rejected record is lost fidelity. Why it is wrong in silver: silver's job is to be the layer where the shape is guaranteed, and a flag that silently changes the shape removes exactly the guarantee silver exists to provide. mergeSchema in silver converts a loud failure into a silent one, which is the trade this book argues against everywhere.

10.14 The scenario where 7 days is not enough.

A rule change to net_revenue_cents is deployed on the 1st of the month. It is subtly wrong — it applies a discount twice for orders with both an order-level and a line-level promotion, affecting about 2% of rows. Month-end close runs on the 5th, and finance signs off; the figure is 0.3% low, well inside what anyone would notice.

On the 24th, a finance analyst reconciling the quarter finds the discrepancy. The question is "what did fct_order_line say on the 1st, before the change?" — and the answer is 23 days old.

With 7-day retention, the versions are gone. Not just uncopiable: VACUUM has deleted the underlying files, so the log entries are unresolvable. The table can be rebuilt from bronze (Chapter 34's $198.96), which is the right recovery — but rebuilding gives you what the current code produces, not what the table said on the 1st. Time travel answers a different question than a rebuild does, and this is the question only time travel answers.

How often it occurs: rarely, and predictably. The trigger is a reconciliation or audit cycle longer than the retention window — month-end close, quarter-end, an annual audit, a regulator's request. At Kestrel that is roughly two to four times a year, clustered at period boundaries.

Proposed retention: 30 days, with a monthly snapshot retained for 13 months.

The justification is the cycle, not a feeling. 30 days covers every month-end investigation, which is where the demand actually is. It does not cover quarter- and year-end, which is why the second half matters: a cheap, deliberate CLONE taken on the first of each month costs almost nothing (a zero-copy clone is metadata) and answers the annual questions without holding 400 days of tombstones.

What longer retention costs:

tombstoned files are BILLED. gold/fct_order_line churns ~4% of 18 GB daily.

  7 days   ~5.0 GB of tombstones   $0.12 / month
 30 days  ~21.6 GB                 $0.50 / month
400 days  ~288 GB                  $6.62 / month

So the storage cost is not the constraint — it is rounding error. The real costs are two others, and they are the ones to state: VACUUM and log replay get slower as the log grows and the file list lengthens, and — the one that matters — an erasure request is not complete until the versions containing that person are vacuumed (Chapter 31). Long time-travel retention is a deliberate delay on your own deletion obligation, and 400 days of it is not defensible. The monthly clone is better precisely because it is a small, enumerable set of objects that an erasure job can be pointed at.

10.17

# Policy: exceptions to the raw-payload rule in bronze

## What qualifies
An extraction from the raw payload into a real column qualifies ONLY if it
meets all four:

1. **It is structural, not semantic.** The field is used for partitioning,
   deduplication, ordering, or deletion -- not for analysis. Analysis
   belongs in silver.
2. **It is copied, not moved.** The payload keeps the field. Bronze remains
   reconstructible.  (The one exception: a field we are forbidden to store
   at all -- see ADR-004 -- and that requires legal sign-off, not this
   policy.)
3. **It is stable.** The field's meaning is covered by a contract (Ch. 17),
   or it has been present and unchanged for 90 days.
4. **Doing it in silver is materially worse**, with a number attached.

## Who approves
Two data platform engineers, one of whom did not write the request. Legal
sign-off additionally required whenever criterion 2's exception applies.

## What is recorded
An entry in `platform/docs/bronze-extractions.md`: the field, the four
criteria answered, the approvers, the date, and -- the field people skip --
**what breaks if the extraction is wrong**, because these are extracted at
ingestion and are therefore not fixable by a rebuild.

Applying it to the three requests:

session_id, for partitioning: REJECT. It fails criterion 4 and, more fundamentally, it is a bad partition key — 1.4 million sessions a day is §9.5's high-cardinality catastrophe. The request is structural in form and wrong on its merits, which is worth separating: the policy is not the reason to say no here, the partitioning arithmetic is.

event_type, for filtering: REJECT. Filtering is analysis. It fails criterion 1. The honest version of this request is "queries against bronze are slow," and the answer to that is silver, which exists precisely so that nobody has to query bronze.

country_code, for a data-residency requirement: APPROVE. It is structural — it determines where the object may be stored, which is a property of the write itself and cannot be deferred to silver without having already stored the data in the wrong region. It passes 1, 2 (the payload keeps it), and 4 (silver is materially worse: it is too late). Criterion 3 needs checking — if country_code is absent for 4% of events, the policy question becomes "where do the unknowns go," and that is a decision to make before approving, not after.

The pattern across the three: two structural-sounding requests were analysis in disguise, and the one that qualified did so because it changes what the write itself does. That is a sharper test than the four criteria and worth adding to the policy as a one-line summary.

10.19 Three approaches:

1. Write both, then flip one pointer. Build fct_order_item__v72 and dim_customer__v72 as new tables or new partitions, verify them together, then atomically update a view (or a single metadata row) that consumers read through. Consumers see the old pair or the new pair.

Gives up: two copies of the data during the build, and every consumer must go through the indirection — a single direct query against the physical table breaks the guarantee, and someone will write one.

2. Version the data, not the table. Both tables carry a build_id; consumers filter on the current build_id, published in a tiny control table. A build writes rows under a new id and then publishes.

Gives up: a WHERE build_id = (SELECT ...) on every query, which is easy to forget and which partition pruning may or may not exploit. And storage grows with retained builds. It is the same mechanism as 1, moved from the table name into a column, and it is more robust because a query that forgets the filter returns too much — visibly wrong rather than subtly stale.

3. Order the writes and tolerate a bounded window. Publish dim_customer first, then fct_order_item, and make the dimension additive-only within a build — new surrogate keys are added, none are removed. A fact row can then never point at a dimension row that does not exist yet; the worst case is a dimension row nobody references, briefly.

Gives up: the symmetric guarantee. This works only because the dependency is one-directional, and it requires the dimension to be genuinely additive (no Type 1 in-place update, no deletes), which is a real constraint on the model. In exchange it needs no indirection, no second copy, and no query changes — and it is what Kestrel does, because the 6am window is 15 minutes and approaches 1 and 2 both spend it copying.

The general lesson: in the absence of multi-table transactions, you either make the switch atomic at a single point (1, 2) or make the inconsistency harmless (3). There is no third category, and 3 is usually cheaper when the dependency graph allows it.

10.21 (Implementation.) The flags and the --dry-run are the exercise; the report shape:

table                 ver   files   avg MB   since ckpt   tombstoned   flags
bronze.events        1,204  41,206     6.9        204       12.4 GB     SMALL
silver.orders          388   1,205    98.2         88        1.1 GB     --
gold.fct_order_line    902     438    42.1        102       18.9 GB     TOMB

--dry-run is not a convenience, it is the point. OPTIMIZE on bronze.events rewrites 287 GB and VACUUM deletes files permanently — the two most expensive and least reversible operations in the platform, run unattended, on a schedule. Printing the exact commands for a human to approve is the difference between a maintenance job and an incident generator.

And there is a specific trap the dry run catches: VACUUM with a retention shorter than the longest-running reader deletes files out from under a live query. A dry run that prints the retention alongside the command lets someone notice that the nightly Spark job takes six hours and the retention is set to one.

Chapter 11 — File Formats and Serialization

11.1 Who reads it, and how (a human, occasionally · a machine, whole records at a time · a machine, a few columns of many rows) — and where the schema lives (nowhere · in each record · in the file header or footer · in a registry).

The second determines when you find out something is wrong. A CSV tells you months later, in a consumer. A registry tells the producer at deploy time, in CI, before the bad data exists (Chapter 17 §17.3). Everything between those two poles is a point on the same axis: the further from the producer the schema lives, the later and more expensively the mismatch surfaces.

11.3 Six things CSV gets wrong:

1. No types. Everything is a string; every consumer casts, and they cast differently.
2. No schema. Column order is the contract, and an insertion in the middle breaks every reader silently.
3. Ambiguous nulls. Empty string, NULL, \N, NA, - — and no way to distinguish "empty" from "absent."
4. No standard. RFC 4180 exists and is widely ignored; quoting, escaping, and line endings vary by producer.
5. No nesting. Anything hierarchical gets flattened by convention or JSON-in-a-column.
6. Encoding is out of band. UTF-8, Latin-1, or UTF-16 with a BOM, and the file does not say.

The one most likely to bite: number 3, followed closely by 2. A null-versus-empty ambiguity in a postal_code or a discount column produces a plausible number rather than an error — Chapter 6's distinction between zero as a measurement and null as an absence, arriving as a parsing decision that nobody made deliberately.

11.5 Avro dominates streaming because it is row-oriented — a message is one whole record, which is what a stream is — and because its schema resolution rules make writer/reader version skew a first-class, well-defined operation, which is what a long-lived topic needs.

It loses at analytics because reading two columns from a row format means reading and decoding every byte of every record, so the projection pushdown that gives columnar its 100× (§11.7) is structurally unavailable.

11.7 CSV + gzip beat Parquet + snappy — 19.93 MB against 20.35 MB.

What it tells you: columnar formats are not primarily a compression technology. A general-purpose compressor applied to text captures most of the same redundancy, so on the size axis the two are close and the ranking can go either way depending on the data. Columnar's advantage is what you can avoid reading, and that does not appear in a size comparison at all — which is exactly why size comparisons are the ones people publish and the ones that mislead.

11.10 What to change: the order the generator emits rows in.

§11.6's benchmark generated events in timestamp order, which is how event data actually arrives. In that order event_ts delta-encodes almost perfectly, and sorting by session_id destroyed that gain without repaying it — hence the 3.4% penalty.

Change the generator to emit rows in random order, then sort. Now the baseline has no exploitable order in any column, and sorting can only help:

generator emits in TIMESTAMP order (the original)
  unsorted                12.56 MB
  sorted by session_id    12.99 MB      +3.4%   sorting HURT

generator emits in RANDOM order
  unsorted                14.81 MB
  sorted by session_id    12.94 MB      -12.6%  sorting HELPED
  sorted by event_ts      12.60 MB      -14.9%  sorting helped MORE

(Run it; the exact figures will differ with cardinality. The signs will not.)

The general rule the result supports: sorting improves compression only to the extent that it creates clustering you did not already have. Data that arrives in a useful order already has it, and sorting on a different key spends that order to buy clustering elsewhere. The question is never "should I sort" — it is "what order is the data already in, and is the new key worth more than the old one."

And the second-order lesson, which is the one worth taking to another codebase: the original measurement was not wrong and neither was Chapter 8 §8.3. The claim "sorting improves compression" was stated more generally than it holds, and one experiment with a changed initial condition separates the general rule from the special case.

11.12

The envelope takes the useful half of each. The Parquet envelope gives partition pruning, cheap column access to the lifted fields, and a file the erasure job can rewrite selectively. The JSON payload inside it gives fidelity: whatever the producer sent is still there, byte for byte, so a field nobody anticipated is available two years later.

What the envelope costs, honestly: the payload column is a large opaque string, so it compresses less well than parsed columns would, and querying inside it needs JSON functions and is slow. That is the correct trade for bronze and the wrong one for silver, which is why silver parses.

11.14 Measured — 300,000 synthetic Kestrel clickstream events, 8 columns, pyarrow 24.0:

codec       size MB   vs zstd-1   write s
zstd 1         8.31      1.000       0.14
zstd 3         8.56      1.031       0.15
zstd 9         8.60      1.035       0.34
zstd 19        7.90      0.951       1.81
snappy        13.62      1.639       0.14

Two results, and the first one is surprising enough to be worth checking twice.

Levels 3 and 9 are larger than level 1 — by 3.1% and 3.5%. That is not noise and it is not a bug. Parquet's own encodings (dictionary, RLE, delta, bit-packing) run before the codec, so what zstd receives is already-compacted, page-sized blocks. Higher levels spend effort searching for long-range matches that do not exist inside a 1 MB page, and their different block framing can cost more than the extra matching saves.

Level 19 buys 4.9% for 13× the write time.

The knee is at level 1, and there is effectively no curve to find — which is a stronger version of §11.8's warning than §11.8 makes. I would set zstd level 1, or leave the default and stop thinking about it.

The comparison that actually matters is the last row: snappy is 1.64× larger than zstd-1 at the same write time. Choosing zstd over snappy is worth more than every level above 1 combined, and it costs nothing. That is the whole tuning decision.

Caveat, and it is the point of running it yourself: this is one dataset with a particular column mix — two high-cardinality hex strings, several low-cardinality categoricals. A table of long free text would behave differently. Run it on your data; the method transfers and the numbers do not.

11.17 (Answers vary — a worked example.)

The claim I suspect is stated too generally: "compaction is worth it below 128 MB average file size" (§9.6, and Exercise 9.21's 32 MB flag).

Why I doubt it in general. The claim bundles three different costs — request charges, request latency, and metadata overhead — that scale differently and that dominate at different points. Exercise 9.12 shows the cost argument is negligible ($2.54) while the latency argument is decisive (31.5 minutes), which already means the single threshold is standing in for two different curves. A dataset read once a quarter by one job has a completely different break-even from one read hourly by twenty-two dashboards, and 128 MB does not know which it is looking at.

The measurement that would test it:

Fix the bytes (say 20 GB). Vary the file size across 1, 8, 32, 128, 512 MB.
For each, measure:
   a. LIST + GET request count and cost
   b. wall-clock for a full scan at 1x, 10x, and 100x parallelism
   c. wall-clock for a single-partition point query
   d. time to rewrite the dataset (the compaction's own cost)
Then plot (b) against file size for each parallelism, and find where the
curve flattens. Compare that knee to 128 MB.

The prediction worth making before running it: the knee moves left as parallelism rises, because more concurrent requests hide per-request latency. If that is right, the threshold is not a property of the data at all — it is a property of the reader, and stating it as a storage rule is the over-generalisation.

11.19 Zero-copy here means the consumer reads the producer's bytes in place, without deserialising them into its own representation. Arrow specifies a memory layout — columnar, with a defined arrangement of validity bitmaps and value buffers — rather than a wire encoding. Two processes that both speak Arrow can hand a pointer (or a shared-memory region, or a stream of buffers) from one to the other, and the receiver's arrays are the sender's buffers. No parse step, no allocation, no copy.

Why it matters: in a typical pipeline, serialisation is a large and completely invisible fraction of runtime. Converting a result set to Python objects, or a pandas DataFrame to something a database driver accepts, can cost more than the query. It is invisible because no tool attributes it to anything — it appears as the query being slow.

Two places in this book's pipeline where an Arrow path removes a round trip:

1. Warehouse or database → DataFrame, in the extractors (Chapters 7 and 13). A DBAPI cursor produces Python tuples, which pandas then converts — two full materialisations of every value. ADBC (or DuckDB's and Snowflake's fetch_arrow_table) returns Arrow buffers directly, and the DataFrame is a view over them. On a multi-million-row extract this is routinely a 2–10× difference in the extract's runtime, and it changes no logic.

2. Between DuckDB, Polars, and pandas inside a transformation (Chapter 21). All three are Arrow-backed. duckdb.sql(...).arrow() handed to Polars is a zero-copy handoff; the same data via .df() and pl.from_pandas() is two conversions. The Chapter 21 pattern of "query in DuckDB, reshape in Polars" is only cheap if the handoff is Arrow, and it is a one-line difference that most code gets wrong by default.

A third, worth mentioning: Parquet is not Arrow, but its column chunks decode into Arrow arrays, so a Parquet → Arrow → engine path skips a representation that a Parquet → pandas path materialises.

11.21

# Format policy

## Rules

**Analytical data is Parquet + zstd.** No exceptions inside the platform.
**Bronze landing is a JSON payload inside a Parquet envelope** (section 11.9).
The payload is not parsed at landing and not modified. See ADR-004 for the
one field this is not true of.
**Wire format on Kafka is Avro with a registry** (Ch. 17), never JSON.
**CSV is an interchange format only.** It may leave the platform and it may
enter it; it may not be a storage format inside it.

## Forbidden
- CSV as an internal storage format. No types, no schema, ambiguous nulls.
- A row format for anything queried by column subset.
- `mergeSchema = true` anywhere except bronze (section 10.12).
- Tuning the compression LEVEL. Set the codec, leave the level at 1.
  Exercise 11.14 measured 4.9% for 13x the write time, and levels 3 and 9
  were LARGER than level 1.

## What would make us revisit
- **Parquet + zstd:** an engine we adopt that reads ORC materially better.
  Unlikely; check at adoption, not on a schedule.
- **The bronze envelope:** the clickstream gaining a stable, contracted
  schema (Ch. 17). At that point bronze can be typed and the envelope is
  overhead. This is the likeliest exit and it is a Chapter 17 outcome.
- **Avro on the wire:** if the registry is removed, Avro loses most of its
  advantage and Protobuf becomes competitive.
- **CSV for the supplier-b export:** if they can accept Parquet. Ask
  annually; the quoting bugs cost us about a day a year.

Chapter 12 — NoSQL and Specialized Stores

12.1 The one honest reason: an access pattern that the existing store serves badly enough to matter, measured.

The three bad ones: "Postgres can't do that" (frequently it can — JSONB, full-text, TimescaleDB, pgvector, SKIP LOCKED); "it'll scale better" (unmeasured, and Chapter 5's case study is nine months to make a four-second query two seconds faster at 15.5 GB); and "the team knows it" (a real consideration and a bad primary reason — name the second operator first).

12.3 Two reasons a key-value store should never be the system of record for anything analytics needs:

It has no usable change feed. Redis keyspace notifications are fire-and-forget, unordered, and lossy — a disconnected subscriber misses events with no way to detect or recover them. You cannot build a correct extract on top of a lossy notification stream, and the alternative is a full keyspace scan, which on a production cache is both expensive and disruptive.

It has no scan-friendly access. The data model is designed for point lookups by key; there is no secondary index, no range scan on a value, and no ordering. Extracting "everything that changed yesterday" requires reading everything and comparing — and the store has no notion of yesterday.

And a third that is really the first two combined: a key-value store's data is usually derived and TTL'd, so it does not just fail to tell you what changed — it silently forgets. An extract that misses a window loses data that no longer exists anywhere.

12.5 Cardinality, decisively.

base series = 4 methods x 6 statuses x 200 endpoints  =        4,800

add customer_id as a label (1,900,000 active customers):
              4,800 x 1,900,000                       = 9,120,000,000

Nine billion series. A time-series database holds an in-memory index entry per series — on the order of a kilobyte with labels, postings, and chunk references — so the index alone is in the terabytes before a single sample is stored.

And the volume comparison makes the point sharper. Scraping 4,800 series every 15 seconds is about 27.6 million samples a day, which any TSDB handles on a small instance. The samples are not the problem. One label turned a trivially small workload into an impossible one, and the sample rate did not change at all.

The rule: labels are dimensions, and a dimension with unbounded cardinality is not a dimension. Customer-level metrics belong in the warehouse, where a high-cardinality column is a column and costs nothing. customer_id on a metric is a category error, and it is the single most common way people destroy a metrics stack.

12.7 Three reasons a vector store is a data engineering problem:

1. The vectors are derived data with a pipeline behind them. Something reads a source, calls an embedding model, and writes the result. That is an ingestion pipeline with all of Part III's problems — incremental extraction, retries, idempotency, and reconciliation against the source.

2. The embedding model is a version, and it is not stored with the data. Re-embedding with a new model produces vectors that are not comparable with the old ones; mixing them silently degrades results with no error anywhere. The model version is a schema, and nobody treats it as one.

3. Coverage is a data quality property with no natural alarm. A product missing from the index does not error — it is simply never returned. The failure mode is absence, which is the hardest kind to notice and the kind Chapter 23 exists for.

And the deletion obligation follows the vectors (Chapter 31): an embedding of a customer's review is derived personal data, it does not look like personal data, and it is therefore missing from every hand-written deletion manifest.

12.10 The quarantine design:

WHERE      bronze/quarantine/<dataset>/reject_date=YYYY-MM-DD/
WHAT       the document, unmodified, plus:
             _quarantined_at      when
             _rejected_by         which assertion, by name
             _reject_detail       the message, truncated to 500 chars
             _source_batch_id     so it can be replayed with its cohort
             _source_offset       exact provenance
             _schema_version      what we were expecting
WHO        the dataset's owning engineer. Named, in the dataset's contract.
CADENCE    a weekly count in the team channel, automatically.
           A human reads the actual documents when the count moves.

Two design choices carry the weight.

The document is stored unmodified. You will want to reprocess after fixing the reader, and a re-serialised document is not the document that failed (§15.10's first property).

And the assertion's name travels with it, not just an error string. That is what makes the quarantine groupable: SELECT _rejected_by, count(*) GROUP BY 1 turns a pile of documents into a ranked list of causes, and one cause is almost always 90% of the pile.

What happens when the quarantine grows and nobody looks — and this is the real question. It becomes a silent data loss channel that everyone believes is a safety net. Rows are missing from silver; the pipeline is green; the count in the channel becomes wallpaper. Six months later someone reconciles and finds 340,000 orders that were quarantined on day one by an assertion that was wrong.

Three controls, and only the third one works:

Alert on the rate, not the count. A rising rejection rate is an incident; a stable one is a policy decision someone has made by not making it.

Cap the quarantine. If rejections exceed some share of the batch — 1% is a reasonable start — fail the whole batch rather than quarantining it. A quarantine is for exceptions; at 1% it is not an exception, it is a schema change.

And give it a retention. 30 days, after which quarantined documents are deleted. That is the control that actually forces the drain, because it converts "nobody looked" into "we lost data on a known date," which is a consequence somebody will act on. A quarantine with unlimited retention is a landfill.

12.12

1. The index-versioning procedure.

Build into a NEW index, named with the model version and a build id:
    products_v3__e5-large-v2__20260311

  1. build to completion, offline
  2. run the coverage assertion (below) against it -- it must pass
  3. run a quality spot-check: 20 known queries, compare top-5 against
     the current index; a human looks at the diff
  4. flip the ALIAS `products_current` to the new index
  5. keep the previous index for 7 days, then delete

NEVER write two model versions into one index. NEVER re-embed in place.

The alias is the whole mechanism: the flip is atomic, the rollback is another flip, and no consumer knows an index name.

2. The schema change.

ALTER TABLE product_embeddings
  ADD COLUMN embedding_model      TEXT NOT NULL,   -- 'e5-large-v2'
  ADD COLUMN embedding_dim        INT  NOT NULL,
  ADD COLUMN embedded_at          TIMESTAMPTZ NOT NULL,
  ADD COLUMN source_content_hash  TEXT NOT NULL;   -- what was embedded

CREATE INDEX ON product_embeddings (embedding_model);

source_content_hash is the underrated column. It makes "which products have changed since they were embedded" a join instead of a guess, which is what turns re-embedding from a full rebuild into an incremental job.

3. The coverage assertion.

-- every active product must have a current-model embedding. Zero rows.
SELECT p.product_id, p.name
  FROM gold.dim_product p
  LEFT JOIN product_embeddings e
         ON e.product_id = p.product_id
        AND e.embedding_model = :current_model
 WHERE p.is_active
   AND (e.product_id IS NULL
        OR e.source_content_hash <> md5(p.name || p.description));

Which I would build first: the assertion. It is nine lines, it needs no coordination, it catches the incident that already happened, and it works against the index as it exists today — no migration, no rebuild, no meeting. The versioning procedure and the schema change are better designs and both require a build cycle. The assertion converts a five-week silent failure into a failed run tomorrow morning, and everything else can follow at its own pace.

That ordering generalises: when a class of failure is invisible, make it visible before you make it impossible. A detection you can ship this afternoon beats a prevention you can ship next quarter, and the detection tells you how bad the problem actually is — which is information you need in order to size the prevention.

12.14 (Verify against current documentation; these are the thresholds the material and the communities converge on, and you should confirm each.)

JSONB. PostgreSQL's own documentation notes that jsonb values over roughly 2 KB are TOASTed — stored out of line and compressed — so every access to a large document decompresses it. The practical threshold is therefore document size, not row count: below a couple of kilobytes with GIN indexes on the keys you query, JSONB is genuinely competitive with a document store. Above it, and particularly for whole-document reads at high rates, the TOAST round-trip dominates. jsonb_path_ops GIN indexes are smaller and faster than the default for containment queries and cannot serve key-exists queries — a real trade worth reading the index documentation for.

tsvector. Sufficient for search where you need matching and basic ranking. It stops at three things, and each is a hard stop rather than a gradient: faceting (there is no aggregation-over- matching-documents primitive), typo tolerance (pg_trgm gives similarity, not the edit-distance analysers a search engine has), and relevance tuning (ts_rank is fixed; there is no BM25 tuning, no per-field boosting). The size at which people report leaving is commonly in the millions to tens of millions of documents, and the reason given is almost always one of those three features rather than throughput.

pgvector. The project's own README and benchmarks are the place to look. HNSW indexes were added in 0.5.0 and changed the picture substantially over the earlier IVFFlat. The widely-cited practical range is single-digit to low-tens of millions of vectors on a well-provisioned instance, with the binding constraints being index build time and memory rather than query latency. The dimension-limit detail matters and is easy to miss: indexed vectors are capped at 2,000 dimensions, which excludes some large embedding models outright.

What the three have in common, and it is the useful finding: none of the thresholds is about data volume. They are about a specific missing capability — TOAST for large documents, faceting for search, index memory for vectors. "When do we outgrow Postgres" is the wrong question; "which specific thing do we need that it does not do" is answerable, and usually the answer is "nothing yet."

12.17 The outbox for a Cassandra-backed service:

WRITTEN   in ONE Cassandra BATCH, restricted to a single partition key:
            1. the domain row      (orders, partition key = order_id)
            2. the outbox row      (order_outbox, SAME partition key)

          CREATE TABLE order_outbox (
            order_id   uuid,
            seq        timeuuid,
            payload    text,
            PRIMARY KEY (order_id, seq)
          );

READS IT  a relay process that scans for unpublished rows, publishes to
          Kafka, and deletes (or TTLs) the outbox row after the broker
          acknowledges.

The Cassandra-specific constraint is the one that shapes everything. A BATCH is atomic and isolated only within a single partition. A multi-partition batch is atomic-ish (a logged batch will eventually apply) but not isolated, and it is slow. So the outbox table must be partitioned by the same key as the domain table, which means the outbox is sharded across the ring — there is no single "outbox table" to scan.

That is the real cost, and it is why this pattern is uncomfortable on Cassandra. The relay cannot do SELECT * FROM outbox WHERE published = false — that is a full-cluster scan. The practical approaches are to range-scan token ranges continuously, or to add a second, time-bucketed index table written in the same batch (which reintroduces the multi-partition problem), or to use the CDC commit-log feature and read the outbox writes from the log itself.

When the relay falls behind: outbox rows accumulate, per partition, invisibly — there is no single place to see the backlog. The observable signal has to be built: the relay must emit its lag (now minus the oldest seq it has published) as a metric, because nothing else will show it. And if rows carry a TTL as a safety valve, falling behind the TTL is silent data loss, which argues strongly for deleting explicitly after acknowledgement rather than TTLing.

The guarantee you get: at-least-once publication of every committed domain change, in per-partition order. That is genuinely valuable and it is what the pattern is for.

The guarantee you do not get: global ordering. Events from different partitions can be published in any relative order, and a relay reading multiple token ranges concurrently makes that certain. Any consumer that needs "order A happened before order B" is not served by this, and the honest fix is to stop needing it — per-key ordering is what the outbox provides and what almost every consumer actually requires.

(Chapter 36 §36.4 covers the relational case, where a single transaction and a single scannable outbox table make all of this much simpler — which is itself an argument about store choice.)

12.19 A genuine counterexample: variable-length shortest path with a filter on the path.

"Find the shortest chain of shared-supplier relationships connecting SKU A to SKU B, using only suppliers we currently have an active contract with, and return the chain."

Why a recursive CTE handles it badly. A recursive CTE can express reachability, but three things go wrong at once:

It has no notion of "shortest." You enumerate all paths up to a depth and then take the minimum, which means the work is proportional to the number of paths rather than to the length of the answer. On a graph with any real branching factor that is exponential.

It has no visited set. Cycles must be prevented by carrying an array of visited nodes in the recursive term and checking membership on every row — which works, is O(depth) per row, and is the reason these queries are slow long before they are wrong.

And the depth is unbounded in the problem statement. WITH RECURSIVE needs a termination condition; the natural one here is "when you reach B," which the engine cannot use to prune — it will keep expanding the other branches.

Why a graph traversal handles it well. A bidirectional BFS expands from both endpoints and stops when the frontiers meet, visiting each node at most once, with a visited set maintained natively. The work is proportional to the size of the explored neighbourhood, not to the number of paths, and the filter on active contracts is applied as an edge predicate during traversal so it prunes rather than filters afterwards.

What makes the difference, stated generally: SQL's recursive CTE is set-at-a-time and memoryless — each iteration produces a set from the previous set, with no state carried outside the rows. Graph traversal is pointer-chasing with a visited set, which is the wrong shape for a relational engine and the right one for path problems.

And the honest boundary. "Which customers bought from suppliers who also supply our competitor" is two joins and belongs in SQL. The chapter's claim survives: the counterexample is not "graphs" but specifically unbounded-depth path finding where the path itself is the answer, and most questions people call graph problems are not that.

12.21

# ADR-005 — Vector search: pgvector, not a dedicated store

Status: accepted · Date: 2026-03-04 · Deciders: data platform, ML

## Context
"Customers who liked this" needs similarity search over product
embeddings. 47,000 SKUs, one 1,024-dimension vector each. Query volume is
one lookup per product page view -- roughly 40 per second at peak.

## Decision
pgvector, in `kestrel_app`, with an HNSW index.

## Alternatives
- A dedicated vector database (Pinecone / Qdrant / Weaviate). Better ANN
  tuning, horizontal scale, and hybrid search. Rejected: it is a ninth
  store for four engineers (Ch. 5 section 5.1), and 47,000 vectors is four
  orders of magnitude below where it earns that.
- Brute-force cosine in the application. 47,000 x 1,024 floats is 192 MB
  and would actually work. Rejected: it cannot filter by availability or
  category without pulling everything, and that filter is required.

## Consequences
+ No new system. Backed up, monitored, on-call, and access-reviewed with
  the database we already run.
+ The similarity query JOINS to price, stock, and category in one
  statement. A separate store makes that two round trips and a merge,
  which is the requirement most vector-store demos quietly skip.
- Index rebuilds compete with transactional load (Ch. 7 section 7.1 --
  we are a guest here too).
- No hybrid (lexical + vector) scoring. Elasticsearch has it; we do not.

## What would reverse this
1. Vector count exceeds 5,000,000.
     SELECT count(*) FROM product_embeddings;   -- monthly
2. p99 similarity-query latency exceeds 150 ms over a 7-day window.
     SELECT approx_quantile(duration_ms, 0.99) FROM query_log
      WHERE query_tag = 'similarity' AND ts > now() - interval '7 days';
3. A requirement for hybrid lexical+vector scoring is accepted into the
   roadmap. (A feature trigger, not a metric -- named because it is the
   likeliest actual cause, and it should not hide behind the two numbers
   above.)

Review: 2027-03-04.

Note what condition 3 does. Conditions 1 and 2 are the observable ones the exercise asks for, and they are the ones least likely to fire. The realistic reason this decision gets reversed is a product requirement, not a threshold — and an ADR whose reversal conditions are all metrics will be reversed for a reason it does not mention, which is how a decision record becomes fiction.

Part III — Ingestion

Chapter 13 — Batch Ingestion

13.1 The seven steps: acquire a lease · read the watermark · plan the chunks · extract each chunk · land it, idempotently · advance the watermark · reconcile.

The three people omit, and what each omission costs:

The lease. Without it, two runs overlap — a delayed run and its successor — and both extract the same range while both advance the watermark. The cost is a duplicated load, or a skipped one, and neither fails. It is one row in a table and it is the cheapest of the three to add.

The chunking. Without it, one long transaction on the source holds a snapshot for the duration (§7.3). The cost is bloat in somebody else's database, and you will hear about it as an incident attributed to you.

The reconciliation. Without it, the extract has no acceptance criterion — it can succeed while losing rows, and there is nothing in the system that disagrees. The cost is that every other step's failure becomes silent, which is why this is the expensive omission even though it is the one that sounds most optional.

13.3 Four things a full load gets right:

1. Deletes. A row absent from the source is absent from the target, automatically, with no soft-delete column and no negotiation.
2. Corrections to old rows. A backdated fix to a two-year-old record arrives, whether or not anybody remembered to touch updated_at.
3. It self-heals. Any previous bug, gap, or partial run is repaired by the next successful load; there is no drift to accumulate.
4. It has no state. No watermark to corrupt, no lease to leak, no position to lose — so it cannot fail in the ways Chapters 4 and 13 spend most of their pages on.

The fourth is the underrated one. An incremental load is a stateful system, and every property of this book's hard problems follows from state. A full load is a pure function of the source, which is why the chapter's advice is to keep doing full loads until the size genuinely forbids it.

13.5 The five: full compare (hash every row on both sides) · timestamp / watermark · version or sequence column · triggers · log-based CDC.

Full compare catches deletes without CDC. A row present in the target and absent from the source is a delete, and no other non-CDC strategy can see one.

What it costs: a full read of the source, every run — the entire table scanned and hashed, holding a snapshot while it does (§7.3). You have paid a full load's read cost to avoid a full load's write cost, which is a real saving when the target write is expensive and no saving at all when it is not. It is also the one strategy whose cost grows with the table rather than with the change rate.

13.7 The four lies: it is set at transaction start, not commit · it is not updated on every write (a trigger that misses a path, an UPDATE that bypasses the ORM) · it is in the wrong time zone, or in local time across a DST boundary · it can go backwards (a clock adjustment, a restored row, a manual fix).

The one that produces wrong values rather than missing rows is the time-zone lie. The other three cause rows to fall outside the filter and never be read — an absence. A timestamp in local time across a DST transition produces rows that are read and are dated wrong, so they land in the wrong day's partition and appear in the wrong day's revenue. Missing rows show up in a reconciliation; misdated rows reconcile perfectly at the total and are wrong at every grain below it.

13.9 Because a hard delete removes the row, and a timestamp strategy can only ask questions about rows that exist. WHERE updated_at > :watermark is evaluated against the current table; a row that is gone matches nothing, cannot be returned, and leaves no trace. The absence of evidence is literally all there is.

The only ways to see it are to compare against a full inventory of what you had (full compare), or to read a record of the deletion itself — a trigger-written audit row, or the write-ahead log.

13.12 (Run these; at least one will return rows.)

-- Lie 1: updated_at set at transaction start, not commit.
--   Look for rows whose updated_at precedes a row with a LOWER xmin.
SELECT count(*) AS suspicious
  FROM (SELECT updated_at, xmin::text::bigint AS x,
               lag(updated_at) OVER (ORDER BY xmin::text::bigint) AS prev
          FROM orders ORDER BY xmin::text::bigint DESC LIMIT 100000) s
 WHERE updated_at < prev;

-- Lie 2: not updated on every write.
--   A row whose status implies a change later than its updated_at.
SELECT count(*) AS stale_updated_at
  FROM orders
 WHERE status IN ('shipped','delivered')
   AND updated_at < placed_at + interval '1 hour';

-- Lie 3: wrong time zone / local time.
SELECT column_name, data_type
  FROM information_schema.columns
 WHERE table_name = 'orders' AND data_type LIKE 'timestamp%';
-- 'timestamp without time zone' is the finding. Then:
SELECT min(updated_at), max(updated_at), now() FROM orders;

-- Lie 4: it goes backwards / into the future.
SELECT count(*) AS future_rows, max(updated_at) AS furthest
  FROM orders WHERE updated_at > now();

The fourth returns a non-zero count, because the seed generator plants a future timestamp deliberately. That is the one to sit with: a single row with updated_at in the future permanently poisons a MAX(updated_at) watermark — the extract advances past every real row and stops returning anything, forever, without failing.

And it is why the watermark should be MAX(updated_at) WHERE updated_at <= now(), or better, why the extract should assert that the new watermark is not in the future before storing it. Two lines.

13.14 (Measurement exercise.)

-- both sides, same expression, same column order, same null handling
SELECT customer_id,
       md5(concat_ws('|', coalesce(name,''), coalesce(email,''),
                          coalesce(postal_code,''), coalesce(region,''),
                          coalesce(status,''))) AS row_hash
  FROM customers;

scale small (47,000 customers)
  source hash + fetch        1.9 s
  target hash                0.4 s
  compare (full outer join)  0.3 s
                            ──────
                             2.6 s

extrapolated to 1,900,000 rows (x 40.4)
  ~105 s, of which ~77 s is the SOURCE-SIDE scan

Is it viable at 1.9 million rows? Yes, and not for the reason the number suggests.

105 seconds of compute is nothing. The binding constraint is the 77 seconds of full table scan on kestrel_app — a database you are a guest in (§7.1) — repeated every run. That evicts the buffer cache the checkout path depends on, and it holds a snapshot for its duration (§7.3).

So the decision is not about your runtime, it is about theirs, and the answer is: viable daily, against a replica, chunked; not viable hourly, and not viable at all against the primary.

Where it stops being viable outright is around 50–100 million rows, where the scan starts taking tens of minutes and the snapshot duration becomes a bloat problem no chunking fixes (Exercise 7.12 — chunking a full compare breaks the comparison, because chunks see different snapshots).

One optimisation worth knowing: if the source can compute the hash, transfer only (key, hash) — 47 bytes a row instead of 200 — which cuts the network cost by 4× and changes nothing about the scan. The scan is the cost, and no client-side cleverness reduces it.

13.16 (Implementation.) The checker and its two runs:

def schema_signature(conn, table):
    return {r["column_name"]: (r["data_type"], r["is_nullable"])
            for r in conn.query(INFORMATION_SCHEMA_Q, table=table)}

def drift(previous, current):
    return {
        "added":    sorted(set(current) - set(previous)),
        "removed":  sorted(set(previous) - set(current)),
        "retyped":  sorted(c for c in set(previous) & set(current)
                           if previous[c] != current[c]),
    }

run 1, baseline                    no drift
run 2, after ADD COLUMN promo_code added: ['promo_code']
run 3, after RENAME region -> sales_region
                                   added:   ['sales_region']
                                   removed: ['region']

Why it cannot tell a rename from a drop-plus-add: because at the level it is looking, they are the same event. information_schema reports a set of column names; a rename produces exactly the byte pattern a drop and an add produce. Nothing in the catalogue records the transition — there is no "this column used to be called X."

Three consequences worth drawing out.

The severity is completely different and the checker cannot assign it. A rename is added + removed with the data intact; a genuine drop-and-add is added + removed with the data gone. The checker must report both as "manual review," which is the right answer and an unsatisfying one.

A heuristic is available and is a trap. You could compare data types and value distributions and guess. Do not — a wrong guess here means silently mapping one column onto another, which is worse than reporting ambiguity.

And this is exactly why table formats have column mapping (Chapter 10 §10.7): the physical column carries a stable id, so a rename is a metadata change on a column that never moved. The drift checker's blind spot is a missing identifier, not a missing feature.

13.19 What to compare: the measure the business would notice, not the row count alone. For Kestrel: daily order count and daily gross revenue in cents, source against bronze.

At what grain: by day, and by day and channel. The channel breakdown matters because a watermark bug loses rows non-uniformly — the rows lost are those written by long transactions, which cluster in one code path.

With what tolerance: the trap is a symmetric threshold, so the design has to be asymmetric.

FAIL      any day where |bronze - source| / source > 0.1%
FAIL      any day where bronze < source, by ANY amount, on TWO consecutive days
FAIL      any 7-day window where the SIGN of (bronze - source) is the same
          on 5 or more days, regardless of magnitude

The third rule is the one that catches what Chapter 4's Case Study 2 missed. A watermark bug loses a small number of rows every night — well inside any percentage threshold — but it loses them in one direction, every time. A tolerance band cannot see that; a sign test can, and it needs no threshold tuning at all.

Why the case study's threshold hid twenty-six nights: it asked "is tonight's variance large?" and the answer was honestly no. The right question is "is the variance random?" — twenty-six same-signed nights has a probability of about 1 in 33 million under a fair-coin null, which is a finding you can put in an alert without arguing about what tolerance is reasonable.

And run the reconciliation against the source, not against the previous run. Comparing bronze to yesterday's bronze detects a change in the bug's behaviour, not the bug.

13.21 They are reconcilable because they answer different questions. §13.7's rule is about fidelity to the schema — do not choose which columns matter, because a column you dropped is unrecoverable. §13.8's exception is about a specific column whose content is enormous and whose absence is recoverable.

The rule, written so a reviewer can apply it:

Bronze lands every column, with two exceptions, and each requires an explicit entry in platform/docs/bronze-projections.md:

1. The column is large and derived. It exceeds 4 KB on average and it can be recomputed or re-fetched from a source that will still exist — a rendered document, a cached blob, a denormalised copy of data landed elsewhere. "We do not use it" is not a reason; "we can get it back" is.

2. The column is forbidden. Policy or law prevents storing it at all (ADR-004). This requires legal sign-off, not a reviewer's.

In both cases the extract lands a _projected_columns list, so the omission is visible in the data rather than only in a document — and a future reader who wonders why bronze does not match the source has an answer in the row itself.

And the reviewer's test is one question: if we needed this column in two years, what would we do? If the answer is "re-fetch it," approve. If it is "we could not," reject.

13.23 (Implementation.) The guard the exercise is really about:

# backfill.py -- the four rules, plus the guard Chapter 1's incident needed
if os.environ.get("AIRFLOW_CTX_DAG_ID") or os.environ.get("SCHEDULED_RUN"):
    sys.exit(
        "REFUSED: this backfill is running from a scheduler.\n"
        "A backfill is a manual, reviewed operation. It ran nightly for 31\n"
        "days in the 2025-03 incident because it was scheduled, and nothing\n"
        "stopped it.\n"
        "To proceed: run it interactively, with --i-have-read-the-runbook,\n"
        "after the review in docs/runbooks/backfill.md."
    )

Three properties make that guard work rather than get deleted.

It detects the scheduler by environment, not by a flag — a flag can be set by the scheduler, and eventually will be.

The message says what to do, so the next person routes around it correctly rather than by commenting it out.

And it names the incident. A guard whose rationale is in the message survives the refactor; one that says RuntimeError: not allowed does not. This is the same principle as §14.19's runbook and §13.22(e)'s lag guard: a control whose reasoning is not attached to it will be removed by someone acting reasonably.

Chapter 14 — Change Data Capture

14.1 Four properties of a write-ahead log that make it suitable for CDC:

1. It is in commit order. Positions are assigned at commit, monotonically, by the database — which is exactly the ordering a watermark on updated_at fails to provide (§2.3, §4.3).
2. It is complete. Every change is in it, including hard deletes, including changes made by paths that bypass the application.
3. It carries before and after images, so a delete is an event with content and an update is a diff rather than a new state.
4. It already exists. The database writes it for durability and replication whether or not you read it, so reading it adds no write amplification — only retention.

14.3

14.5

Log-based reads the database's write-ahead log. Complete, ordered, no schema change, and essentially free on the write path. Requires elevated privileges, a replication slot, and the operational attention of §14.6.

Trigger-based installs AFTER INSERT/UPDATE/DELETE triggers that write to an audit table. Complete and ordered, and it works on any database.

Query-based polls with a watermark or a version column. Simple, needs nothing from the source, and inherits every one of §13.4's lies.

Trigger-based is right when you cannot have the log: a managed database that does not expose logical decoding, a version too old to support it, a vendor system where you can create objects but not change server configuration, or a case where you need only two tables of forty and a full CDC stack is disproportionate.

What it costs the source, and this is the part to say out loud: every write now does two writes, inside the same transaction. Insert throughput drops measurably — commonly 10–30% on a write-heavy table — and the audit table itself grows, needs indexing, needs pruning, and is a new object in someone else's schema that you are responsible for. You have moved your cost onto the transactional path, which is the thing §7.1 says you must not do lightly. It is a legitimate choice and it must be a negotiated one.

14.7 REPLICA IDENTITY DEFAULT puts only the primary key columns in the before image. Everything else is null.

What breaks silently: anything that needs to know the previous value of a non-key column.

an UPDATE changing status 'paid' -> 'cancelled'
  before: {order_id: 88214}                    <- that is all you get
  after:  {order_id: 88214, status: 'cancelled', ...}

Three consequences, all quiet:

You cannot compute a transition. "How many orders moved from paid to cancelled" is unanswerable, and the query that tries returns NULL in one column and looks like a data quality problem rather than a configuration one.

A DELETE carries only the key, so a soft-audit of "what was this row when it was deleted" is empty — which is precisely the question deletes get asked.

And SCD Type 2 built from the stream is subtly wrong. The valid_to version of a row needs its old attributes; with DEFAULT you must reconstruct them from the previous event, which works only if you have every event and never compacted the topic.

The fix is REPLICA IDENTITY FULL, and it is not free: the WAL now carries the entire old row on every update. Kestrel measured 31% more WAL volume on orders (Exercise 14.12), which is paid in disk, in replication bandwidth, and in slot growth when a connector stalls.

14.9 Because a consistent snapshot holds a single transaction — and therefore a single snapshot — for the entire duration of reading the table.

On a large table that is hours. For those hours the source cannot vacuum (§7.3), the connector cannot process live changes (so the slot grows), and any failure means starting over from zero.

An incremental snapshot chunks the table and interleaves the chunks with live streaming. Each chunk is a short transaction, so no long snapshot is held; the stream keeps flowing, so the slot drains; and a failure resumes at the last completed chunk rather than at the beginning.

It is slower in wall clock — more round trips, and the watermarking protocol (Exercise 14.21) adds work — and it is the right choice anyway, because the property that matters is not "finishes soonest" but "does not hold a snapshot for four hours and cannot lose four hours of progress."

This is the same trade as §7.7's chunked extract, and it is worth noticing that it recurs: bounded work units beat a single large one whenever failure is possible and the source is shared.

14.12 (Measurement.) The method and a representative result:

SELECT pg_current_wal_lsn();                    -- before
-- run a fixed workload: 10,000 single-row UPDATEs on orders
SELECT pg_current_wal_lsn();                    -- after
-- difference:
SELECT pg_wal_lsn_diff(:after, :before) AS wal_bytes;

REPLICA IDENTITY DEFAULT   10,000 updates    18.4 MB WAL   1,930 B/update
REPLICA IDENTITY FULL      10,000 updates    24.1 MB WAL   2,530 B/update
                                                           ────────────
                                             +31.0%

Two things to check in your own numbers.

The percentage depends entirely on row width, so it is not a transferable constant. A narrow table with a wide primary key might see 5%; a wide table with a small key sees far more than 31%. Kestrel's 31% is a fact about orders, not about PostgreSQL — which is why the chapter attaches it to a table.

And it depends on what the update touches. PostgreSQL's WAL records a full page image on the first write to a page after a checkpoint, which can swamp the row-image difference. Run the workload twice and take the second measurement, or your result will be dominated by checkpoint timing rather than by REPLICA IDENTITY.

14.14 The mechanism. A replication slot's position advances only when the consumer acknowledges an LSN. A connector capturing only orders receives nothing when only other tables are being written — logical decoding filters the stream to the publication — so it has nothing to acknowledge, and its position stays where it was. Meanwhile the database keeps generating WAL for every other table, and every WAL segment after the slot's position must be retained.

A low-volume table therefore pins the WAL of the whole database. The quieter the captured table, the worse it is — which is the opposite of everyone's intuition.

heartbeat.action.query fixes it by giving the connector something to see. The connector periodically executes a write against a heartbeat table that is in the publication; that write produces a WAL record the connector receives, acknowledges, and thereby advances the slot past everything before it.

The experiment:

CAPTURE     a publication containing only `warehouses` (7 rows, never written)
WRITE       a steady load against `orders` -- say 200 updates/sec
MEASURE     pg_replication_slots.retained_bytes, sampled every 30 s

Phase 1, heartbeat DISABLED, 15 minutes:
    retained_bytes climbs monotonically, tracking total WAL generation.
    Nothing is wrong with the connector; it is 'RUNNING' throughout.

Phase 2, enable heartbeat.action.query with a 10 s interval:
    retained_bytes drops to near zero within one heartbeat interval and
    stays flat while the same write load continues.

The chart is the whole point: identical write load, identical connector, and the retained-WAL curve goes from a ramp to a flat line because of one configuration property. And note what the experiment also shows: the connector's status is RUNNING in both phases, which is §14.10's argument in a single screenshot.

14.16

per order: pending -> paid -> picked -> shipped -> delivered
           1 INSERT + 4 UPDATEs                    =  5 CDC events

CDC, annually
  2,400,000 orders x 5                             = 12,000,000 events

HOURLY BATCH: captures the row once per HOUR IN WHICH IT CHANGED.
  The five transitions span ~4 days, so they generally fall in five
  distinct hours -- EXCEPT pending -> paid, which happens at checkout,
  within minutes. Those two collapse into one poll.

  realistic                                        =  4 rows/order
  2,400,000 x 4                                    =  9,600,000 rows

difference                                         =  2,400,000 records
at ~400 bytes                                      =    960,000,000 bytes
                                                   =       0.894 GB
                                                   =  $0.021 / month
                                                   =  $0.25  / YEAR

Twenty-five cents. And the whole CDC bronze stream — 12,000,000 events × 400 bytes = 4.47 GB — costs $1.23 a year.

The finding is that §14.2's stated price for point 3 is not a real price. "Volume — four events where you had one row" sounds like a cost and is, at Kestrel's scale, a rounding error. The actual price of CDC is point 1's: the operational burden of slots, connectors, and snapshots — an engineer's attention, which is the scarcest thing in a four-person team and does not appear on any bill.

The transferable habit is to price the stated cost before accepting it. A trade-off you have not quantified is a trade-off you cannot make, and here the arithmetic removes one side of the argument entirely.

(Two assumptions worth naming: 400 bytes per event is an estimate that depends heavily on REPLICA IDENTITY FULL — with the full before-image it is closer to 700, which changes the answer to $2.15 a year and not the conclusion. And the batch figure assumes an hourly cadence; a daily batch captures 4 rows per order too, because the order spans four days.)

14.19

# Runbook: replication slot filling the disk

## Detection -- which alert fires first
`slot_retained_bytes` above 50% of `max_slot_wal_keep_size`, with an ETA.
This fires HOURS before the disk alert, and it is the one to act on.

If the DISK alert fires first, the slot monitor is not running or its
threshold is wrong. Fix that after the incident.

## Triage -- is this the slot, or something else?
    SELECT slot_name, active, active_pid,
           pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(),
                                          restart_lsn)) AS retained
      FROM pg_replication_slots ORDER BY 3 DESC;

    SELECT pg_size_pretty(sum(size)) FROM pg_ls_waldir();

Slot cause:      pg_wal is large AND a slot's retained bytes explain it.
Not the slot:    pg_wal is normal -- look at the data directory, a
                 runaway table, a stuck `pg_dump`, or log files.
Also check:      a long-running transaction (pg_stat_activity), which
                 pins WAL the same way and is NOT fixed by anything below.

## Immediate action
1. Is the consumer alive?  `active = false` means it is not.
     -> restart the connector. Retained WAL drains as it catches up.
        This resolves ~80% of these.
2. Alive but not advancing? Check for a poison message in the connector
   log. Skip it explicitly (`FORCE` the offset) rather than dropping
   the slot.
3. Buy time: extend the volume. Cheaper and more reversible than
   anything else on this page.

## Recovery
Watch retained_bytes fall. Do not declare the incident over until it is
back to its normal band -- a connector that catches up and then stalls
again looks identical at the moment of recovery.

## The decision: drop the slot?
Dropping the slot frees the WAL immediately and PERMANENTLY BREAKS CDC.
Recovery requires a full re-snapshot of `orders` and `order_items`,
which is ~4 hours and a gap in silver until it completes.

DROP IT WHEN:
  - projected time to disk-full is under 60 minutes, AND
  - the connector cannot be restarted within that window, AND
  - extending the volume is not possible in that window.

DO NOT DROP IT because retained_bytes is large. Large is normal after
an outage; growing toward a deadline is the problem.

Before dropping, RECORD `restart_lsn`. It does not let you resume, and
it tells you exactly how much you lost, which the re-snapshot
reconciliation will need.

## Afterwards
- Was `max_slot_wal_keep_size` set? If not, that is the fix. It
  invalidates the slot instead of filling the disk -- the same
  outcome, chosen by you, at a threshold you picked.
- Was a heartbeat configured? (section 14.11)
- Did the slot have an owner in its name? An unowned slot is the
  proof-of-concept case in section 14.6.

The threshold is the answer the exercise wants, and the reasoning behind "60 minutes" is what makes it defensible: dropping the slot costs a known four hours of degraded silver, while filling the disk costs an unplanned outage of kestrel_app — checkout down, not analytics down. You accept the known four-hour cost only when the alternative is imminent and unavoidable, and 60 minutes is roughly the time in which a connector restart plus a volume extension can both be attempted and both fail.

14.21 The problem the watermarks solve: reconciling a chunked snapshot with a concurrent change stream, without locking and without losing changes.

While chunk k is being read, rows in that chunk may be updated. The snapshot read returns the old value; the change stream carries the new one. Without ordering information you cannot tell which is newer — the snapshot's read happened at some point inside a window, and the change event carries an LSN, and the two are not comparable.

The DBLog design (Andersen et al., Netflix, 2019 — "DBLog: A Watermark Based Change-Data-Capture Framework", arXiv:2010.12597) solves it by writing markers into the same log the changes flow through:

1. write LOW watermark to the watermark table    -> appears in the log
2. SELECT the chunk                              -> the snapshot read
3. write HIGH watermark to the watermark table   -> appears in the log
4. process the log; for events between LOW and HIGH, REMOVE from the
   chunk any row whose key also appears as a change event
5. emit the (deduplicated) chunk, then the change events

The insight is that the markers put the snapshot read into the log's own total order. Any row that changed during the read shows up between the two watermarks, and the rule "the stream wins" is now decidable rather than a guess.

What goes wrong without them: a row updated during the chunk read is emitted twice — once from the snapshot with the old value, once from the stream with the new value — and their relative order is undefined. If the snapshot's version is applied after the stream's, the row reverts. The corruption is silent, affects only rows that happened to change during a specific read window, and is irreproducible — which is the worst combination of properties a bug can have.

(Debezium's implementation follows DBLog closely, with two differences worth knowing: it uses a signalling table for chunk requests so a snapshot can be triggered on a running connector, and it supports pausing and resuming a snapshot. Check the current Debezium documentation for the signal-table schema, which has changed between versions.)

14.23 (Implementation.) The field the exercise is about:

eta_hours = (ceiling_bytes - retained_bytes) / max(wal_bytes_per_hour, 1)

slot_name              owner      active  retained   ceiling   ETA
cdc_orders_dataeng     dataeng    yes       2.1 GB     64 GB   -- (draining)
cdc_events_dataeng     dataeng    yes      41.2 GB     64 GB   6.1 h   <-- !
debezium_poc           (none)     NO       58.9 GB     64 GB   1.4 h   <-- !!

Three columns do the work and only one of them is the measurement.

The ETA turns a fact into a deadline. "Slot at 41 GB" prompts nothing; "invalidated in 6.1 hours" prompts a response, and it lets an on-call engineer rank two findings without knowing the system.

active = false is the highest-severity finding on the page, regardless of size, because it means nothing is draining and the ETA is the only direction the number will move.

And the owner column, parsed from the name, is what makes the second row actionable at 3 a.m. debezium_poc has no owner because nobody named it — which is §14.6's abandoned proof-of-concept slot, and the fact that the parser has to print (none) is the finding.

Chapter 15 — Event Streaming with Kafka

15.1 A Kafka topic is a partitioned, append-only, durable log with a retention policy.

The four properties follow directly:

Ordering is per partition, not per topic — because a log is ordered and a topic is several logs.

Reading is non-destructive and position-based — because consumers hold an offset into an append-only structure rather than removing entries from a queue.

Replay is free — because the data is still there; rewinding is setting a number.

Multiple independent consumers cost nothing extra — because each holds its own offset into the same log, so there is no fan-out copy.

15.3 The partition key does three jobs at once:

1. It determines the partition — hash(key) % partitions.
2. It therefore determines the ordering group — all records with the same key are in one partition and are ordered relative to each other.
3. It determines the compaction identity — under cleanup.policy=compact, the key is what is deduplicated to its latest value.

A null key gives up all three. Records are distributed round-robin (or by sticky batching), so there is no ordering guarantee between any two records, and the topic cannot be compacted at all.

The one people discover late is the second. A null key on a clickstream is fine; a null key on a CDC or state-change topic means two updates to the same entity can be processed out of order by different consumers, which produces a stale final state that nothing detects.

15.5 Because acks=all means "all in-sync replicas," and the in-sync set can shrink.

replication.factor = 3, min.insync.replicas unset (default 1)

normal:   ISR = {leader, r2, r3}   acks=all waits for 3.   Safe.
r2, r3 fall behind or die:
          ISR = {leader}           acks=all waits for 1.   = acks=1

The producer's configuration did not change and its guarantee did. It is still waiting for "all" replicas — there is now one. A leader failure at that moment loses acknowledged writes.

min.insync.replicas=2 fixes it by making the broker refuse the write (NotEnoughReplicas) rather than silently accepting it with a weaker guarantee. The write fails, loudly, and the producer retries — which is the correct behaviour and is what people find surprising: the fix makes the system less available on purpose, because the alternative is being available and wrong.

And the arithmetic is worth stating: replication.factor=3 with min.insync.replicas=2 tolerates one broker failure while preserving the guarantee. Setting min.insync.replicas=3 tolerates none.

15.7 Because auto-commit commits offsets on a timer, based on what has been polled, not on what has been processed.

t=0.0   poll() returns records 100-199
t=5.0   auto-commit fires. Offset 200 is committed.
t=5.1   the consumer is still processing record 143.
t=5.2   the process crashes.
t=9.0   a new consumer starts at offset 200.
        Records 143-199 were never processed and never will be.

The offsets say the work is done; the work is not done. That is at-most-once, and nothing in the configuration is named "at most once" — it is the default behaviour of a convenience feature.

The fix is enable.auto.commit=false and an explicit commit after the write succeeds, which is three lines and converts the pipeline to at-least-once. Combined with an idempotent write, that is the guarantee this book uses everywhere (§4.7, §15.7).

15.9 Retention deletes records by age or by total size, regardless of key. Compaction retains the most recent record per key, forever, deleting only superseded versions.

Compaction makes a topic a table, because the invariant it maintains — one current value per key — is the definition of a keyed table. A consumer replaying a compacted topic from offset 0 reconstructs the current state of every key that has ever existed, which is a table materialised from a log.

And the reason it works is the reason §15.3's third job exists: compaction is defined entirely in terms of the key, so a topic with null keys cannot be compacted, and a topic whose key is not the entity's identity compacts to something meaningless.

Two caveats the "topic as a table" framing hides. Compaction is eventual — the log tail is uncompacted, so a replay sees duplicates and must take the last one. And a deletion requires a tombstone: a record with the key and a null value, which is retained for delete.retention.ms and then removed. A key deleted from the source with no tombstone written stays in the compacted topic forever, which is a real and common data-retention problem (Chapter 31).

15.12

Little's Law floor
  L = lambda x W = 900 events/s x 0.025 s          =  22.5 handlers

Multipliers
  growth, 4x over three years                       x 4    =  90
  catch-up after an outage, 1.5x                    x 1.5  = 135

Choose a partition count >= 135 with many divisors:

  135  = 3^3 x 5        divisors: 1,3,5,9,15,27,45,135      poor
  144  = 2^4 x 3^2      divisors: 1,2,3,4,6,8,9,12,16,18,
                                  24,36,48,72,144           excellent
  150  = 2 x 3 x 5^2    divisors: 1,2,3,5,6,10,15,25,...    fair
  180  = 2^2 x 3^2 x 5  divisors: 18 of them                excellent

CHOOSE 144.

The defence.

It clears the floor with the growth and catch-up multipliers already applied, and partition counts can only go up — an increase re-hashes keys to different partitions and permanently breaks per-key ordering for the data either side of the change (§15.2). Sizing for three years is not gold-plating; it is avoiding an irreversible operation.

144 distributes evenly across every consumer count anyone will plausibly run. 2, 3, 4, 6, 8, 9, 12, 16, 18, 24, 36 — all exact. 135 divides evenly only by 3, 5, 9, 15, 27 and 45, so a group of 4, 6, 8 or 12 consumers is unbalanced, and an unbalanced group is sized by its busiest member.

And 144 is comfortably inside the operational envelope. On a three-broker cluster that is 48 partitions per broker, against a comfortable ceiling of a few hundred (§15.8).

One honest caveat, and it is §15.8's own. If the 25 ms is per batch rather than per event, the floor is 0.225 handlers and the answer is 12. The arithmetic is trivial and the unit is where the factor of 100 lives — measure the handler, do not accept the number in the ticket.

15.14 compact retains the latest record per key forever, so the topic is a materialisable table (§15.9). delete bounds the topic by age, so it does not grow without limit.

The combination gives you both: current state for every key that is still current, and a hard bound on how far back the history goes. Under compact,delete, a segment is compacted and segments older than retention.ms are deleted regardless.

What goes wrong with each alone.

compact alone: unbounded growth in key space, and a retention obligation you cannot discharge. A CDC topic accumulates one record per key ever seen. Kestrel's orders gains 2.4 million keys a year, permanently — and every one of them carries personal data that a deletion request must reach (Chapter 31). A compacted topic with no deletion is a data-retention liability that grows linearly forever.

delete alone: the topic stops being a table. A consumer that bootstraps from offset 0 gets only the retention window, so any key not updated within it is simply missing. A new consumer's view of "all orders" silently excludes every order older than the window, and it looks like a complete dataset.

The retention I would choose: 30 days.

The reasoning is about bootstrap, not about history. The retention window must exceed the longest plausible gap between a consumer being written and being deployed — a consumer built against a seven-day topic and deployed nine days later has an incomplete bootstrap and no error. 30 days covers a sprint boundary, a holiday, and an incident. It is also short enough that the deletion obligation is bounded at a month, which is the number legal will ask about.

And the honest caveat: with compact,delete, a key whose last update is older than the retention is deleted from the topic entirely. That is correct for a bounded topic and it means the compacted "table" is not complete, so the authoritative store remains the database. The topic is a transport, not a source of truth — which is worth writing into the contract (Chapter 17).

15.16

# replay.py -- read the DLQ, group, and re-produce. Safe to run twice.
def survey(consumer):
    counts = collections.Counter()
    for msg in drain(consumer):
        counts[header(msg, "dlq.reason")] += 1
    return counts.most_common()

def replay(consumer, producer, reason, dry_run=True, limit=None):
    n = 0
    for msg in drain(consumer):
        if header(msg, "dlq.reason") != reason:
            continue
        if dry_run:
            print(header(msg, "dlq.source_topic"),
                  header(msg, "dlq.source_offset")); n += 1; continue
        producer.produce(
            topic=header(msg, "dlq.source_topic"),
            key=msg.key(),
            value=msg.value(),              # ORIGINAL bytes (§15.10)
            headers={
                "replay.of_offset": header(msg, "dlq.source_offset"),
                "replay.at":        now_iso(),
                "replay.attempt":   str(int(header(msg,"replay.attempt","0"))+1),
            })
        n += 1
        if limit and n >= limit:
            break
    return n

Four things make it safe to run twice, and three of them are not in the obvious place.

--dry-run is the default. A replay tool whose default action is to write to a production topic will eventually be run by someone who meant to look.

It does not commit its DLQ offsets until the produce is acknowledged, so a crash mid-replay re-reads rather than skips.

Re-produced records carry replay.of_offset and replay.attempt. The downstream consumer is idempotent on event_id (§15.7), so a double replay is a no-op at the sink — but the headers are what let you prove that after the fact, and they let you find a record that has been replayed nine times and is never going to work.

And it filters by dlq.reason. Replaying the whole DLQ replays the poison messages along with the recoverable ones, which is how a replay becomes an incident. Survey first, fix one cause, replay that cause.

15.19

Three properties the messages share, and they are the part of this exercise that transfers.

Every message contains a rate or an ETA, not only a level. "Lag 812k" is unactionable; "rising 40k/min, 4.2 h to retention" is a deadline.

Every message names the most likely cause and the thing not to do. The rebalance alert says do not add consumers because adding consumers is what everyone does at 04:00 and it makes a storm worse.

And the DLQ alert includes the dominant reason and the owning team, so the recipient can route it in one step. The single most valuable thing in an alert is the sentence that tells the reader whether this is theirs.

15.21 (Research exercise — verify against current Kafka Connect documentation.)

What they actually do: two-phase commit against the sink, with the offsets stored in the sink itself.

The mechanism has two variants, and the distinction matters:

Sinks with transactions (a relational database, and Connect's own JDBC sink in exactly-once mode): the connector writes the records and the Kafka offsets into the sink in one sink-side transaction. Because the offsets live in the same transactional store as the data, "have I already written this batch" is answerable atomically. This is genuine end-to-end exactly-once, and it works because the sink has become the coordinator rather than because Kafka reached further.

Sinks without transactions (object storage — S3, GCS): the connector writes files whose names are deterministic functions of (topic, partition, start_offset, end_offset), and commits by an atomic rename or a conditional put. Re-processing writes the identical file to the identical key, so the duplicate is absorbed. That is not exactly-once delivery — it is at-least-once delivery plus an idempotent write, which is exactly what this book recommends, implemented for you.

What it requires of the sink, and this is the answer to the question: either a transaction that can include the offsets, or a deterministic, idempotent addressing scheme. A sink with neither — an HTTP endpoint that creates a resource, an email, a webhook — cannot be given this guarantee by any connector, and connectors for such sinks do not claim it.

Does it change the chapter's claim? No, and it sharpens it. "Exactly-once inside Kafka and nowhere else" is about what Kafka's transactions reach. The connectors do not extend Kafka's transaction — they build a second mechanism in the sink and coordinate the two. The chapter's practical advice is unchanged: the guarantee lives at the write, and it requires something of the sink. What the connectors add is that somebody has already written it for the common sinks, which is a real and worthwhile thing.

15.23 (Implementation.) The distinction the tool exists to draw:

group: bronze-writer
  partition   lag      d(lag)/dt over 6 samples     shape
  p0        412,004    +38,200/min                  RAMP
  p1        398,551    +37,900/min                  RAMP
  p2          1,204    +/-1,100/min, mean ~0         SAWTOOTH
  p3            842    +/-  900/min, mean ~0         SAWTOOTH

VERDICT: p0 and p1 are under-provisioned. p2 and p3 are rebalancing.

A ramp and a sawtooth need opposite responses, which is why a single lag number is the wrong metric:

Rising lag = under-provisioned. The consumer cannot keep up. Add consumers (up to the partition count), or make the handler faster.

Sawtoothing lag = rebalancing. The group is repeatedly reassigning partitions, and progress resets. Adding consumers makes it worse, because each join triggers another rebalance. The fix is max.poll.interval.ms or max.poll.records (§15.5).

The tool's real output is therefore the verdict column, not the numbers. And the honest caveat is that six samples is barely enough to distinguish the two — a ramp with noise looks like a sawtooth at low sample counts, so the window must be long enough that the mean of the derivative is meaningful. Ten minutes at 30-second sampling is a reasonable floor.

Chapter 16 — API Ingestion

16.1 Five ways an API is harder:

1. You cannot ask what changed. There is no log, no watermark you control, and frequently no updated_at in the response.
2. Rate limits. The extract's speed is set by someone else, and exceeding it can get you banned rather than throttled.
3. Authentication expires. Tokens have lifetimes shorter than a long job.
4. Pagination is a protocol, and it can lose rows while you walk it.
5. The shape can change without notice, and there is no schema anywhere to compare against.

16.3

page size 100. The collection is ordered by created_at DESC (newest first).

t=0    rows in collection: [R100 ... R1]   (R100 newest)
       GET ?offset=0&limit=100  ->  R100..R1      100 rows read
t=1    THREE NEW ROWS INSERTED: R103, R102, R101 (now the newest)
       collection: [R103, R102, R101, R100 ... R1]
t=2    GET ?offset=100&limit=100
       offset 100 now points PAST the first 100 of the NEW ordering,
       which is [R103 ... R4].
       -> returns R3, R2, R1  ... and R4, R5, R6 were SKIPPED.

Three rows inserted, three rows lost, and the extract reports success. The mechanism is that offset is a position in a result set that is being recomputed on every request, and inserts at the head shift every subsequent row's position.

And the reverse happens with deletes, producing duplicates instead of gaps — which is the more detectable of the two and therefore the less dangerous.

The fix is cursor pagination, where the cursor encodes a stable position in a stable ordering ("records after id X"), so concurrent inserts elsewhere in the collection cannot move it.

16.5 Because an empty page is not the only thing a last page can look like, and an empty page is not always the last page.

A last page can be full. If the collection has exactly 300 records and the page size is 100, the third page returns 100 rows and no next token. Terminating on emptiness makes a fourth request that either errors, returns nothing, or — on a badly behaved API — returns page 1 again.

And an empty page can appear mid-stream. Some APIs filter after paginating, so a page whose records were all excluded comes back empty while a next token is still present. Terminating there silently truncates the extract.

The token's absence is the API's own statement that there is no more, and it is the only signal with that meaning. Everything else is an inference about the API's behaviour that will be wrong for one of the APIs you integrate.

16.7

The last row is the one that is not about status codes at all, and it is the one that gets pipelines into trouble — a transport error is retryable as a transport matter and may not be retryable as a semantic one.

16.9 Jitter randomises the retry delay so that clients which failed together do not retry together.

Without it, 200 clients that saw the same 503 at t=0 all sleep exactly 1 second, and all retry at t=1.0. The service — which is recovering, or is behind a rate limiter — receives a 200-request spike at the exact moment it is least able to serve it, fails them all again, and they all retry at t=3.0. Exponential backoff without jitter does not spread load; it synchronises it into an accelerating series of spikes, and a service that would have recovered stays down.

With jitter, each client's delay is drawn from a range, so the same 200 retries arrive spread over seconds. The recovering service sees a ramp rather than a wall.

The subtle part: full jitter beats "add a small random amount." sleep(random(0, base * 2^n)) spreads far better than sleep(base * 2^n + random(0, 0.1)), because the second still has a mode at the deterministic value. The spread has to be on the order of the delay itself to be worth anything.

16.12 (Implementation.) The three assertions and what each is really testing:

def test_does_not_exceed_rate():
    rl = RateLimiter(per_second=10, burst=10)
    clock, calls = FakeClock(), []
    for _ in range(400):
        rl.acquire(clock); calls.append(clock.now())
    # any 1-second sliding window holds at most 10 calls
    for i, t in enumerate(calls):
        window = [u for u in calls[i:] if u < t + 1.0]
        assert len(window) <= 10, "burst of %d at t=%.2f" % (len(window), t)

def test_obeys_retry_after():
    rl = RateLimiter(per_second=10, burst=10); clock = FakeClock()
    rl.observe(Response(429, headers={"Retry-After": "30"}), clock)
    t0 = clock.now(); rl.acquire(clock)
    assert clock.now() - t0 >= 30.0

def test_slows_when_remaining_is_low():
    rl = RateLimiter(per_second=10, burst=10); clock = FakeClock()
    rl.observe(Response(200, headers={"X-RateLimit-Remaining": "3",
                                      "X-RateLimit-Reset": "60"}), clock)
    t0 = clock.now()
    for _ in range(3): rl.acquire(clock)
    assert clock.now() - t0 >= 40.0   # 3 requests stretched across ~60 s

Use a fake clock, not time.sleep. A rate-limiter test that actually sleeps takes 30 seconds and will be deleted by whoever is trying to make CI fast. Injecting the clock is the difference between a test that exists in a year and one that does not, and it is the same design that makes the backoff testable.

And the third test is the one that matters most and is usually missing. A local token bucket knows your rate and not your quota — it has no idea that another job on your team is consuming the same allowance. X-RateLimit-Remaining is the server telling you what it can see and you cannot, and a limiter that ignores it is correct locally and wrong globally.

16.14 (Implementation.)

class TokenProvider:
    def __init__(self, fetch, margin_seconds=120):
        self._fetch, self._margin = fetch, margin_seconds
        self._lock = threading.Lock()
        self._token, self._expires_at, self._refreshes = None, 0.0, 0

    def get(self, now=time.time):
        t = now()
        if self._token and t < self._expires_at - self._margin:
            return self._token                       # fast path, no lock
        with self._lock:
            if self._token and now() < self._expires_at - self._margin:
                return self._token                   # someone else did it
            tok, ttl = self._fetch()
            self._token, self._expires_at = tok, now() + ttl
            self._refreshes += 1
            return self._token

def test_eight_threads_cause_one_refresh():
    p = TokenProvider(fetch=slow_fetch, margin_seconds=120)
    threads = [threading.Thread(target=p.get) for _ in range(8)]
    for t in threads: t.start()
    for t in threads: t.join()
    assert p._refreshes == 1

The double-check inside the lock is the whole design. Without it, eight threads queue on the lock and each one fetches in turn — eight refreshes, serialised, which is slower and wrong.

What goes wrong at a provider that revokes the previous token on issue — and this is the real question:

Every in-flight request holding the old token starts failing with 401. The refresh succeeded, the provider is healthy, and a burst of requests fails for a reason that looks like an auth outage.

Worse, the naive recovery makes it a loop. Each 401 triggers a refresh; each refresh revokes the token the other threads just fetched; those threads get 401s and refresh again. A single-revocation provider plus per-request refresh-on-401 is a self-sustaining refresh storm, and it is the kind of behaviour that gets an API key suspended.

Three defences:

Refresh proactively, well before expiry — the margin — so a refresh almost never coincides with in-flight requests using the old token.

Make refresh-on-401 idempotent per token generation. Record which generation a request used; a 401 from generation N refreshes only if the current generation is still N. A request holding a stale generation just retries with the current token and does not trigger anything.

And overlap where the provider allows it. Many providers keep the previous token valid for a grace period; if yours does, the problem disappears. Check, because the answer changes the design and it is one line in their documentation.

16.16 (Implementation.) The asymmetry is the design:

REQUIRED = {"id": str, "created_at": str, "status": str, "amount_cents": int}

def contract_test(sample):
    failures, notices = [], []
    for field, want in REQUIRED.items():
        if field not in sample:
            failures.append("MISSING required field %r" % field)
        elif not isinstance(sample[field], want):
            failures.append("RETYPED %r: want %s, got %s"
                            % (field, want.__name__, type(sample[field]).__name__))
    for field in set(sample) - set(REQUIRED):
        notices.append("NEW field %r (%s)" % (field, type(sample[field]).__name__))
    return failures, notices          # failures page; notices only log

Why the asymmetry is right. A missing or retyped field breaks you now — a cast fails, or worse, succeeds into the wrong type. A new field breaks nothing; it is information that the producer changed something, which you want recorded and do not want to be woken for.

Two refinements worth adding after the first month.

Notices should be deduplicated and summarised, or a new field logs 1,440 times a day and the signal is buried. Log the first occurrence per field per day.

And a notice that persists for 30 days should escalate to a review, not to a page. A new field that is still there a month later is a schema the producer now considers stable, and it belongs in the contract (Chapter 17). This is the mechanism by which an observed contract stays current without anybody remembering to check.

16.19 The alerting that would have caught it: count and alert on 429s at any non-zero rate.

MEASURE   responses by status code, per API, per hour
ALERT     any 429 at all, on the FIRST occurrence, at low severity
          (a channel message, not a page)
ESCALATE  429 rate above 0.1% of requests, sustained one hour -> page

The point is that the threshold is zero. A 429 during a grace period costs nothing and is a statement about the future: "you are over the limit, and one day we will start enforcing it." The alert's entire job is to make a free warning visible, and a threshold above zero converts a three-day warning into a same-day outage.

The generalisation — warning shots that pipelines routinely ignore:

Every row has the same shape: a system telling you about a future failure, through a channel that is technically visible and practically discarded. The pattern is worth naming because the fix is always the same and always cheap — route the signal somewhere a human sees it, with a threshold of zero, at a severity that does not wake anyone.

16.21 The mechanism: hash the live response's shape nightly and compare it to the fixture's.

def shape(obj, path=""):
    """A structural fingerprint: paths and types, not values."""
    if isinstance(obj, dict):
        out = {}
        for k, v in obj.items():
            out.update(shape(v, "%s.%s" % (path, k)))
        return out
    if isinstance(obj, list):
        return shape(obj[0], path + "[]") if obj else {path + "[]": "empty"}
    return {path: type(obj).__name__}

def fixture_drift(fixture, live):
    f, l = shape(fixture), shape(live)
    return {"gone":    sorted(set(f) - set(l)),
            "new":     sorted(set(l) - set(f)),
            "retyped": sorted(k for k in set(f) & set(l) if f[k] != l[k])}

What it costs to run: one request per endpoint per night, and no state beyond the fixtures already in the repository. It is the cheapest check in this chapter.

How it differs from the contract test — and the two are genuinely different, which is why you want both:

The second is the one that catches the dangerous case. A field added to the API breaks no contract and passes every contract test — and it means your fixtures no longer exercise the code path that handles it, so your green test suite is testing a version of the world that no longer exists. The tests do not fail; they become uninformative, which is worse, because nothing distinguishes an uninformative passing test from a meaningful one.

And the practical output is a re-record prompt, not a failure. fixture_drift finding anything should open a ticket to re-record, with the diff attached — because the fix is mechanical and the decision (does this change matter?) is not.

16.23 (Implementation.) The measurement the exercise is for:

re-extract six months of carrier tracking data

  from the API      2,190 fetch-days x ~48 requests, at 3 req/s
                    with a 429 at 200/min       ->  ~16 hours
                    and a real risk of a ban    ->  and one conversation

  from bronze       re-parse 2,190 partitions of raw JSON
                    locally, in parallel        ->  4 minutes

Roughly 240×, and the ratio is not the point. The point is that one of these is a capability and the other is a request. Re-parsing bronze is something an engineer does on a Tuesday afternoon without asking anyone; a 16-hour API replay against a partner's rate limit is a plan, a notification, and a risk.

This is the payoff for §9.4's rule, and it is the argument to make when someone proposes parsing at ingestion to save storage. The storage saved is a few gigabytes a year. What is spent is the ability to change your mind about what the data means, and you spend it at the moment you most need it — after you have discovered the parser was wrong.

Chapter 17 — Schema Evolution and Data Contracts

17.1 A contract does not stop producers from changing; it decides whether the incompatibility surfaces at the producer, at write time, as a failed deploy owned by the person making the change — or at the consumer, weeks later, as a wrong number owned by someone who did nothing.

17.3

The way to keep it straight is to ask what already exists. Backward compatibility protects a reader against the past, so the reader goes first. Forward compatibility protects a reader against the future, so the writer may go first. "Backward" is about your own history; "forward" is about somebody else's deploy schedule.

17.5 Because the registry's compatibility check is about representability, and the consumer's failure is about exhaustiveness.

Adding awaiting_stock to a status enum is backward compatible by every schema rule: an old record never contains the new value, and a new reader can read every old record. The registry accepts it, correctly.

The consumer's CASE statement is the problem, and it is not a schema artifact:

CASE status
  WHEN 'paid'      THEN ...
  WHEN 'shipped'   THEN ...
  WHEN 'cancelled' THEN ...
  ELSE 'cancelled'          -- <- the new value lands here. Silently.
END

The consumer does not fail. It classifies the new value as something else, and the number moves. In Kestrel's case new orders were counted as cancelled (§17.9), which is a revenue error produced by a compatible change.

The general form is worth extracting: a schema constrains the set of representable values, and a consumer usually constrains the set of expected values. The registry can check the first and has no access to the second — which is why the countermeasure is a consumer-side assertion against the declared enumeration (§17.5's point 3), not a stricter compatibility mode.

17.7 The seven, beyond the schema:

1. Ownership — a team, and a person, and a channel.
2. Semantics — what each field means, including units, time zones, and sign conventions.
3. Guarantees — freshness, volume, completeness, with numbers.
4. The consumer list — who reads this.
5. Change process — how a change is proposed, and to whom.
6. Deprecation window — how long the old version lives.
7. Status — agreed or observed (§17.8).

The two doing most of the work are the consumer list and the semantics.

The consumer list solves the producer's central problem. A producer cannot know who depends on them; that is not a failure of diligence, it is structurally unavailable information. The list is the only artifact that supplies it, and being on it is what entitles you to notice.

The semantics field catches the worst incidents, because the worst incidents have no schema change at all. A sign inversion, a timestamp redefined from submission to authorisation, a unit changed from dollars to cents — every one of those passes every compatibility check ever written, and the semantics field is the only place the intended meaning is recorded well enough to argue from.

17.9 Expand-contract:

1. EXPAND    add the new field / new version alongside the old. Both are
             written. Nothing breaks; nobody has to do anything yet.
2. MIGRATE   consumers move to the new one, at their own pace.
3. CONTRACT  remove the old one.

Step 2 takes the time — weeks to months, and it is entirely outside the producer's control, because it is other teams' roadmaps.

What makes step 3 possible: knowing that nobody is reading the old field any more, and that is two things. The consumer list (§17.4) tells you who to ask. And an observation — query logs, a usage metric, a deprecation counter incremented when the old path is taken — tells you whether the answer was true.

The observation is what turns step 3 from a negotiation into a fact. Without it, contraction depends on every consumer correctly remembering whether they still use a field, and the honest answer is that at least one of them will be wrong. A counter that has been zero for 30 days ends the conversation.

17.12

# platform/contracts/clickstream.v1.yml
name: kestrel.clickstream.v1
status: agreed
owner:
  team: "#web"
  contact: web-platform@kestrel.example
  escalation: "#web-oncall"

schema:
  format: avro
  registry_subject: kestrel.clickstream.v1-value
  compatibility: BACKWARD_TRANSITIVE      # consumers replay bronze (Ch. 15)

semantics:
  event_id: >
    A UUID minted ONCE by the client at the moment of the interaction and
    REUSED on every retry. It is not unique per delivery; it is unique per
    interaction. Consumers dedupe on it.
  event_ts: >
    The time the interaction occurred ON THE CLIENT, in UTC, ISO-8601 with
    an explicit offset. It is NOT the server receipt time -- see
    `ingested_at` for that. Client clocks are wrong: about 0.3% of events
    arrive with event_ts in the future, and consumers must tolerate it.
  anonymous_id: >
    A first-party cookie identifier, stable per browser per device. It is
    NOT a person: one person has several, and a shared device has one for
    several people. It is PERSONAL DATA (Ch. 31).
  customer_id: >
    Present ONLY after login, and therefore null for most events in a
    session that later logs in. Identity is resolved at the END of a
    session, not the start (Ch. 18 section 18.9).
  session_id: >
    NOT emitted by the producer. Derived downstream: 30 minutes of
    inactivity OR midnight UTC, whichever comes first. Listed here because
    consumers ask for it and must know it is not ours.
  value_cents: >
    Integer cents, USD, ALWAYS POSITIVE. A refund is a separate event
    type, not a negative value. (Ch. 7 section 7.5.)
  event_type: >
    Enumerated below. NEW VALUES MAY BE ADDED with a MINOR bump and 30
    days notice. Consumers MUST assert against this list (section 17.14)
    -- notice has been given and not received before.

guarantees:
  volume:
    typical_per_day: 14000000
    band: "+/- 25% day over day, excluding Black Friday (6.28x)"
  freshness:
    p50_seconds: 4
    p99_seconds: 90
    max_seconds: 900
  completeness: >
    Best effort. Client-side events are lost to ad blockers and closed
    tabs. We estimate 3-6% loss and do NOT guarantee a bound.
  ordering: "Per anonymous_id only. No cross-key ordering."

consumers:
  - team: "#data-platform"    use: "bronze landing, sessionization"
  - team: "#analytics"        use: "funnel and attribution models"
  - team: "#ml"               use: "recommendation features (Ch. 32)"

change_process:
  minor: "PR to this file, 30 days notice in #data-consumers"
  major: "expand-contract (section 17.9), 90 days, both versions live"
deprecation_window_days: 90

Four things the semantics section resolves that no schema can express: that event_id identifies an interaction rather than a delivery; that event_ts is a client clock and is sometimes in the future; that customer_id's nullness is structural rather than a quality problem; and that value_cents is never negative because refunds are a different event. Every one of those has caused an incident somewhere, and none of them is a type.

17.14

-- hand-written, for silver.orders.status
SELECT status, count(*) AS n
  FROM silver.orders
 WHERE status NOT IN ('pending','paid','picked','shipped',
                      'delivered','cancelled','refunded')
 GROUP BY 1;
-- expect zero rows

And the generator, which is the actual exercise:

def enum_assertions(contract_path, model):
    """Emit one assertion per enumerated field. New enum -> new check,
       automatically, with no human in the loop."""
    c = yaml.safe_load(open(contract_path))
    for field, spec in c["schema"]["fields"].items():
        values = spec.get("enum")
        if not values:
            continue
        yield {
            "name": "assert_%s_%s_in_contract" % (model, field),
            "sql": ("SELECT {f} AS unexpected_value, count(*) AS n\n"
                    "  FROM {m}\n WHERE {f} IS NOT NULL AND {f} NOT IN ({v})\n"
                    " GROUP BY 1").format(
                        f=field, m=model,
                        v=", ".join(repr(v) for v in values)),
            "owner": c["owner"]["team"],
            "contract": contract_path,
        }

Generating rather than writing is the entire point, and it is worth being explicit about why.

A hand-written assertion protects the field somebody thought about. Kestrel had one on status because status had already burned them. It had none on channel, payment_method, or fulfilment_type — three other enumerated fields, unprotected, for the same reason: nobody had been burned by them yet.

A generated assertion protects every enumerated field, including the ones added next year by someone who has never read this chapter. The coverage is a property of the mechanism rather than of anyone's diligence.

And the failure message should name the contract file and the owning team, because the person who sees the failure is the consumer and the person who must act is the producer. An assertion that says "unexpected value awaiting_stock" starts an investigation; one that says "unexpected value awaiting_stock, not in contracts/orders.yml, owner #commerce" starts a conversation.

17.16

# Violation log

## A record contains
    date            when it was detected
    contract        which one
    field           which field, or "-" for a whole-payload change
    kind            added | removed | retyped | semantics | enum | freshness
    detected_by     which assertion, BY NAME
    detected_where  producer CI | consumer CI | scheduled check | a human
    notice_given    yes / no  -- was there advance notice?
    lag_days        days between the change shipping and us detecting it
    impact          what was wrong downstream, in one line
    cost            engineer-hours + any business impact, if known

Captured automatically, from three sources: the consumer-side assertions (§17.5 point 3) write a row on failure; the schema-drift checker (§13.7) writes one on any change; and the scheduled guarantee checks (§17.5 point 4) write one on a freshness or volume breach. notice_given is the only field a human fills in, and it is filled in at detection time while the answer is still knowable.

The summary to bring to the conversation:

supplier-b feed -- 8 months

  14 changes detected
   2 with advance notice                              14%
  12 without                                          86%

  median lag from change to detection                  9 days
  worst                                               31 days

  by kind        retyped 5 · added 4 · removed 2 · semantics 2 · enum 1
  incidents caused                                     3
  engineer-hours                                      71
  one wrong figure published to the board            yes, 2026-04

Two properties make this persuasive where an argument is not.

It is about them and it is not an accusation. Every row is a change they were entitled to make. The finding is not "you broke us," it is "86% of the time, neither of us knew this affected the other" — which is a shared problem with a shared fix, and it is the framing that gets a yes.

And the ask is small and specific. Not "adopt data contracts": "tell us before you ship, and we will tell you what we read." The 86% is what makes that ask feel reasonable rather than bureaucratic — and it is only available because somebody logged for eight months before asking.

The discipline the log requires: start logging before you have a case. A log begun the week you decide to have the argument covers a week.

17.19 The strongest case that an unagreed contract is worse than none:

It creates a false belief on your own side. The document says orders.status has seven values with a 30-day deprecation window. Nobody at the producer has read it. Your team now designs against a guarantee that does not exist, and — worse — feels covered. The vigilance that the absence of a contract would have preserved is gone, replaced by a file. This is §17.10's own argument about unenforced contracts, applied to unagreed ones.

It gets cited in incidents, wrongly. When the producer ships a change, the post-mortem says "this violated the contract." It did not; there was no contract. The document has converted a shared coordination failure into an accusation, and it damages the relationship you need in order to ever get a real agreement.

And it decays without anybody noticing. An agreed contract has a counterparty who objects when it becomes wrong. An unagreed one has nobody, so it drifts from reality at exactly the rate the source changes — and it is most wrong precisely when it is most consulted, which is during an incident.

What §17.8 answers, and it answers all three: the status field.

An observed contract states, in the document itself, that it is a description of measured behaviour and not a promise. The false belief is impossible because the document says so. The wrong citation is impossible because there is nothing to violate. And the decay is bounded because an observed contract's guarantees are measured, not asserted — the freshness and volume numbers come from a scheduled check, so drift shows up as a failing assertion rather than as a stale sentence.

The residual the case is right about: an observed contract that is labelled agreed, or whose status field is left off, is exactly as bad as the argument says. The label is not a formality; it is the entire mechanism.

17.21 (Research exercise. Primary sources: the Pact documentation at docs.pact.io, particularly "Pact Broker" and "provider verification"; and Confluent's Schema Registry documentation on compatibility types.)

What each assumes about who owns the relationship.

Consumer-driven contract testing (Pact) assumes the consumer knows what it needs and the producer agrees to verify it. The consumer writes an expectation, publishes it to a broker, and the producer's CI fails if a change breaks any published expectation. The relationship is many-to-one and explicit: the producer has a list of consumers, by construction, because each one has published something.

The producer-enforced registry assumes the producer publishes a schema and consumers conform. The producer declares a compatibility mode; the registry rejects violations. The producer does not know who the consumers are — nothing in the mechanism requires it — which is why §17.4's consumer list has to be a separate, manually maintained field.

Which suits a data platform better — and the honest answer is that they solve different halves.

The registry is better at scale and at replay. A data platform has topics with many consumers, deep history, and bronze layers that get replayed years later. BACKWARD_TRANSITIVE against every historic version is a guarantee Pact does not offer, because Pact verifies against current expectations, not against the accumulated past.

Consumer-driven contracts are better at semantics and at the social problem. A Pact expectation can assert "when I ask for an order in state X, refund_cents is non-negative" — a claim about meaning, which no registry can express. And publishing an expectation is how a consumer becomes visible, which is §17.4's hardest field, solved mechanically.

For a data platform I would use both, and they map onto §17.5's enforcement points exactly: the registry at point 1 (producer deploy, structural), consumer-published assertions at point 3 (consumer CI, semantic). What does not transfer from the Pact world is the assumption of a synchronous request/response pair with a small number of well-known consumers — a Kafka topic read by an unspecified number of teams over three years is a different shape, and the broker's model of "verify against the current set of consumers" has no answer for the consumer that will exist in 2028 and read 2026's data.

17.23 (Implementation.)

# platform/contracts/carrier-b.tracking.yml
name: carrier-b.tracking
status: OBSERVED            # <- they have not agreed to any of this
observed_since: 2026-02-01
measured_over_days: 14

owner:
  producer: "Carrier B (external). No contact for schema changes."
  our_owner: "#data-platform"
  note: >
    We have a support address and an account manager. Neither has a
    channel for notifying schema changes, and we have asked twice.

observed_schema:
  tracking_number: {type: string, present_pct: 100.0}
  status:          {type: string, present_pct: 100.0,
                    observed_values: [in_transit, out_for_delivery,
                                      delivered, exception, returned]}
  updated_at:      {type: string, present_pct: 100.0,
                    note: "ISO-8601, NO OFFSET. We assume UTC. UNVERIFIED."}
  events:          {type: array,  present_pct: 98.6,
                    note: "absent for 1.4% of shipments, all recently created"}

observed_guarantees:                     # MEASURED, not promised
  freshness:
    p50_minutes: 12
    p99_minutes: 184
    max_observed_minutes: 511            # one Sunday
  volume:
    typical_records_per_day: 6412
    min_observed: 118                    # 2026-02-09, a Sunday
    max_observed: 9004
  completeness: "unknown -- we have no independent count to compare to"

what_we_do_about_it:
  - consumer-side assertions on every field above, in OUR ci
  - a volume band alert at +/- 40% (wide, because Sundays)
  - an enum assertion on `status`, generated from observed_values
  - raw JSON landed to bronze, so a schema change is REPLAYABLE (§16.23)

review: monthly, and after any assertion failure

The two fields that make this honest are status: OBSERVED and the note on updated_at.

OBSERVED prevents §17.19's failure — nobody on our side can mistake this for a promise.

And "we assume UTC, UNVERIFIED" is the single most valuable line in the file. An offsetless timestamp from an external carrier is exactly Chapter 34's Case Study 2, and writing down that the assumption is unverified is what makes it findable later. A contract that records what you do not know is more useful than one that quietly asserts what you guessed.

Part IV — Transformation

Chapter 18 — SQL Transformations

18.1 The three tells, and what each is usually replaceable by:

1. A loop issuing queries — one query per row, per customer, per day. Replaceable by a join or a GROUP BY, almost always, and the rewrite is usually shorter than the loop.
2. A correlated subquery in the SELECT list — "for each row, go and compute something about its neighbours." Replaceable by a window function, which computes it once per partition instead of once per row.
3. A self-join to compare a row with its neighbour — a.seq = b.seq - 1. Replaceable by LAG/LEAD, which needs one pass rather than two and one sort rather than a join.

The common shape: all three are "for each row, look at other rows." Set-based SQL has exactly two constructs for that — the join and the window — and every procedural tell is one of them, written the long way.

18.3 Because RANK assigns the same value to ties, so WHERE rn = 1 can return more than one row per key — which is precisely the outcome deduplication exists to prevent.

key   updated_at   RANK   DENSE_RANK   ROW_NUMBER
 88   10:00           1            1            1
 88   10:00           1            1            2      <- the duplicate
 88   09:00           3            2            3

RANK and DENSE_RANK both keep two rows for key 88. ROW_NUMBER is the only one of the three that guarantees exactly one, because it is defined to be distinct within a partition.

And the corollary is Exercise 18.20's: ROW_NUMBER guarantees one row, not which row. With a tie in the ORDER BY, it picks arbitrarily and non-deterministically — which is why the tiebreaker is not optional (§18.7).

18.5 LAST_VALUE returns the current row under the default frame because the default frame is RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW. The window ends at the current row, so the last value in the window is the current row's. The function is behaving exactly as specified; the specification is not what anyone expects.

Two fixes:

-- 1. state the frame explicitly
LAST_VALUE(status) OVER (PARTITION BY order_id ORDER BY updated_at
                         ROWS BETWEEN UNBOUNDED PRECEDING
                                  AND UNBOUNDED FOLLOWING)

-- 2. use FIRST_VALUE with a reversed ORDER BY -- the default frame is
--    now harmless, because the frame START is what you want
FIRST_VALUE(status) OVER (PARTITION BY order_id ORDER BY updated_at DESC)

The second is the one to prefer in code review, because it is correct even if someone later edits the frame clause out.

What it returns when the ORDER BY has ties, and this is the part almost nobody knows: the last value among the current row's peer group. A RANGE frame is defined in terms of values, not rows, so it includes every row that ties with the current one on the ORDER BY key — the frame ends at the end of the peer group, not at the current row.

updated_at   status      LAST_VALUE(status) over the DEFAULT frame
10:00        'paid'      'picked'     <- the peer group's last, not this row's
10:00        'picked'    'picked'
11:00        'shipped'   'shipped'

So the function returns neither the current row nor the partition's last, and which peer is "last" within the tie is itself arbitrary. Two layers of surprise stacked on one default, which is why the frame clause is worth writing every time.

18.7 UNION ALL because a recursive CTE's recursive term is defined over the rows produced by the previous iteration, and UNION (distinct) requires materialising and deduplicating the entire accumulated result at every step — quadratic work, and most engines simply reject UNION in a recursive CTE for that reason.

It is tempting to reach for UNION as cycle protection, and that is the trap: deduplication would terminate a cycle, silently, by producing a truncated result that looks complete. You want a cycle to be detected, not absorbed.

What the depth guard converts an infinite loop into: a finite, bounded, testable result.

WITH RECURSIVE chain AS (
    SELECT id, parent_id, 1 AS depth FROM parts WHERE id = :root
    UNION ALL
    SELECT p.id, p.parent_id, c.depth + 1
      FROM parts p JOIN chain c ON p.parent_id = c.id
     WHERE c.depth < 20                       -- the guard
)
SELECT * FROM chain;
-- and then the assertion that makes the guard useful:
SELECT count(*) FROM chain WHERE depth >= 20;   -- expect zero rows

The guard alone turns a hang into a wrong answer, which is worse. The guard plus the assertion turns it into a failed build with a specific diagnosis: "something in this graph is 20 deep or is a cycle." Never ship the guard without the assertion.

18.9 Flag the rows that start an island, take a running sum of that flag as a group id, and GROUP BY it.

SUM(is_island_start) OVER (PARTITION BY key ORDER BY seq
                           ROWS UNBOUNDED PRECEDING) AS island_id

Everything else — sessionisation, consecutive-day streaks, contiguous status runs, gap detection — is that sentence with a different definition of is_island_start.

18.12 (Measurement.)

SELECT order_date, order_id, net_revenue_cents,
       SUM(net_revenue_cents) OVER (ORDER BY order_date)                     AS range_default,
       SUM(net_revenue_cents) OVER (ORDER BY order_date
                                    ROWS BETWEEN UNBOUNDED PRECEDING
                                             AND CURRENT ROW)                AS rows_frame
  FROM fct_order
 WHERE order_date BETWEEN '2026-11-13' AND '2026-11-15'
 ORDER BY order_date, order_id;

order_date   order_id   net_cents   range_default   rows_frame
2026-11-13     88214        4,250           4,250        4,250
2026-11-14     88215        3,100          10,050        7,350   <- differ
2026-11-14     88216        2,700          10,050       10,050
2026-11-15     88217        1,900          11,950       11,950

Two orders share 2026-11-14, and under the default RANGE frame both rows show the day's completed total. RANGE is defined over values: every row tied on order_date is a peer, so the frame includes all of them regardless of which row you are on.

ROWS counts rows, so it produces the running total everyone means by "running total."

The reason this is dangerous rather than merely surprising: with no ties the two are identical, so the bug is invisible in development, in a small fixture, and on any day with one order per key. It appears the first time two rows share a sort value in production, which for a date column is immediately and for a timestamp column is eventually.

18.14 (Implementation.) The four patterns, and the one that is wrong:

pattern                                   rows      correct?
1. ROW_NUMBER + QUALIFY, with tiebreak   412,006    YES
2. DISTINCT ON (Postgres/DuckDB)         412,006    YES
3. GROUP BY key + MAX(each column)       412,006    NO   <- see below
4. self-join to MAX(lsn) per key         412,006    yes, and slow

Every one returns the same row count, which is exactly why pattern 3 survives in codebases.

What is wrong with pattern 3:

SELECT order_id, MAX(lsn) AS lsn, MAX(status) AS status, MAX(placed_at) AS placed_at
  FROM bronze.orders_cdc GROUP BY order_id;

MAX is applied per column, independently. For order 88214 with two CDC events:

lsn        status       placed_at
88,102     'paid'       2026-11-14 09:12:00
88,410     'cancelled'  2026-11-14 09:12:00

MAX(lsn) = 88,410   MAX(status) = 'paid'   <- alphabetically! from the OTHER row

The output row never existed. It has the newer row's lsn and the older row's status, because 'paid' > 'cancelled' alphabetically. The count is right, the grain is right, every column is a real value, and the row is fiction.

No uniqueness test catches this, which is Chapter 38's Case Study 2 in miniature: the assertion everyone writes checks the property that is not broken.

18.16 (Measurement.)

sessions under 30 seconds, starting within 5 minutes of the window start

  without overlap    1,847
  with overlap          12
                     ─────
  artifact           1,835 phantom sessions per boundary

1,835 of 1,847 were artifacts — session tails whose beginnings were in data the batch had not loaded. At one boundary a day, that is roughly 670,000 phantom sessions a year, every one of them short, every one of them looking like a bounce.

And the reason it survives is that it looks like real user behaviour. Short sessions clustered at the start of a window are indistinguishable from short sessions generally, unless you go looking at the distribution by time-since-window-start — which nobody does, because there is no reason to unless you already suspect this.

The twelve remaining are probably genuine, and it is worth not chasing them to zero: a real bounce exists, and a fix that produces exactly zero has almost certainly filtered something it should have kept.

18.18 (Implementation.)

What happened: nothing. The fifth channel does not appear. The pivot's column list was written into the SQL:

SUM(CASE WHEN channel = 'web'       THEN net_cents ELSE 0 END) AS web,
SUM(CASE WHEN channel = 'ios'       THEN net_cents ELSE 0 END) AS ios,
SUM(CASE WHEN channel = 'android'   THEN net_cents ELSE 0 END) AS android,
SUM(CASE WHEN channel = 'wholesale' THEN net_cents ELSE 0 END) AS wholesale

What breaks downstream, and it is worse than a missing column. The four columns still sum to less than the table's total, so every "revenue by channel" figure is now silently incomplete. A dashboard summing the four gets a number lower than the same dashboard's own total — and the two disagree by exactly the new channel, which nobody is looking for.

A missing column would be loud. A silently incomplete decomposition is quiet, and it is the shape of the bug.

The fix, in three parts:

1. Generate the pivot rather than writing it — Jinja over the values, or the engine's PIVOT where it has one, so a new channel produces a new column automatically.

2. Add a residual column and assert it is zero:

SUM(CASE WHEN channel NOT IN ('web','ios','android','wholesale')
         THEN net_cents ELSE 0 END) AS unclassified_cents
-- test: unclassified_cents = 0

3. And assert that the parts sum to the whole, which catches the class rather than the instance:

SELECT * FROM pivoted
 WHERE web + ios + android + wholesale <> total_cents;
-- expect zero rows

Part 2 is the one to ship first — it is one line, it needs no generation, and a residual bucket converts a silent omission into a failing test on the day the fifth channel appears. This is §17.14's enum assertion, in pivot form, and the generalisation is the same: assert against the declared set, and give the undeclared somewhere visible to land.

18.20 The test must not depend on running twice and hoping, so it cannot test the output. It tests the ordering specification instead.

-- Are there ties under the dedup's ORDER BY key(s) alone?
-- If yes, the tiebreaker is doing real work and its absence is a bug.
SELECT order_id, lsn, count(*) AS tied_rows
  FROM bronze.orders_cdc
 GROUP BY order_id, lsn
HAVING count(*) > 1;
-- Non-zero here means: without a unique tiebreaker, this dedup is
-- non-deterministic. It does not mean the current run was wrong.

And the stronger, static form, which is what Chapter 38's audit uses:

# fails if any ROW_NUMBER/RANK in the project lacks a known-unique
# column as the last ORDER BY term
WINDOW = re.compile(r"(ROW_NUMBER|RANK|DENSE_RANK)\s*\(\s*\)\s*OVER\s*\("
                    r"[^)]*ORDER\s+BY\s+([^)]*)\)", re.I | re.S)
for fn, order_by in WINDOW.findall(sql):
    last = order_by.split(",")[-1].strip().split()[0]
    assert last in UNIQUE_COLUMNS, "no unique tiebreak in %s: %s" % (fn, order_by)

Why it must fail when the tiebreaker is missing and pass when present: the first query returns rows in both cases if ties exist in the data, so it is a test of the data, not of the code. The second is a test of the code and is the one that satisfies the exercise: it fails on the SQL text regardless of what today's data happens to contain.

What I am assuming about the engine, and it should be stated because the whole exercise rests on it: that ROW_NUMBER with an incomplete ORDER BY may return any of the tied rows, and that the choice may differ between runs, between engine versions, and between physical layouts of the same data. No engine documents a stable choice, and several explicitly document that it is undefined. A test that runs the query twice on the same warm cache will usually get the same answer — which is exactly why "run it twice and diff" is not a test.

18.22 It is not correct in every case, and the failure is worth knowing because it is a data quality bug that produces plausible output.

MAX(customer_id) over a session works when a session contains at most one identity. It breaks when a session contains two.

The scenario: a shared device — a household tablet, a shop-floor terminal, a library computer. One person browses and logs in; they log out; a second person logs in within the 30-minute inactivity window. The sessioniser sees one session (there was no 30-minute gap) with two customer_id values, and MAX picks the numerically larger one.

The result: the first person's entire browsing history is attributed to the second person, including the purchase intent that drove it. In a recommendation feature store (Chapter 32) that is a wrong training label; in a marketing attribution model it is a wrong conversion.

And MAX picks by customer id, which is an arbitrary number — so the attribution is decided by who registered first, which is a property with no meaning at all.

A better rule, in increasing order of correctness:

1. Use the last observed identity, not the maximum:

LAST_VALUE(customer_id IGNORE NULLS) OVER (
    PARTITION BY anonymous_id, session_seq ORDER BY event_ts
    ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)

Still one identity per session, and at least the choice is meaningful.

2. Split the session on an identity change. Add customer_id <> LAG(customer_id) (ignoring nulls) to §18.9's is_session_start expression. A logout-login is a session boundary, which is what it actually is.

3. And flag rather than hide. Emit n_distinct_identities on every session and assert it is 1. Sessions where it is not are a real phenomenon — shared devices exist — and the pipeline should know how many there are rather than silently resolving them.

I would ship 2 and 3 together. 2 fixes the attribution; 3 tells you whether 2 mattered, which is the number you need in order to know whether the fix was worth shipping.

18.24 (Implementation.) The assertion the exercise singles out:

-- no session may exceed a plausible maximum
SELECT session_id, started_at, ended_at, duration_seconds
  FROM silver.sessions
 WHERE duration_seconds > 86400        -- 24 hours. See below.

Writing down "plausible maximum" is the exercise, and here is the reasoning.

The definition already bounds it. §18.9's rule is 30 minutes of inactivity or midnight UTC, whichever comes first. A session therefore cannot exceed 24 hours, ever, by construction — so anything above 86,400 seconds is not an implausible user, it is a bug in the sessioniser or a corrupt timestamp. That is a much stronger assertion than a judgment call, and it is available because the definition is written down.

A second, softer threshold is still worth having. A session of 11 hours is possible under the rule and is almost certainly a device left open rather than a person. Assert at 24 hours as an error and report the p99.9 as a warning, because the first is a logic bug and the second is a data characteristic somebody should decide about.

And the midnight assertion is the sharper of the two:

SELECT count(*) FROM silver.sessions
 WHERE started_at::date <> ended_at::date;      -- expect 0, by the rule

That one cannot be argued with. It is the definition, expressed as SQL, and it fails the moment somebody changes the sessionisation without changing the definition — which is the failure worth catching.

Chapter 19 — dbt and Analytics Engineering

19.1 dbt is three things: a templating engine (Jinja-rendered SQL, compiled to target/), a dependency graph (derived from ref() and source()), and a test runner (assertions beside the models, in the same command).

Four things it is not:

The distinction that matters most is the third, because it is the one teams get wrong in a way that compounds: dbt build runs a DAG of dbt models, once, when invoked. Something else must decide when to invoke it and how it relates to the twelve things that are not dbt — and a team that treats dbt as its orchestrator ends up with a scheduler that can only see a third of its platform.

19.3 Four consequences of writing analytics.stg_orders instead of {{ ref('stg_orders') }}:

1. No DAG edge. dbt does not know this model depends on that one, so build order is no longer guaranteed — the model can be built before its upstream, reading yesterday's data, successfully.
2. No environment resolution. ref() renders to the target's schema; a literal name does not. A developer running on a laptop reads production, which is Case Study 1's four-month defect.
3. Invisible to node selection. state:modified+, +model, model+ and slim CI all traverse the DAG. A model outside it is not selected when its upstream changes, so CI passes on a change that breaks it.
4. Invisible to lineage, docs, and exposures. The dependency does not appear in dbt docs, so the blast-radius question is answered wrongly by every tool and every human who consults them.

The one that disables the check that would have caught it is the first. Consequences 3 and 4 are derived from the DAG, so removing the edge removes the mechanism by which anything else could have noticed. The defect and the detector are destroyed by the same character.

19.5 Two costs of ephemeral:

It is inlined into every consumer. An ephemeral model referenced by four downstream models is compiled as a CTE into all four, so its work is done four times per build, not once.

It produces no object, so it cannot be tested directly, cannot be queried for debugging, and does not appear in the warehouse's object list — which means "what is this CTE and where did it come from" is answerable only by reading the compiled SQL.

The invisible-in-a-cost-report one is the first. Because there is no object and no separate query, the duplicated compute is billed under the four consuming models' names. A cost report by model shows four models that are each slightly more expensive than expected, with no line item explaining why, and no amount of staring at the four models reveals a shared cause. The cost is real, it is attributed, and it is attributed to the wrong thing — which is the worst of the three possible states.

19.7 Three ways a test becomes decorative:

1. It is tautological. not_null on a column defined as coalesce(x, 0); accepted_values on a column produced by a CASE that can only emit those values. The SQL guarantees the assertion, so it can never fail.

2. Its severity is warn and nobody reads warnings. The build is green, the warning scrolls past in CI, and the test has become a comment.

3. It is not run. The model was excluded by a selector, the test was disabled with enabled: false, or it lives on a model that is no longer built. It appears in the project and in every audit that counts tests.

The one question that identifies all three: "what would have to be true in the data for this to fail, and could that ever actually happen?"

Tautological tests have no answer. Warn-only tests have an answer and no consequence. Unrun tests have an answer and no execution. And the question is answerable in about ten seconds per test, which is why it is a better audit than any tooling — though Exercise 23.20's automated version scales further.

19.9

dbt build --select "tag:a,tag:b"     COMMA  = AND = INTERSECTION
dbt build --select tag:a tag:b       SPACE  = OR  = UNION

What the wrong one does to a scheduled job, and it is asymmetric.

Meaning union and writing the comma is the dangerous direction. The intersection of two tags is frequently empty, and dbt builds zero models, prints a warning, and exits 0. The scheduled job succeeds. Every downstream freshness check that depends on the models is now measuring a table nobody rebuilt, and — because nothing failed — the first symptom is a stale dashboard days later.

Meaning intersection and writing the space is merely wasteful: you build the union, which is more models than intended, more expensive, and correct.

The control worth adding to any scheduled dbt invocation:

dbt ls --select "$SELECTOR" | grep -q . || {
    echo "SELECTOR '$SELECTOR' matched no models. Refusing to report success."
    exit 1
}

"Zero models is a failure" is a policy you have to state, because dbt's own view — that it did exactly what it was asked — is defensible and unhelpful.

19.12 (Measurement.)

mart: gold.daily_revenue (365 rows out of 6.5M source rows)

materialization      build     object created      query
view                 0.4 s     a view              2.81 s
table                8.2 s     a table             0.04 s
ephemeral            0.0 s     nothing             2.79 s
materialized_view    9.1 s     an MV (auto-refresh) 0.05 s

The crossover:

table:  8.2 + 0.04q
view:   0.4 + 2.81q

8.2 + 0.04q  <  0.4 + 2.81q
       7.8   <  2.77q
         q   >  2.82

Three queries per build. Above that, table is cheaper; below it, view is.

Two things the arithmetic does not say, and they usually decide it.

A dashboard is not one query. Twenty-two dashboards refreshing hourly is 528 queries a day against one build — three orders of magnitude past the crossover, and the decision is not close.

And latency is not cost. A view that takes 2.81 seconds is cheap and slow, and a human waiting on a dashboard does not care which. The crossover computed in seconds and the crossover computed in patience are different numbers, and for anything a person looks at, the second one wins well before three queries.

19.14 (Implementation.)

tests:
  - dbt_utils.expression_is_true:
      expression: "(SELECT count(*) FROM {{ this }}) >= 5800"
      config:
        error_if: ">0"

Defending 5,800. Kestrel's fct_order receives about 6,575 orders on an average day. The observed minimum over 18 months is 5,912 — a Christmas Day. 5,800 sits just below the observed floor, so it fires on any day that is materially below anything that has ever happened.

What would make it fire falsely: a genuine business event — a site outage, a payment-provider failure, a holiday the calendar has not seen yet. All three are things you want to be told about, so a false positive here is closer to a true positive about something else. That asymmetry is what makes a volume floor a good first test.

What it fails to catch, and this is the honest half:

Anything above the floor. A day that loads 6,000 of 6,575 orders — 8.7% missing — passes. The floor catches catastrophe, not erosion, and erosion is the more common failure (Chapter 20's Case Study 2 lost rows for eight months, well inside any floor).

And it cannot catch too many rows, which is Chapter 1's duplicate-rows incident. A floor is half a test; the ceiling is the other half and is the one Chapter 30's Case Study 1 argues for.

The stronger version, once the floor has earned its place: compare to a trailing median rather than a constant, with a band — count(*) BETWEEN 0.75 * median_28d AND 1.30 * median_28d. It adapts to growth, and it needs a Black Friday exemption, which is a maintenance cost the constant does not have. Start with the constant.

19.16

# GENERIC -- reusable, declarative, cheap
columns:
  - name: order_id
    tests: [unique, not_null]

-- SINGULAR -- one specific claim, arbitrary SQL, returns failing rows
-- tests/assert_revenue_reconciles_to_source.sql
SELECT g.revenue_date, g.revenue_cents, s.revenue_cents AS source_cents
  FROM {{ ref('daily_revenue') }} g
  JOIN {{ source('kestrel_raw','source_daily_totals') }} s USING (revenue_date)
 WHERE g.revenue_cents <> s.revenue_cents

# UNIT -- fixed input, expected output, no warehouse data at all
unit_tests:
  - name: gift_cards_excluded_from_revenue
    model: fct_order_line
    given:
      - input: ref('stg_order_lines')
        rows:
          - {order_id: 1, sku: 'KS-1',  cents: 2500}
          - {order_id: 1, sku: 'GC-10', cents: 5000}
    expect:
      rows: [{order_id: 1, revenue_cents: 2500}]

What each catches that the others cannot:

The generic test catches a property across every row of real data, cheaply and uniformly. Neither of the others scales to "every row of every model" — a singular test is written per claim, and a unit test never touches real data at all.

The singular test catches a relationship between two objects, including objects outside dbt. Only it can reconcile against a source system, which is the assertion Chapter 1 §1.7 calls the acceptance criterion.

The unit test catches a logic error with no data present. It runs in CI in milliseconds, before anything is built, and it fails on a rule change even when today's data happens not to contain a gift card. Neither of the others can test a code path that the current data does not exercise — which is precisely the path that breaks at month end.

The three map onto a useful ordering: unit tests for logic, generic tests for shape, singular tests for truth.

19.18

-- models/marts/order_status_pivot.sql
SELECT order_date,
{% for s in ['pending','paid','shipped','delivered','cancelled'] %}
  COUNT(*) FILTER (WHERE status = '{{ s }}') AS n_{{ s }}{{ "," if not loop.last }}
{% endfor %}
  FROM {{ ref('fct_order') }}
 GROUP BY 1

-- target/compiled/kestrel/models/marts/order_status_pivot.sql
SELECT order_date,

  COUNT(*) FILTER (WHERE status = 'pending') AS n_pending,

  COUNT(*) FILTER (WHERE status = 'paid') AS n_paid,

  COUNT(*) FILTER (WHERE status = 'shipped') AS n_shipped,

  COUNT(*) FILTER (WHERE status = 'delivered') AS n_delivered,

  COUNT(*) FILTER (WHERE status = 'cancelled') AS n_cancelled

  FROM analytics.fct_order
 GROUP BY 1

§19.6's test: could a SQL developer who does not know Jinja predict the output? Yes — just.

The loop is over a literal list on one visible line, the body is one line, and loop.last is self-explanatory in context. A reader who has never seen Jinja can see five names and guess five columns, and they would be right.

Two observations worth making about where this stops being true.

The blank lines are the tell that whitespace control is missing ({%- for -%}), and on a twenty-column pivot the compiled output becomes genuinely hard to read — which matters because the compiled SQL is what you debug.

And the test fails the moment the list stops being literal. {% for s in var('statuses') %} or a run_query to fetch the values from the warehouse is where a SQL reader loses the ability to predict the output, and where the model stops being reviewable by the people who own the business logic. That, rather than any line count, is the boundary §19.6 is drawing.

19.20 The strongest case for a tautological assertion: it is not testing today's SQL, it is testing tomorrow's.

not_null on a column currently defined as coalesce(x, 0) cannot fail. It can fail after somebody removes the coalesce — which is a one-character change in a refactor, made by someone who did not know the downstream consumer breaks on nulls. The test is a guard on the invariant, not on the current implementation, and invariants outlive implementations.

Three specific things worth protecting this way:

A contract with a downstream consumer. If a BI tool or an exported feed requires non-null, the assertion documents and enforces that requirement in the place where it can be violated. Deleting it because "the SQL guarantees it" moves the guarantee into code nobody has annotated.

A property that is currently guaranteed by an accident. A column is unique today because the source happens to be, not because anything enforces it. The assertion is how you find out when the accident ends — and it will, silently.

A property the engine guarantees on one adapter and not another. A project that may migrate warehouses (Chapter 37) is relying on behaviour that is not portable.

Where the case fails, and it is worth conceding cleanly. A tautology whose invariant nobody depends on is pure cost — runtime plus, much more importantly, attention, because a project with 400 tests of which 200 cannot fail teaches its team that tests are furniture (§23.10). The distinguishing question is not "can it fail" but "if it failed, would anyone care" — and that reframing keeps the useful tautologies and deletes the rest.

19.22 Detecting the general class — a defect that produces no observable output — by looking for the absence of expected artefacts rather than the presence of wrong ones.

# 1. Every model should be built by SOMETHING, on a schedule.
#    Compare the project's node list against the models the last 7 days
#    of production runs actually built.
never_built = set(manifest_models) - set(models_built_last_7d)

# 2. Every model should be READ by something, or be an exposure.
#    Warehouse query logs -> tables referenced -> compare to the DAG.
never_read = set(models_built) - set(tables_queried_last_30d) - exposures

# 3. Every ref-able name that appears as a LITERAL anywhere in the project.
hardcoded = [m for m in manifest_models
             if re.search(r"\b(analytics|prod)\.%s\b" % m, all_sql)]

# 4. Every test that has never failed AND never could (section 19.7).
#    Combine: zero historical failures + a tautology check.

Check 3 is the one that catches Case Study 1's specific defect, and it is a five-line grep that would have run in CI from day one.

What this cannot cover, honestly, and it is a lot:

It cannot see defects whose output is wrong rather than absent. A model that builds, is read, uses ref() correctly, and computes the wrong number passes every check above. That is the entire subject of Chapter 23, and no artefact audit reaches it.

It cannot see a correct artefact that is not doing its job — Exercise 23.20's five: the test that cannot fail, the threshold that cannot fire, the alert to a dead channel. Check 4 gestures at the first of those and does not cover the rest.

And it produces false positives that erode it. A model built quarterly appears in never_built; a mart read only at year end appears in never_read. Every exception needs an annotation with a reason and an expiry, or the report becomes noise within two months and joins the class of things it was built to detect.

19.24 (Implementation.)

def unfireable(source, observed_interval_hours):
    """A freshness threshold that cannot fire is worse than none."""
    err = to_hours(source["freshness"]["error_after"])
    if err is None:
        return "no error_after configured"
    if err > observed_interval_hours * 3:
        return ("error_after=%.0fh but the source loads every ~%.1fh; "
                "this can only fire after %.0f missed loads"
                % (err, observed_interval_hours, err / observed_interval_hours))
    return None

source                  configured        observed load    verdict
kestrel_raw.orders      error_after 7d    hourly           UNFIREABLE (168 missed loads)
kestrel_raw.customers   error_after 6h    daily            TOO TIGHT (fires every night)
kestrel_raw.events      error_after 4h    continuous       ok
supplier_b.feed         (none)            weekly           MISSING

Why this is the exercise most worth doing carefully, and the sentence is worth restating precisely: a threshold that cannot fire appears on every audit as present. Someone asks "do we monitor freshness on orders?" and the answer is yes, with a YAML block to point at. The artefact exists, the count is right, the coverage report is green, and the check has never been capable of firing.

Note the second row is a different bug and equally worth catching. A threshold tighter than the cadence fires every single night, gets muted within a week, and becomes §23.16's oldest mute. Both directions produce a check that does nothing; only one of them is loud about it.

Chapter 20 — Incremental Processing and SCD

20.1 The rule of thumb: go incremental when a full refresh no longer fits the window you have, and not before.

The two numbers from §20.1 that give opposite answers in one project: a model whose full refresh takes 4 seconds and one whose full refresh takes 90 minutes against a 15-minute SLA window. Same technique, same project, and the correct decision is opposite — the first should never be incremental, because incrementality buys 4 seconds and costs a class of correctness bug; the second has no choice.

The transferable form: incremental processing is not an optimisation, it is a trade — you exchange compute for a stateful system with lookback windows, late arrivals, and non-idempotent write paths. Make the trade when compute is the binding constraint, and never for tidiness.

20.3 The four strategies: partition overwrite (delete + insert the window), merge/upsert on a unique key, insert-overwrite of whole partitions, and deduplicate-on-read.

The single question that selects between them: what is the smallest unit you can rewrite completely?

a date partition        -> partition overwrite. Simplest, and the default.
a single row by key     -> merge on that key.
a whole table           -> full refresh; you are not incremental.
nothing                 -> deduplicate-on-read.

If the answer is "I cannot rewrite anything completely," you have not chosen a strategy — you have deferred the problem to every consumer. Deduplicate-on-read means every query against the table must apply the dedup, forever; the one query that forgets returns duplicates and looks right. It is a legitimate answer for an append-only landing zone (bronze) and a bad one anywhere a consumer reads directly, because correctness now depends on the discipline of people who have never read this chapter.

20.5 --full-refresh drops the target table and rebuilds it from scratch, ignoring is_incremental() entirely.

Three consequences:

The table does not exist during the rebuild. Queries against it fail or return nothing for the duration — which on a large fact table is not seconds.

Anything present only in the target is destroyed. Rows outside the source's retention window, manual corrections, and historic data whose source has since been purged. The rebuild is a pure function of the source as it is now, and the target may know things the source has forgotten.

There is no dry run and no undo. The drop happens first.

The safer four-step procedure:

1. BUILD into a new relation, not over the old one.
     dbt build --select fct_order_item --full-refresh \
               --target prod_rebuild        # a separate schema
2. TEST it. Every test the model has, against the new relation.
3. RECONCILE against the current one:
     row counts, the revenue sum, and a per-day diff.
     Investigate every day where they differ -- including days where the
     NEW one is higher, which is the direction people skip.
4. SWAP atomically -- a rename, or a view flip -- and KEEP the old
   relation for a retention window before dropping it.

Step 3 is the one that earns the procedure. A full refresh that produces a different answer from the incremental table is telling you something important, and --full-refresh in place destroys the evidence before you can look at it. The differences are the finding, and Chapter 20's Case Study 2 is a case where they would have been visible eight months earlier.

20.7 Size it from the observed maximum, plus margin — and add an assertion that fires when a new maximum exceeds it.

The defence in one sentence: the cost of an over-wide lookback is compute, which is linear, small, and visible; the cost of an under-wide one is silent, permanent row loss, which is unbounded and invisible — so the asymmetry says buy margin, and the maximum is a statistic that grows, so the margin must be monitored rather than chosen once.

-- the assertion that makes the choice safe
SELECT max(date_diff('day', event_date, _ingested_at)) AS worst_lateness
  FROM bronze.orders_raw
 WHERE _ingested_at > current_date - 30;
-- alert when worst_lateness > 0.7 * configured_lookback_days

p99 is the wrong basis for a reason worth stating: at 6,575 orders a day, p99 lateness leaves 66 orders a day outside the window — 24,000 orders a year, lost silently. A percentile is the right tool for a latency target and the wrong tool for a completeness guarantee.

20.9 The three SCD2 invariants:

1. Exactly one current row per natural key (is_current = true).
2. No overlapping validity intervals for a key.
3. No gaps — the intervals for a key are contiguous from first appearance to valid_to infinity.

Overlap is the one that multiplies your revenue. A point-in-time join from a fact to the dimension matches every dimension row whose interval contains the fact's timestamp. With two overlapping rows, every fact matches twice — the fact table's row count doubles, and every additive measure with it.

And the failure has a specific, misleading signature. The dimension's own tests pass (it is a dimension; nobody counts its rows against a source). The fact table's uniqueness test passes if it runs before the join. The number moves in the report, which is where nobody has an assertion, and the first symptom is somebody saying revenue looks high.

Invariant 1 is the loud one (a unique test on where is_current catches it immediately) and invariant 3 is the quiet one (a gap means a point-in-time join finds no row and the fact silently drops out of the join, which is the opposite failure and just as invisible).

20.12 (Implementation.) The diff in both directions is the point:

-- run the load twice, then:
SELECT 'in_second_not_first' AS side, * FROM run2 EXCEPT SELECT 'in_second_not_first', * FROM run1
UNION ALL
SELECT 'in_first_not_second', * FROM run1 EXCEPT SELECT 'in_first_not_second', * FROM run2;
-- expect zero rows from BOTH

strategy               run1 rows   run2 rows   diff(2-1)   diff(1-2)   idempotent?
append                   412,006     824,012     412,006           0   NO
delete + insert          412,006     412,006           0           0   yes
merge on unique_key      412,006     412,006           0           0   yes
insert_overwrite         412,006     412,006           0           0   yes

Running EXCEPT in one direction only is the mistake this exercise exists to prevent. A load that loses rows on the second run shows zero in the run2 EXCEPT run1 direction — which looks like a pass. A row count comparison catches that one and misses a row whose values changed while the count stayed the same. Both directions of EXCEPT, on the whole row, catches all three failure shapes.

20.14 (Measurement.)

-- completeness: every source row must be in the fact table
SELECT count(*) AS missing
  FROM silver.order_items s
  LEFT JOIN gold.fct_order_item f USING (order_item_id)
 WHERE s.order_date >= '2026-01-01'
   AND f.order_item_id IS NULL;

missing: 1,847

Reporting it honestly includes reporting it when it is zero, and the reason is that zero is the result you will get on the first run of a model built last week — which tells you the test works and nothing about whether it was needed.

What I would have concluded if I had never run it: that the model was fine, because everything else said so. The build was green, the uniqueness test passed, the row count grew every day, and the revenue figure was plausible. Nothing in the platform disagreed with the model except this query, and the query had to be written before it could disagree.

The 1,847 rows here are the lookback-window failure (§20.7): source rows that arrived more than three days after their order_date and were never inside any incremental run's window. They are not recoverable by waiting; only a rebuild or a targeted backfill brings them in, which is why the tolerance on this test should be zero and why it should have existed on day one.

20.16 (Implementation.) Breaking each invariant, and the one with a number attached:

invariant broken            assertion that fired            PIT join fan-out
1. two current rows         unique(customer_id) where       1.0x  (no fan-out;
   for one key              is_current                       the join picks one)
2. overlapping intervals    no_overlaps(customer_id,         2.7x  <-- !!
                            valid_from, valid_to)
3. a gap in coverage        no_gaps(customer_id)             0.94x (rows DROP OUT)

The fan-out on invariant 2 is the number worth recording: 2.7×. 412,006 fact rows became 1,112,416 after a point-in-time join to a dimension with overlaps on 63% of keys — and total revenue went from $21.9M to $59.2M.

Note that invariants 1 and 3 fail in the two directions the fan-out does not. A duplicate current row does not multiply a point-in-time join (which filters by interval, not by is_current), and a gap removes facts from the result — a 0.94× "fan-out" that reads as a small, plausible decline rather than as an error.

Three invariants, three completely different symptoms, and only one of them looks like a bug.

20.18 (Measurement.)

{{ config(materialized='incremental', unique_key='order_item_id',
          incremental_strategy='merge',
          incremental_predicates=["DBT_INTERNAL_DEST.order_date >= "
                                  "dateadd(day, -3, current_date)"]) }}

                          elapsed   TARGET rows scanned   partitions scanned
without predicates          184 s          412,006,004                 1,095
with incremental_predicates  11 s            1,238,102                     4

Report the rows scanned in the target, not the elapsed time, because the elapsed time is the consequence and the scan is the cause — and because elapsed time is confounded by warehouse size, concurrency, and cache while the scan count is not.

The mechanism: a MERGE without predicates must consider every row of the destination as a potential match. The source side is already restricted by the model's is_incremental() filter; the destination side is not, and nothing in the model's SQL restricts it. incremental_predicates adds a WHERE to the destination side of the merge condition.

And there is a correctness trap that comes with the speedup. The predicate must be at least as wide as the source filter. If the model looks back 3 days and the predicate looks back 1, a row arriving 2 days late is not matched, so the merge inserts it — as a duplicate. The two windows must be derived from one variable, and the fact that they are separate strings in separate config keys is exactly the kind of thing Chapter 38's audit finds.

20.20 The nightly rebind:

-- facts pointing at the unknown member, whose dimension row has since arrived
UPDATE gold.fct_order_item f
   SET customer_sk = d.customer_sk
  FROM gold.dim_customer d
 WHERE f.customer_sk = -1
   AND d.customer_id = f.source_customer_id
   AND f.order_ts >= d.valid_from AND f.order_ts < d.valid_to;

Where it is wrong: when the unknown member is the correct historical answer, not a placeholder.

The scenario. A guest checkout. At the time of the order there was genuinely no customer — the person had no account. Three weeks later they register with the same email, and an identity-resolution job links the historical order to the new customer_id.

The rebind now attributes a purchase to a customer who did not exist when it happened. Every "orders in the first 30 days after signup" cohort analysis is wrong; every "new versus returning customer" split moves; and a report run in January and re-run in March gives different answers for the same historical month. The fact table has stopped being a record of what happened.

How to distinguish the two cases — and it cannot be done from the data as modelled, which is the finding. Both look like customer_sk = -1 followed by a matching dimension row appearing.

The distinction has to be recorded at write time, and it is a distinction between two different reasons for the unknown member:

customer_sk = -1   -- LATE: the customer exists, the dimension has not
                   --       caught up. Rebind.
customer_sk = -2   -- ABSENT: there was no customer. Do NOT rebind, ever.

Guest checkouts get -2 at load time, because the source knows: the order has no customer_id at all, as opposed to a customer_id that does not yet resolve. A null foreign key and an unresolved foreign key are different facts, and collapsing them into one sentinel destroys the information the rebind needs.

The general lesson is bigger than SCD: a single sentinel for "no value" that covers several distinct reasons is a lossy encoding, and the loss surfaces later as a decision nobody can make. Two sentinels cost nothing at write time and are unrecoverable afterwards.

20.22 The systematic approach: find every place a downstream aggregate could be masking a grain violation, and test the grain directly at each one.

-- 1. Find the maskers: aggregates that are duplicate-insensitive.
--    COUNT(DISTINCT x), MAX, MIN, and any GROUP BY on the full key.
--    Scan the BI layer's SQL and the dbt project for these.
grep -rniE "count\s*\(\s*distinct|(\bmax|\bmin)\s*\(" models/ bi_exports/

-- 2. For every model feeding one, assert the grain DIRECTLY.
SELECT <grain columns>, count(*) FROM <model> GROUP BY 1,2 HAVING count(*) > 1;

-- 3. And the inverse: for every model, list the aggregates applied to it
--    downstream. A model consumed ONLY by duplicate-insensitive
--    aggregates has no natural detector and needs the direct test most.

Step 3 is the ranking that makes this actionable. A model consumed by SUM has a detector — someone notices the number is high. A model consumed only by COUNT(DISTINCT ...) has none, and it is the one to test first, precisely because nothing will ever complain.

Honest coverage, and it is limited in three ways:

It only covers grain violations. The general class — "a defect masked by a downstream accident" — includes wrong values masked by a rounding, missing rows masked by a filter that would have excluded them anyway, and a wrong join masked by a WHERE that happens to remove the fan-out. Grain is the tractable subset because it has a mechanical test.

The BI layer is usually not in the repository, so step 1 misses the aggregates that matter most. Extracting them from warehouse query logs is better and is a project of its own.

And it says nothing about the masking that has not happened yet. A report rewritten from COUNT(DISTINCT customer_id) to COUNT(*) next quarter turns a latent defect into a visible one, at a moment nobody connects to the rewrite. The point of testing the grain directly is that it does not depend on what any consumer happens to do, and that is the whole argument for the approach.

20.24 (Implementation.)

snapshot with check_cols: all, 14 simulated days, source has last_login_at

  customers                     1,904,221
  snapshot rows after 14 days   9,318,447
  new versions per day           ~529,000

Why every row it produced is true, and the dimension is nonetheless useless:

Every one of those 9.3 million rows is a correct record of a real change. last_login_at did move. The snapshot faithfully captured it, on the day it happened, with correct valid_from and valid_to. There is no bug. check_cols: all did exactly what it says.

The dimension is useless because a Type 2 dimension exists to answer one question — "what were this customer's attributes at the time of this fact?" — and the question is asked about attributes that mean something to the business: their region, their segment, their account status. Those change a few times a year. last_login_at changes daily, and it is not an attribute anyone slices by.

The consequences, in order of how much they hurt:

The point-in-time join gets 22× more expensive, because it must scan 22× the rows to find the version valid at a given moment.

The business-meaningful history is buried. Finding when a customer's region changed means filtering 9.3 million rows for the 4,000 that represent a real change — and every consumer must know which columns matter, which is knowledge that was supposed to live in the dimension.

And the dimension grows without bound at a rate set by login behaviour, so its size is a function of site traffic rather than of customer count. A capacity plan built on "1.9 million customers" is wrong by more than an order of magnitude within a year.

The fix is check_cols with four enumerated columns, and the rule that produces it is §6.20(c)'s: every Type 2 attribute must have one sentence naming the business question that requires its history. last_login_at has no such sentence — nobody asks "what was their last login as of March" — so it is Type 1, or a fact, or a column on a current-state table. Anything you cannot write the sentence for is not Type 2.

Chapter 21 — Distributed Processing with Spark

21.1 The bands: under ~10 GB, single machine (Chapter 22); ~10–200 GB, DuckDB or Polars on one large machine, or the warehouse; ~200 GB–10 TB, Spark or a warehouse; over ~10 TB, Spark or a warehouse designed for it.

Three non-size reasons, all legitimate: the data is already in a lake and the transformation is not expressible in SQL; you need a library that only exists on the JVM or only scales this way; and the team already runs Spark and a second engine is a second operational surface (Chapter 5 §5.1).

The third is the one people apologise for, and they should not. "We already run it" sounds like inertia and is a real engineering argument: a team of four operating Spark competently will produce a better outcome with Spark than with a technically superior tool nobody can debug at 3 a.m. The correct version of the argument names the alternative's cost explicitly — one more system to monitor, upgrade, secure, and staff — rather than mumbling about familiarity.

What is worth apologising for is the same argument made in the other direction: adopting Spark because it is what the reference architecture had, for 15 GB of data.

21.3 Narrow: each output partition depends on one input partition. No data moves. select, filter, withColumn, map, union.

Wide: each output partition depends on many input partitions. Data crosses the network. groupBy, join, distinct, orderBy, repartition, and every window function with a PARTITION BY.

A stage boundary is a shuffle boundary. Spark fuses consecutive narrow operations into a single pass over the data; a wide operation forces the pipeline to stop, write every partition to local disk, and have every executor fetch the pieces it needs over the network. A stage is the run of narrow work between two shuffles, which is why counting Exchange nodes counts the job's real cost.

21.5 The order:

1. Count the Exchange nodes. The job's cost in one number.
2. Find the join strategies. BroadcastHashJoin is cheap; SortMergeJoin is two shuffles.
3. Check PushedFilters on the scans. An empty list means you are reading everything.
4. Only then look at partition counts and memory.

Most people start at step 4, because it is the one with a knob attached. spark.sql.shuffle.partitions is a number you can change without understanding anything, and changing it produces a different runtime, which feels like progress. Steps 1 to 3 require reading a plan and then changing the query, which is slower to start and is where the factors of ten live.

21.7 coalesce merges adjacent partitions without a shuffle. repartition performs a full shuffle and produces even partitions.

What coalesce costs you is twofold, and the second is the one that surprises people.

Partitions can be very uneven, because merging adjacent partitions cannot balance them. Coalescing 200 partitions to 10 where one input partition holds 40% of the data produces one enormous task.

And it propagates its parallelism backwards. .coalesce(1) before a write does not only produce one output file — it makes the entire upstream narrow stage run with one task. A job that reads 300 GB, filters, and writes one file will read all 300 GB single-threaded, because there is no shuffle boundary between the read and the coalesce to stop the constraint travelling upstream.

"The job stopped using the cluster" is almost always this, and the fix is repartition(1) — which costs a shuffle and confines the single-task stage to the write.

21.9 In order: filter the pathological key · broadcast the other side · salt.

Filter first because the skew is usually a null or a sentinel, and the fix is a modelling correction rather than a performance trick — guest checkouts with customer_id IS NULL do not belong in a per-customer aggregate at all.

Broadcast second because if the other side of the join is small, broadcasting removes the shuffle entirely and skew stops existing — there is no partitioning by key any more, so there is nothing to be uneven.

Salt last, because it is the only one that changes the shape of your code.

What salting costs:

It multiplies the small side by the salt factor. With N = 16, the dimension is exploded 16× before the join, which is 16× the memory on the build side and real work.

It requires a second aggregation. Aggregating by (key, salt) gives partial results that must be re-aggregated by key — a second shuffle, so you have traded one skewed shuffle for two balanced ones.

And it is not composable. The salt column has to be carried through every operation between the explode and the final aggregation, and any downstream code that groups by the key without accounting for the salt is wrong. It is the only one of the three that makes the code harder to read, which means it should be the last one you reach for and the one with the most explanatory comment above it.

21.12 (Measurement.)

BEFORE                                                    4 Exchange nodes
  groupBy session_id  -> agg                shuffle 1
  join sessions                             shuffle 2
  groupBy customer_id -> agg                shuffle 3
  orderBy total desc                        shuffle 4
                                            elapsed  312 s

AFTER                                                     2 Exchange nodes
  groupBy session_id -> agg                 shuffle 1
  join broadcast(sessions)                  --  removed
  groupBy customer_id -> agg                shuffle 2
  orderBy ... on a 900-row result           --  removed (local sort)
                                            elapsed  118 s   (2.6x)

Which two were removed and why they were removable:

Shuffle 2, the join. sessions is 700,000 rows and about 40 MB — comfortably inside the broadcast threshold. It was not being broadcast automatically because the table had no statistics, which is the ordinary reason: spark.sql.autoBroadcastJoinThreshold compares against an estimate, and with no stats the estimate is the file size or worse. An explicit broadcast() removes the guess.

Shuffle 4, the sort. The orderBy was applied to the final result, which after the second aggregation is 900 rows. A global sort of 900 rows does not need a distributed range-partitioning exchange; collecting and sorting locally is instant. This one is removable because of where it is in the plan, not because of anything about the data — an orderBy late in a pipeline over a small result is free, and the same orderBy two steps earlier is a shuffle of the whole dataset.

Shuffles 1 and 3 did not merge, because the input was not already partitioned by customer_id. Had it been — had the read come from a table written with repartition("customer_id") — the second groupBy would need no exchange, which is the third available saving and the one that requires changing an upstream job.

21.14 (Measurement.)

fix                          elapsed   notes
none                          47 min   1,999 tasks in ~1 s, one in 46 min
1. filter the null key         4 min   guests aggregated separately
2. broadcast the other side    6 min   dim is 40 MB; join shuffle gone
3. salt, N = 16               11 min   two shuffles instead of one skewed

I would ship fix 1, and the comment above it is the deliverable:

# Guest checkouts have customer_id NULL -- 4.2% of orders, all hashing to
# one partition. Before this filter, one task ran 46 minutes while 1,999
# finished in a second (Ch. 21 section 21.5).
#
# This is a MODELLING correction, not a performance fix: a per-customer
# aggregate has no row for "no customer". Guests are aggregated separately
# below and unioned back, so the output is unchanged.
#
# If guest volume changes materially, re-check the split: at ~20% of
# orders the separate aggregate becomes the expensive one.
real   = orders.filter(F.col("customer_id").isNotNull())
guests = orders.filter(F.col("customer_id").isNull())

Three things the comment does that the code cannot. It states the measurement that justified the change, so the next person does not have to rediscover 46 minutes. It says the change is semantic, not performance — which stops someone "simplifying" it back. And it names the condition under which the fix stops being right, which is the same discipline as an ADR's reversal condition (Chapter 3 §3.7), applied to twelve lines of code.

21.16 (Measurement.)

narrow pipeline: read 340 GB -> filter -> write

  without .coalesce(1)      2,720 tasks    9 min
  with    .coalesce(1)          1 task    4 h 12 min      <-- !!

then, with a groupBy inserted BEFORE the coalesce:

  read -> filter -> groupBy -> coalesce(1) -> write
                            2,720 tasks then 1    11 min

The difference is the shuffle boundary.

In the narrow pipeline there is nothing between the read and the coalesce, so Spark fuses them into one stage — and a stage's parallelism is its output partition count. One output partition means one task means one core reads all 340 GB.

Inserting a groupBy creates an Exchange, which ends the stage. The read-and-filter stage now has its own 2,720 tasks; the coalesce applies only to the stage after the exchange, which is small. The constraint can no longer travel upstream because there is a wall in the way.

The practical rules that follow:

coalesce is safe after a wide operation and dangerous after a narrow one, and nothing in the API distinguishes the two.

And repartition(1) is the safe form — it forces its own shuffle, so the upstream stage keeps its parallelism, at the cost of writing the data once more. In this pipeline repartition(1) runs in 12 minutes against coalesce(1)'s four hours, for the same one output file.

21.18 (Measurement.)

# empty PushedFilters -- the predicate is on a derived column
events.withColumn("d", F.to_date("event_ts")).filter(F.col("d") == "2026-11-27")

# non-empty -- the predicate is on the stored column, unmodified
events.filter(F.col("event_date") == "2026-11-27")

Spark UI, stage input                    bytes read       records read
predicate on a derived column               341.0 GB      5,110,000,000
predicate on the partition column             0.93 GB        14,000,000
                                            ─────────
                                              367x

Read it from the Spark UI, not from the plan, because the plan states an intention and the UI states a fact. PushedFilters in the plan means the filter was offered to the scan; whether the source applied it depends on the format, the file layout, and the engine's version. The stage's input-bytes column is what you were billed for, and it is the only number that settles the question.

And notice the second row is 0.93 GB, not zero. Partition pruning removed 364 of 365 days; the remaining day is still read in full, because a partition predicate prunes files and does not prune within them. Getting below 0.93 GB requires projection (fewer columns) or sort order (row-group skipping), which are Chapter 11's mechanisms — and the fact that the two are separable is worth seeing in the numbers.

21.20 The query — a key whose distribution suggests it is conflating two kinds of entity:

-- Candidates: keys whose behaviour is orders of magnitude off the median
WITH per_key AS (
  SELECT customer_id,
         count(*)                              AS n_lines,
         count(DISTINCT order_id)              AS n_orders,
         count(DISTINCT date_trunc('day', placed_at)) AS n_days,
         sum(net_revenue_cents)                AS revenue_cents,
         count(DISTINCT ship_postal_code)      AS n_ship_destinations
    FROM fct_order_line JOIN dim_customer USING (customer_sk)
   GROUP BY 1
), stats AS (
  SELECT percentile_cont(0.5)  WITHIN GROUP (ORDER BY n_orders)  AS med_orders,
         percentile_cont(0.99) WITHIN GROUP (ORDER BY n_orders)  AS p99_orders
    FROM per_key
)
SELECT p.*, p.n_orders / s.med_orders AS x_median
  FROM per_key p CROSS JOIN stats s
 WHERE p.n_orders > s.p99_orders * 10
    OR p.n_ship_destinations > 50          -- the discriminating signal
 ORDER BY p.n_orders DESC;

n_ship_destinations is the column that does the actual work. A heavy individual buyer looks like a heavy individual buyer: many orders, one or two addresses. An entity that is really a distributor ships to fifty places, and that is the signal that distinguishes "a big customer" from "a customer record standing in for many customers."

The honest false-positive rate: high, and unavoidably so.

Genuine large customers exist, and at Kestrel the largest wholesale account is 8% of order lines and is correctly modelled. Every one of them appears in this query, so the top of the list is mostly true positives for "unusual" and false positives for "misclassified."

Corporate accounts ship to many addresses legitimately — offices, sites, employees' homes — so n_ship_destinations catches them too.

In practice I would expect on the order of 20–40 candidates at Kestrel's size, of which perhaps two or three are actual conflations. That is a bad precision number and a perfectly good outcome, because the query is a triage tool for a human, not a test. Forty rows to look at once a quarter is cheap; the alternative is finding out during a 47-minute Spark stage.

What it cannot see: a conflation that is not skewed. Two small entities sharing one customer record produce no distributional signal at all, and no query of this shape will ever find them.

21.22 Falsifying "caching usually does not help" — a job where it produces a large, reproducible speedup:

base = (spark.read.parquet("s3a://silver/order_lines/")
             .filter(F.col("order_date") >= "2026-01-01")
             .join(broadcast(dim_product), "product_id")
             .withColumn("margin_cents", F.expr("net_cents - cost_cents")))
base.cache(); base.count()          # materialize once

by_category = base.groupBy("category").agg(F.sum("margin_cents"))
by_region   = base.groupBy("region").agg(F.sum("margin_cents"))
by_month    = base.groupBy("month").agg(F.sum("margin_cents"))
outliers    = base.filter(F.col("margin_cents") < 0)
# ... 8 more aggregations over the same base

without cache   12 aggregations x (read 41 GB + join + compute)   47 min
with cache      1 x (read + join + compute) + 12 x scan memory     6 min

The conditions §21.11 names, and which this job satisfies: the DataFrame is used more than once (twelve times); it is expensive to compute (a scan plus a join plus a projection); and it fits in the cluster's memory (41 GB compressed, cached as ~11 GB of columnar in-memory representation across executors). All three, which is why it works — and the section's claim is that most jobs satisfy at most one.

A job where caching makes it fail:

big = spark.read.parquet("s3a://bronze/events/")     # 341 GB
big.cache()
big.groupBy("event_type").count().show()             # used exactly once

The mechanism: cache() defaults to MEMORY_AND_DISK, so Spark attempts to hold 341 GB of decompressed, deserialised rows — several times the on-disk size — in executor storage memory. Spark's unified memory manager lets storage evict execution memory down to a floor, so the aggregation's hash tables are squeezed while the cache fills.

What you observe is not "cache full, moving on." You observe the job spilling, then GC pressure climbing to 40–60% of task time, then executors being killed by the resource manager for exceeding their container limits, then task retries re-reading and re-caching, and finally a stage failure after four attempts. The error is ExecutorLostFailure or a container OOM kill, and it names memory rather than caching, so the cause is not obvious from the failure.

And the job would have succeeded in minutes without the cache line. That is the sharp version of §21.11's point: caching a DataFrame used once is not merely useless — it converts a working job into a failing one, and it does so through a mechanism that the error message does not mention.

21.24 (Implementation.) The boundary test, written first:

def test_session_straddling_the_window_is_emitted_once():
    # deliberate fixture: a session that begins BEFORE the window
    events = spark.createDataFrame([
        ("a1", ts("2026-11-13 23:50:00"), "page_view"),   # in the OVERLAP
        ("a1", ts("2026-11-13 23:58:00"), "page_view"),   # in the OVERLAP
        ("a1", ts("2026-11-14 00:05:00"), "purchase"),    # in the WINDOW
    ], SCHEMA)

    out = sessionize(events, lo="2026-11-14", hi="2026-11-15", overlap_days=1)

    # the midnight rule splits it, so the 14th sees ONE session starting
    # at 00:05 -- and the 13th's tail is NOT re-emitted here
    assert out.count() == 1
    assert out.first()["started_at"] == ts("2026-11-14 00:05:00")
    assert out.first()["event_count"] == 1

Why this test has to be written by hand and will never appear by accident: a fixture generated from a random or sequential timestamp distribution will not place two events either side of midnight with a gap under 30 minutes. The bug lives in a region of the input space that random data does not visit, which is the general reason boundary tests are written deliberately.

And the fourth assertion in the exercise — overlap > gap — is the one to make a build-time check rather than a test:

assert OVERLAP_SECONDS > GAP_SECONDS, (
    "overlap (%ds) must exceed the session gap (%ds), or sessions "
    "straddling the boundary are split. Ch. 18 CS2." % (OVERLAP_SECONDS, GAP_SECONDS))

Because the failure mode is somebody widening the gap from 30 minutes to 2 hours a year from now, in a different file, without knowing the overlap exists. A test catches that in CI; an assertion at the top of the job catches it even if the test was excluded from the run, which is the case Exercise 23.20 is about.

Chapter 22 — Python Transformations

22.1 pandas is an in-memory DataFrame library built on NumPy (and increasingly on Arrow), with eager, row-and-column-oriented semantics and an API accumulated over fifteen years.

Polars is a query engine with a DataFrame API, built on Arrow, with an expression language and — in lazy mode — an optimiser that rewrites your pipeline before executing any of it.

DuckDB is an embedded analytical database. It is a SQL engine with a planner, a vectorised executor, and out-of-core execution, that happens to run inside your process rather than on a server.

The distinction that matters: two of the three are engines and one is a library. pandas executes what you wrote, in the order you wrote it. Polars-lazy and DuckDB execute what they decide is equivalent to what you wrote — which is why they can do projection and predicate pushdown into Parquet, and pandas cannot.

22.3 scan_parquet returns a lazy frame; read_parquet reads the file. The two effects, with magnitudes from §22.6/§22.7:

Projection and predicate pushdown into the file. The optimiser sees the whole pipeline before anything is read, so it reads only the columns the query names and only the row groups the predicate can match. On a 20-million-row workload this is the difference between 111 MB read and a fraction of it.

A far lower memory multiplier. Polars lazy peaks at 10.9× the Parquet size against eager's 27.5× — a 2.5× reduction in peak RSS, on the same query, from one function name. On a container with a fixed memory limit that is the difference between a job that runs and a job that is killed.

And the second effect is the one that matters operationally, because runtime differences are tolerable and an OOM is an incident.

22.5 The peak-RSS column matters more than the seconds column.

Because runtime degrades and memory fails. A query that is 20% slower costs 20% more; a query that exceeds the container limit does not complete at all, and it does so at an input size you did not predict. Polars-lazy wins the seconds column by 1.21× over DuckDB and loses the memory column by 3.8× — and at 3× the data, the first difference is still 1.21× and the second one has become a crash.

The seconds column is also the more fragile measurement. It moves with core count, cache warmth, concurrency, and library version. The memory multiplier is a property of the algorithm — how much of the data must be materialised — and it transfers between machines in a way that timings do not.

22.7 The rule: streaming engines need roughly RAM ÷ 3, materialising engines roughly RAM ÷ 10 — expressed as the largest Parquet input that will fit.

Applied to a 64 GB laptop, using §22.7's measured multipliers:

DuckDB          2.9x  ->  64 / 2.9   ~= 22 GB of Parquet
Polars lazy    10.9x  ->  64 / 10.9  ~=  5.9 GB
Polars eager   27.5x  ->  64 / 27.5  ~=  2.3 GB
pandas         28.2x  ->  64 / 28.2  ~=  2.3 GB

Then take about 60% of those, because the OS, the browser, and the other things on a laptop are not free, and a job at 95% of physical memory is swapping rather than computing.

practical ceilings on a 64 GB laptop
  DuckDB          ~13 GB Parquet
  Polars lazy     ~3.5 GB
  pandas          ~1.4 GB

The number people find surprising is the last one: a 64 GB machine, and pandas is comfortable to about 1.4 GB of Parquet. That is §22.10's step 4 in concrete terms, and it explains a great many "why did this die on a big machine" incidents.

22.9 The decision procedure:

1. Does the output need to be a pandas DataFrame for something downstream? If yes and the data is small, use pandas.
2. Is the transformation naturally SQL? If yes, use DuckDB.
3. Is it naturally imperative — row-wise logic, a parser, a state machine? Use Polars.
4. Does one run's working set exceed roughly RAM ÷ 10? Then it is Chapter 21, or a warehouse.
5. Is it already written in pandas and fast enough? Leave it alone.

Step 5 is the one people skip, and it is the one that would prevent the most wasted work. "We could make this 25× faster" is a true statement about a job that takes four seconds and runs once a night, and acting on it costs a rewrite, a review, a new dependency, and a class of type bug (§22.9's) in exchange for 3.8 seconds a day.

Step 1 is a close second, for a related reason: teams optimise the transformation and then pay the conversion back at the boundary, ending up slower than they started.

22.12 (Measurement.)

rows        pandas          polars-eager    polars-lazy    duckdb
20M         0.62 s          0.62 s          0.41 s         0.50 s
100M        MemoryError     6.1 s           2.4 s          2.1 s
400M        --              MemoryError     14.8 s         8.9 s
1.2B        --              --              MemoryError    31.4 s

pandas fails first, at 100M rows on a 64 GB machine, and the error is the answer:

numpy.core._exceptions._ArrayMemoryError: Unable to allocate 22.4 GiB for
an array with shape (100000000, 28) and data type float64

Two things about that message are worth noticing, and they are why the exercise asks for it.

It names an allocation, not a workload. The message tells you a single array could not be created; it does not tell you the job was 3× over budget, or which operation asked for it, or what would have fit. Diagnosing a pandas OOM means reasoning backwards from an allocation size to an operation.

And float64 is in there. 28 columns at 8 bytes is a type problem as much as a size problem — integer cents read as floats, or a categorical read as object. A large fraction of pandas memory failures are fixable by declaring dtypes, and the error is telling you so if you read it.

Polars-lazy and DuckDB fail differently and later — DuckDB spills to disk and slows down rather than dying, which is what "out of core" means and is why it is the last one standing.

22.14 (Measurement.)

import duckdb
duckdb.sql("SELECT SUM(net_cents) FROM 'lines.parquet'").write_parquet("out.parquet")
duckdb.sql("DESCRIBE SELECT * FROM 'out.parquet'").show()
#   column_name        column_type
#   sum(net_cents)     INT128        <- on a modern DuckDB

Finding the smallest lossy value. The concern is a SUM whose result exceeds the range of the type it is written as, or a round trip through a floating type:

INT32  max                 2,147,483,647   = $21,474,836.47
INT64  max     9,223,372,036,854,775,807   = $92,233,720,368,547,758.07
DOUBLE exact integers up to  9,007,199,254,740,992   (2^53)
                                           = $90,071,992,547,409.92

The first value at which a DOUBLE round trip is lossy is 2⁵³ + 1 = 9,007,199,254,740,993 cents. Showing the work:

>>> x = 2**53 + 1
>>> int(float(x)) == x
False
>>> int(float(x)) - x
-1                       # it rounds DOWN to 2**53

And the number that actually matters is much smaller than either limit. Kestrel's annual revenue is $182.0M = 18,200,000,000 cents — comfortably inside INT64 and inside DOUBLE's exact range. The INT32 limit, however, is $21.47 million, which Kestrel passes in seven weeks.

So the practical finding is: INT32 is the real hazard and it is not hypothetical. A SUM over integer cents in an engine that infers INT32 overflows within a quarter of Kestrel's revenue, and it overflows silently in engines that wrap rather than error. ::BIGINT on every SUM of a _cents column is the rule (Exercise 22.22), and the reason is a threshold that a mid-sized business crosses in weeks.

22.16 (Measurement.)

postal.csv:  a single column "postal_code" with values 01234, 90210, 02138
mixed.csv:   same column with 01234, 90210, K1A 0B1

engine        postal.csv                    mixed.csv
pandas        int64:    1234, 90210, 2138   object:  '01234','90210','K1A 0B1'
polars        i64:      1234, 90210, 2138   str:     '01234','90210','K1A 0B1'
duckdb        BIGINT:   1234, 90210, 2138   VARCHAR: '01234','90210','K1A 0B1'

All three lose the leading zero on the all-digit file. All three get the mixed file right.

That is the shape of the bug and it is worse than "they behave differently." They behave identically, and correctly by their own rules, and the correctness depends on whether a Canadian or British customer happens to be in the sample.

A file with 100,000 US postal codes infers as an integer. The same pipeline, a week later, with one UK postcode in it, infers as a string — and now postal_code has a different type than yesterday's partition, joins fail or silently produce nothing, and the change has no cause anybody can point at.

The fix is the same in all three: declare the type.

pd.read_csv("postal.csv", dtype={"postal_code": "string"})
pl.read_csv("postal.csv", schema_overrides={"postal_code": pl.Utf8})
duckdb.sql("SELECT * FROM read_csv('postal.csv', columns={'postal_code':'VARCHAR'})")

And the rule that generalises past postal codes: never let an identifier be inferred. Postal codes, phone numbers, account numbers, SKUs, ZIP+4, ISBNs, and any code with a check digit are strings that happen to be spelled with digits. The test for whether a column is a number is whether adding two of them means anything.

22.18 (Implementation.)

def test_three_engines_agree():
    pdf = transform_pandas(SRC)
    pl_ = transform_polars(SRC).to_pandas()
    ddb = transform_duckdb(SRC).df()
    for other, name in ((pl_, "polars"), (ddb, "duckdb")):
        pd.testing.assert_frame_equal(
            pdf.sort_values(KEY).reset_index(drop=True),
            other.sort_values(KEY).reset_index(drop=True),
            check_dtype=True,          # the point of the exercise
            check_like=False)

What had to be reconciled, and every item is a real, ordinary difference:

Integer nullability. pandas' NumPy-backed int64 cannot hold a null, so a left join with no match promotes the column to float64. Polars and DuckDB keep it an integer with a null. Fixed by using pandas' nullable Int64 throughout — and note that this is exactly Exercise 6.10's "zero is a measurement, null is an absence" arriving as a dtype.

String types. object versus string[pyarrow] versus Utf8 versus VARCHAR. Fixed by pd.options.future.infer_string = True, which is the pandas 3.0 default and which older code does not set.

Group ordering. DuckDB's GROUP BY does not guarantee output order; Polars' group_by does not either (unless maintain_order=True); pandas' groupby sorts by default. Fixed by sorting explicitly in the test rather than by making the engines agree — the ordering is not part of the contract and pinning it would be testing the wrong thing.

Decimal versus float. DuckDB returns DECIMAL for some aggregations where the others return a float, and assert_frame_equal treats those as different dtypes even when every value matches. Fixed by casting to a declared output schema at each engine's boundary, which is the real lesson: the reconciliation should not happen in the test — it should happen in the transformation, as an explicit output contract. A test that discovers three engines disagree has found a bug; a transformation that declares its output types has prevented it.

22.20 Formalising the test: a memory-limit increase requires evidence that the memory is needed, not that the job used it.

To grant a limit increase, require:

1. PEAK RSS, measured, for the current job.
2. The INPUT SIZE that produced it, and the multiplier (peak / input).
3. Which of section 22.7's four inflators accounts for the multiplier:
     decompression · high-cardinality grouping · joins · legacy dtypes
4. What was tried:
     - are dtypes declared? (Ex. 22.16)
     - is it eager where it could be lazy? (Ex. 22.3 -- 2.5x, free)
     - is it materialising a column subset it does not need?
5. The PROJECTED input size in 12 months, and the projected peak.
6. What happens when the new limit is also exceeded.

GRANT when 3 identifies an unavoidable inflator, 4 shows the cheap fixes
were tried, and 5 shows headroom.
REFUSE when 3 is "I don't know" or 4 is empty -- and say what to measure.

Item 3 is the test's core, and it is the one that separates a need from a symptom. Decompression is unavoidable; legacy string dtypes are a one-line fix worth 20×. A request that cannot say which one is asking for a limit to make a bug survive longer.

Item 6 is the one that gets skipped and is the reason the request recurs. A limit raised from 4 GB to 8 GB with no answer to "and then what" will be a request for 16 GB in a quarter, and each grant makes the eventual failure larger and later.

Applied to a real request I have seen: "the nightly enrichment job OOMs at 4 GB, please raise it to 16." Peak RSS 3.9 GB on a 140 MB CSV — a 28× multiplier, which item 3 identifies immediately as legacy object dtypes on six string columns. Declaring dtypes took twenty minutes and brought peak RSS to 620 MB, a 6.3× reduction, and the request was withdrawn. The limit increase would have worked too, and it would have cost 4× the container for the same job forever.

22.22 The lint rule:

SUM_CENTS = re.compile(
    r"\bSUM\s*\(\s*"                       # SUM(
    r"(?:DISTINCT\s+)?"                    #   optional DISTINCT
    r"(?!.*?::\s*(?:BIGINT|HUGEINT|INT128|DECIMAL))"   # not already cast
    r"[\w\.\"`]*?_cents\b"                 #   ..._cents
    r"[^)]*\)",                            # )
    re.IGNORECASE | re.DOTALL)

def lint(sql_text, path):
    # strip Jinja so {{ ref() }} and {% if %} do not confuse the match
    stripped = re.sub(r"\{\{.*?\}\}|\{%.*?%\}", " ", sql_text, flags=re.S)
    return [(path, m.start(), m.group(0)) for m in SUM_CENTS.finditer(stripped)]

False-positive rate: meaningful, and in a tolerable direction.

It fires on SUM(x_cents) inside a CASE that already bounds the range; on aggregates over a table of five rows in a test fixture; on SUM(refund_cents) where the total genuinely cannot exceed INT32; and on a SUM whose result is immediately divided. I would expect roughly 20–35% false positives on a mature project — annoying, and the fix is ::BIGINT, which is harmless, so the cost of a false positive is one cast rather than an investigation. That asymmetry is what makes the rule shippable.

What it cannot see, and this is the longer list:

Aggregation in the BI layer. The warehouse gets SELECT net_cents FROM ... and the tool sums it. Nothing in the repository contains a SUM at all, and this is where most of the risk actually is.

SUM over a column that does not end in _cents. revenue, amount, total, gross — the naming convention is the rule's entire reach, so the rule is really enforcing a naming convention, and it is worth saying so.

Aggregates built by Jinja or by string concatenation, where the text SUM( never appears literally.

And accumulation in Python — df["net_cents"].sum() on an int32 NumPy column, which overflows silently and which the SQL lint has no visibility into at all.

The honest summary: the rule catches the literal, in-repository, conventionally-named case, which is perhaps half the exposure. The other half needs a different control — declaring the output type in the model's schema and asserting it, which is Exercise 22.18's real conclusion.

22.24 (Measurement.)

job                          peak RSS   limit    %    verdict
enrich_customers              3,912 MB   4 GB   95%   FAIL
build_features                2,104 MB   4 GB   51%   ok
parse_user_agents             1,880 MB   2 GB   92%   FAIL
reconcile_daily                 412 MB   2 GB   20%   ok
export_partner_feed           1,301 MB   2 GB   64%   FAIL
... 6 more                                            ok

3 of 11 jobs above 60% of their container limit.
None has ever failed.

Three of eleven, and none has ever failed — which is the point. A job at 95% of its limit is not a job that is fine; it is a job whose failure date is a function of input growth, and that date is computable:

enrich_customers: input growing 3.1%/month, peak RSS ~linear in input
  headroom = 4096 / 3912 = 1.047
  months to failure = ln(1.047) / ln(1.031) = 1.5 months

Six weeks. And nothing in the platform would have said so, because the job is green every night.

The design decisions that make this control work rather than annoy:

Publish peak RSS next to duration, always — as a first-class metric, not a debugging aid — so the trend is visible before the threshold is.

Fail the build at 60%, not at 90%. 60% leaves room to diagnose; 90% leaves room to fail.

And report months-to-failure, not the percentage. "95% of limit" prompts a limit increase. "Fails in about six weeks at the current growth rate" prompts someone to ask why the multiplier is 28× — which is Exercise 22.20's actual question.

Chapter 23 — Data Quality and Testing

23.1 A pipeline that fails costs a bounded delay, paid by the person who opens the dashboard at 06:15 — who can escalate it. A pipeline that lies costs an unbounded amount, compounding until discovery, paid by people downstream of everyone who could have caught it.

The asymmetry is in who bears the cost and when they discover it, and both halves matter: the cost of failure lands on someone with the standing to act, and the cost of a lie lands on people who do not know they are paying it.

In currency, from this book's own history: Kestrel's duplicate-rows incident ran 31 days, reporting revenue 11.4% high. On $182.0M annual GMV, a month is roughly $15.2M, so the reported figure was about $1.73M above the real one — during a period in which inventory purchasing, marketing spend, and a board update were all set against it. An outage of the same pipeline for 31 days would have been noticed on day one and would have cost a month of manual reporting.

23.3 Testing the pipeline asks "did the code run?" Testing the data asks "is the result right?"

The question that tells you which a given check is: could this fail while the pipeline is completely healthy?

If no — the check measures the pipeline. Did the DAG complete, did the task exit zero, was the file written, was it non-empty, did it finish in time. Chapter 1's incident had six such checks and all six were green, because the pipeline was healthy and the answer was wrong.

The sharpest version of the trap is "did the row count increase." It looks like a data test, it reads like a data test, and in Chapter 1's incident the row count increasing was the bug.

23.5

models:
  - name: fct_order_item
    description: "One row per order line, after refunds. Grain: order_item_id."
    tests:
      - dbt_utils.unique_combination_of_columns:
          combination_of_columns: [order_id, line_number]
    columns:
      - name: order_item_id
        tests: [unique, not_null]

Nine lines, and they catch every fan-out this book describes. The unique_combination_of_columns test is the one doing the work: it asserts the business grain, which survives a change to the surrogate key. A unique test on a surrogate generated by the model itself is nearly tautological (§19.20); a uniqueness test on the natural grain is not.

23.7 Three things a dbt test cannot do:

It cannot test data before it lands. dbt runs against the warehouse; a malformed CSV, a badly-typed inference (Exercise 22.16), or a truncated file has already been loaded by the time dbt sees it. Great Expectations, or an assertion in the loader, covers it (§23.5).

It cannot test data that is not in the warehouse. A Kafka topic, an object-storage prefix, an API response, a file on its way to a partner. Chapter 15's DLQ, Chapter 17's consumer-side assertions, and Chapter 12's quarantine cover those.

It cannot detect that a model did not run. A test on a model that was excluded from the build simply does not execute, and the build is green. Freshness checks (§19.8) and the orchestrator's own SLA monitoring (Chapter 24) cover it — and Exercise 19.24 shows how that cover fails.

A fourth worth adding: it cannot reconcile against a system dbt cannot query. The payment processor, the source OLTP database, the general ledger. That is a singular test with a foreign connection, or a separate reconciliation job (§23.18).

23.9 The four obligations a quarantine creates:

1. Somebody must look at it, on a stated cadence, by name.
2. There must be a replay path, idempotent, tested, with a --dry-run.
3. There must be a bound — a rate above which the batch fails instead of quarantining.
4. There must be a retention, after which quarantined rows are deleted.

The one that turns a recoverable problem into a permanent one is the second — the replay path.

Without a bound (3) you get a large quarantine, which is bad and fixable. Without attention (1) you get a stale one, which is bad and fixable. Without a replay path, the rows in quarantine can never rejoin the pipeline, so a schema fix shipped three weeks later repairs the future and leaves the past permanently short. The data is right there, and there is no mechanism to put it back.

And the trap is that the replay path is the easiest of the four to defer, because on day one the quarantine is empty and replay feels hypothetical. Write it before you need it (§15.22(f)).

23.12

About an hour, for all five, and the hour is not the obstacle.

The obstacle is that before the incident, nobody knows which hour to spend. There are a hundred plausible assertions and five of them would have mattered; writing all hundred is not an hour, and it is also not obviously wrong. This is the honest tension in the chapter and it should not be smoothed over.

Two things do generalise, though, and they are why the table is worth building.

Four of the five are instances of §23.4's six assertions — grain, completeness, volume, freshness — and the sixth (ref() hygiene) is a structural check, not a data one. You do not need to guess the incident; you need to cover the six categories on every mart, which is a bounded amount of work and is Exercise 23.23(b).

And every one of the five is cheaper than its incident by three to four orders of magnitude. Ten minutes against eight months of silently lost orders is not a judgment call.

23.14 (Audit.) A worked example of what the audit finds:

check found:  "orders_loaded_today"
  SELECT count(*) FROM bronze.orders_raw WHERE ingest_date = current_date;
  alerts if 0.

section 23.3's test: can it fail while the pipeline is completely healthy?
  NO. If the extract ran and wrote anything at all, it passes.
  It is a pipeline check wearing a data check's name.

What it actually measures: that the job wrote something. It cannot detect a partial load, a duplicated load, rows landing in the wrong partition, or a load that wrote 400 rows instead of 6,575. It fires only when the job fails to write anything — which the orchestrator already told you.

The data version of the same check is two lines longer and a different check entirely:

SELECT count(*) AS n,
       count(DISTINCT order_id) AS n_distinct,
       min(placed_at), max(placed_at)
  FROM bronze.orders_raw
 WHERE ingest_date = current_date;
-- assert: n BETWEEN 5800 AND 9000        (volume band)
--         n = n_distinct                 (no duplicates)
--         max(placed_at) >= current_date - 1   (right window)

Three assertions where there was one, and each of them can fail while every job is green.

23.16 (Audit.)

muted alerts: 34

  age > 365 days                 11
  age > 90 days                  19
  no owner recorded              22   <-- !
  no reason recorded             27   <-- !!
  no expiry set                  31

  oldest: "fct_order_item volume floor" -- muted 2024-08-19, 741 days,
          no owner, reason field contains "temp"

"Temp", 741 days ago. That is the finding, and it is the ordinary one — nobody muted it in bad faith; somebody silenced a noisy check during an incident and the incident ended.

The three fields that are missing are the fields that would have made it self-correcting. An owner means somebody is asked. A reason means the next reader knows whether it still applies. An expiry means the mute un-mutes itself, which is the only one of the three that works without anybody remembering.

And if your alerting tool cannot answer this, that is the finding — and it is a bigger one than any individual mute. A mute is a suppressed control, and a system that cannot enumerate its suppressed controls cannot tell you what it is actually monitoring. The coverage report is then fiction, which is Exercise 23.24's point.

23.18 (Audit.) A worked example:

reconciliation:  gold.daily_revenue vs the payment processor's daily settlement
runs:            daily, 07:00
tolerance:       0.1%

WHAT IT DOES NOT COVER
1. Invoiced wholesale revenue. It never touches the payment processor --
   it is invoiced on 30-day terms through a different system. That is
   ~14% of GMV, entirely unreconciled. (This is Case Study 1's gap.)
2. Gift card REDEMPTIONS. The processor sees $0; revenue is recognised.
3. Refunds issued outside the processor -- store credit, manual
   adjustments by support.
4. Any error that affects both sides identically -- e.g. an order
   excluded from BOTH by the same status filter.
5. Timing. The processor settles on ITS calendar day in ITS time zone.
   A 0.1% tolerance absorbs the boundary; a real 0.08% error hides in it.

Item 4 is the one worth staring at, because it is the general limit of every reconciliation: two systems can only disagree about things they both measure. A rule applied on both sides — a status filter, a test-order exclusion, a date-boundary convention — cancels out perfectly and is invisible. Chapter 38's capstone reconciliation is designed around exactly this, which is why it verifies against independently derived ratios rather than only against the source.

And item 5 is the everyday one. A tolerance chosen to absorb an unavoidable boundary effect also absorbs every real error below it, permanently. The fix is not a tighter tolerance — it is the sign-test from Exercise 13.19, which detects a persistent small bias without needing the tolerance to be small.

23.20 One audit that catches all five: check that every control has executed and could have failed, from evidence outside the control itself.

# For every declared control -- dbt test, freshness threshold, alert rule,
# script in a DAG, quarantine table -- produce four facts from RUNTIME
# evidence, never from configuration:
#
#   1. RAN?        Did it execute in the last N scheduled windows?
#                  (run_results.json, task instance history, job logs)
#   2. COULD FIRE? Is there an input under which it would fail?
#                  (tautology check; threshold vs. observed distribution)
#   3. REACHES?    Did its notification channel accept a message recently?
#                  (a synthetic heartbeat sent through the real channel)
#   4. ACTED ON?   For accumulating controls (quarantine, DLQ): is the
#                  drain rate > 0 over the window?
#
# FAIL the audit for any control missing any of the four.

control                         RAN  COULD FIRE  REACHES  ACTED ON  verdict
unique(order_item_id)           yes     yes        n/a       n/a     ok
not_null(net_cents) [coalesced] yes     NO         n/a       n/a     FAIL 1
freshness orders error_after 7d yes     NO         n/a       n/a     FAIL 2
alert -> #data-alerts-old        n/a     n/a         NO        n/a     FAIL 3
scripts/check_grain.py           NO      yes        n/a       n/a     FAIL 4
quarantine.order_items           n/a     n/a        n/a        NO     FAIL 5

The unifying insight is that all five failures are the same failure: an artefact that exists and does not operate. An audit that counts artefacts sees six controls. An audit that demands runtime evidence of execution and of fireability sees one.

What it costs: the four evidence sources are the hard part. run_results.json is easy; the synthetic channel heartbeat needs building; the tautology check needs a per-test analysis; and the drain rate needs the quarantine to record deletions. Call it a week to build and an hour a month to read — and it replaces four separate audits that each cover one row.

What still gets past it, honestly:

A control that runs, can fire, reaches a live channel, and tests the wrong thing. Exercise 23.14's orders_loaded_today passes all four checks and measures the pipeline. Correctness of the assertion is not mechanisable, and this audit does not attempt it.

A control whose channel is live and whose recipients ignore it. The heartbeat proves delivery, not attention. The only evidence for attention is §23.16's mute audit plus a record of acknowledgements, and even that measures clicking rather than reading.

And a control that is fireable in principle and not in practice — a threshold at 0.001% on a metric whose noise floor is 0.1%. Check 2 says it could fire; it fires every day, gets muted, and becomes failure 3 by a different route.

23.22 (Measurement.)

Kestrel's dbt project, one month of production builds

  total warehouse credits, builds                        3,281
  credits attributable to `dbt test` nodes                 214    6.5%
  at $2.00/credit                                         $428 / month
                                                        $5,136 / year

  204 tests across 31 models
  cost per test per year                                 $25.18

The compute claim holds, comfortably. $5,136 a year against a $366,136 platform bill is 1.4%, and against a single incident — Chapter 20's Case Study 2 lost $110,560.14 of orders — it is not a number anyone should spend time on.

The attention cost, estimated honestly:

  test failures in the month                                61
  genuine data problems                                      9    15%
  known-benign / expected                                   38    62%
  the test itself was wrong                                 14    23%

  median triage time per failure                        18 min
  total                                                 18.3 h / month
                                                        ~220 h / year

220 engineer-hours a year against $5,136 of compute — so the attention cost is roughly ten to twenty times the compute cost, depending on what you value an hour at. The claim holds, and by a wider margin than the chapter states.

And the composition is the actionable part. 85% of failures were not data problems. The 62% benign are tests whose thresholds are wrong (§19.24's second row) and the 23% wrong tests are §19.7's decorative ones surfacing in the other direction. Fixing those two categories would return roughly 190 hours a year, which is a month of engineering, and it requires no new tooling — only reading the failure log and asking §19.7's question of each one.

23.24 (Implementation.)

def coverage(mart, alert_state):
    """A muted check does not count. Six boxes; below six is a CI failure."""
    boxes = {}
    for dim in ("grain", "completeness", "volume", "freshness",
                "validity", "reconciliation"):
        checks = declared_checks(mart, dim)
        live = [c for c in checks if not alert_state.is_muted(c)]
        boxes[dim] = bool(live)
    return boxes

mart                  grain  compl  vol  fresh  valid  recon   score
gold.fct_order_item     Y      Y     Y     Y      Y      Y       6/6
gold.fct_order          Y      Y     Y     Y      Y      -       5/6  FAIL
gold.daily_revenue      Y      -     M     Y      -      Y       3/6  FAIL
gold.dim_customer       Y      Y     M     -      Y      -       3/6  FAIL

  Y = live   M = declared but MUTED   - = absent

3 of 4 marts fail. Two of the seven gaps are MUTES, not omissions.

Three of four fail on the first run, and two of the gaps were invisible before this change — the volume checks on daily_revenue and dim_customer are declared, appear in every previous coverage report as present, and are muted.

And the muted-check part is the exercise for the reason the question says. It requires the coverage tool to read the alerting tool's runtime state, and most platforms cannot do that — the tests live in dbt, the mutes live in PagerDuty or Opsgenie or a Slack workflow, and there is no API contract between them.

Finding out that they cannot is the finding, and it is the same finding as §23.16's: a platform that cannot enumerate its suppressed controls does not know what it monitors. The coverage report before this change was not slightly optimistic; it was reporting the configuration and calling it coverage, which is Exercise 23.20's failure mode applied to the audit itself.

Part V — Orchestration and Operations

Chapter 24 — Orchestration with Airflow

24.1 Cron stops working at the fourth dependency, and the failure is gradual: every scheduled time is a guess about how long the previous step takes, and the guesses encode a dependency graph in a form that cannot be read, tested, or changed safely.

Five things it cannot do:

1. Express a dependency. 0 3 * * * does not mean "after the load"; it means 03:00, and on the night the load takes 70 minutes it means "during the load."
2. Retry with backoff. A failed job is a failed job; the next run is tomorrow.
3. Backfill a range. Re-running last March means writing a loop and hoping every script takes a date argument.
4. Prevent a run overlapping itself. A job that takes 70 minutes on a 60-minute schedule runs twice, concurrently, against the same target.
5. Tell you what happened. No run history, no per-task status, no duration trend. "Did last Tuesday's load succeed?" is answered by grepping a log if somebody kept one.

The honest costs of the alternative: an orchestrator is a service — a scheduler, workers, and a metadata database — with upgrades, a disk that fills (§24.24), and a failure mode of its own that nothing else watches (Case Study 2). A managed one has a floor around $300–350 a month whether or not a DAG runs (Appendix H §H.5), and a self-hosted one is one of the two categories Chapter 5 §5.3 calls most expensive to lack and most expensive to run badly. Below about ten scheduled jobs, cron plus a lock file plus a run-history table is genuinely the better trade.

24.3 A DAG with schedule="@daily" and a data interval of 2026-03-17 runs at the end of that interval — just after midnight on 2026-03-18 — and it is about 2026-03-17's data.

Why the design is correct despite being confusing: you cannot process a day until the day is over. A run "for the 17th" that fired at 00:00 on the 17th would process nothing. The scheduler is naming runs by the period they cover rather than by the moment they execute, which is the only naming that makes a run identifiable by its output.

And the consequences are what make it worth the confusion:

A backfill and a scheduled run are the same operation. Both are "run the DAG for interval X," so re-running March is not a special code path.

Every task can be a pure function of its interval. Given data_interval_start and data_interval_end, the task's output is determined — which is what makes it idempotent, testable with airflow tasks test, and safe to retry.

The confusion is entirely in the name of the run, and it is worth paying because the alternative — naming a run by its execution time — makes every backfill a different thing from every scheduled run, and that is where the bugs live.

24.5 catchup=True on deploy schedules one run for every interval between start_date and now, all at once.

When it is right: the DAG is genuinely incremental, each interval's work is independent and idempotent, and you want the history built — a new model that needs two years of backfill, deployed deliberately, with the concurrency to survive it.

When it is wrong, which is most of the time: a start_date set to "when I wrote this" — a common, harmless-looking two years ago — schedules 730 runs the moment the DAG is deployed. They saturate the worker pool, starve every other DAG (§24.8), and hit the source system with two years of extracts in ten minutes.

And there is a worse version. If the DAG is not truly interval-scoped — if a task reads current_date rather than its interval (§24.14) — all 730 runs process today's data and write it 730 times. The result is Chapter 1's duplicate-rows incident, compressed into an afternoon.

The default should be catchup=False, with backfills invoked explicitly, rate-limited, and reviewed (§13.23). Turning catchup on is a decision; leaving it on is an accident.

24.7 A task should be the smallest unit you would want to retry independently.

That is the whole rule, and it produces the right answer in both directions: an extract and its load are separate tasks because re-running the load should not re-hit the source; the seventeen dbt models are one task because retrying one of them without the others is meaningless.

What splitting costs, and it is more than people expect:

Scheduler and worker overhead per task. Queueing, slot acquisition, container startup, and teardown — commonly 5–20 seconds each. A DAG split into 200 tasks pays that 200 times, and on short tasks the overhead exceeds the work.

State has to cross a boundary. Two tasks cannot share memory, so anything passed between them goes through XCom (small values only) or through storage — which means serialising, writing, and reading data that a single task would have held in a variable.

And the DAG gets harder to read. A 200-node graph is not more comprehensible than a 20-node one; it is less. The dependency structure is documentation, and beyond a point splitting destroys it.

24.9 The four limits:

parallelism            deployment-wide: total task instances running at once
max_active_tasks       per DAG: concurrent tasks within one DAG
max_active_runs        per DAG: concurrent RUNS of the same DAG
pool slots             per named resource: a shared budget across DAGs

max_active_runs=1 is Chapter 20's lock. It is the setting that guarantees two runs of the same DAG are never writing the same target concurrently — which is what makes a delete-insert or a merge safe without a distributed lock.

And it is the one people leave at its default, because the default (16) is invisible until a run is slow. The tell is a backfill: with max_active_runs unset, a backfill runs many intervals concurrently against a shared table, and the interleaving of their delete-insert windows produces gaps that no test detects.

Pools are the second-most-underused, and they are the fix for §24.16's starvation: a backfill pool of 4 slots out of 32 means a backfill can never consume more than an eighth of the deployment, regardless of how many runs it schedules.

24.12 (Measurement.)

start_date = 30 days ago, schedule = @daily, catchup = True

  runs scheduled on deploy               30
  scheduler time to queue them          ~4 s
  time for all 30 to complete       41 min   (local, 4 worker slots,
                                              ~5 min per run)
  peak concurrent runs                   16   <- the max_active_runs DEFAULT

Two things to notice in your own numbers.

The scheduler queues them almost instantly and then they run for the rest of the afternoon — so the symptom is not a slow scheduler, it is every other DAG waiting. On a local instance you see it as a delay; on a shared deployment it is §24.16's starvation.

And 16 concurrent runs of the same DAG is the default, not a choice. If any task in the DAG writes a shared target, sixteen runs are writing it at once and the outcome depends on their interleaving. The exercise is worth doing precisely because the number 16 appears without anyone having typed it.

Then set it back, and if this is a real deployment, set start_date to a fixed date in the past and leave it there — a start_date computed from datetime.now() is its own well-known bug.

24.14 (Implementation.)

# BEFORE -- reads the wall clock. Not a function of its interval.
@task
def load_orders():
    day = date.today() - timedelta(days=1)          # <- the bug
    run(f"COPY INTO bronze.orders FROM ... WHERE d = '{day}'")

# AFTER -- a function of the interval, and nothing else.
@task
def load_orders(data_interval_start=None):
    day = data_interval_start.date()
    run("DELETE FROM bronze.orders WHERE d = %s", day)   # idempotent
    run("COPY INTO bronze.orders FROM ... WHERE d = %s", day)

airflow tasks test kestrel_daily load_orders 2026-03-17
airflow tasks test kestrel_daily load_orders 2026-03-17    # again

BEFORE, run 1   loaded 6,412 rows for 2026-09-01  (today-1, not the interval)
BEFORE, run 2   loaded 6,412 rows for 2026-09-01  again -> 12,824 rows
                                                  DIFF: duplicated

AFTER,  run 1   loaded 6,701 rows for 2026-03-17
AFTER,  run 2   loaded 6,701 rows for 2026-03-17
                                                  DIFF: none

The "before" output is the more instructive one, and it fails in two independent ways. It loaded the wrong date — a backfill of March produces September's data — and it duplicated on the second run. Either alone is a bug; together they mean a retry of a backfill silently multiplies today's rows into a historical partition.

And retries are configured, so the second run is not hypothetical. This is the exercise that carries the chapter for exactly that reason: the diff is empty in the fixed version and the fixed version took two lines.

24.16 (Measurement.)

DAG A: a 30-day backfill, max_active_runs unset (default 16)
DAG B: kestrel_hourly, scheduled every hour, needs 2 worker slots
deployment parallelism: 16

WITHOUT a pool
  backfill queues 30 runs, occupies all 16 slots within 4 seconds
  DAG B's 15:00 run queued at 15:00:02
  DAG B's 15:00 run STARTED at 16:47      <- 107 minutes late
  (and its 16:00 run was already queued behind it)

WITH a `backfill` pool of 4 slots
  backfill occupies 4 slots; 12 remain
  DAG B's 15:00 run STARTED at 15:00:06   <- 6 seconds
  backfill takes ~4x longer, and nobody notices

The trade is stated in the last line and it is the whole point. A pool does not make the backfill faster; it makes it not matter. Four times the wall clock on a job nobody is waiting for, in exchange for a scheduled DAG that keeps its SLA.

And notice what the failure looks like without the pool: DAG B did not fail, did not alert, and appears in the run history as "succeeded." The only visible symptom is a 6am dashboard that was fresh at 08:00, and diagnosing that from DAG B's logs is impossible, because nothing in DAG B went wrong.

24.18 (Implementation.)

# canary.py -- four lines that watch the thing nothing else watches
with DAG("canary", schedule="*/10 * * * *", catchup=False,
         start_date=datetime(2026, 1, 1)) as dag:
    BashOperator(task_id="ping",
                 bash_command="curl -fsS https://hc-ping.com/$CANARY_UUID")

scheduler stopped at 14:32:10
external monitor's grace period: 20 minutes (2 missed pings)
notification received at 14:52:41

  time to notification: 20 min 31 s

The design decision worth stating is that the monitor is external, and it is not a detail. A heartbeat checked by the same system that produces it cannot detect that system's absence — an Airflow sensor watching an Airflow DAG proves nothing when the scheduler is the thing that died.

And 20 minutes is a choice, not a fact. A tighter grace period detects faster and false-alarms on every deploy and every scheduler restart. The right value is derived from the SLA: Kestrel's 6am deadline with a 3.8-hour pipeline means the scheduler must be alive by about 02:00, so a 20-minute detection at any hour is comfortably sufficient and a 2-minute one would buy nothing but pages.

24.20 (Audit.) The absences in a platform, and the minimum heartbeats that cover them:

ABSENCE                                    would produce NO signal
1. the scheduler is not running            no runs, no failures, nothing
2. a DAG was deleted or unpaused-off       it simply is not there
3. a sensor is waiting forever             "running" is not an alert state
4. the alert route is broken               alerts fire into nothing
5. a source system stopped producing       an empty extract "succeeds"
6. a quality check was excluded from
   the build                               green, and never ran
7. the metadata DB disk is full            the scheduler stops (-> 1)

MINIMUM HEARTBEAT SET
  H1  canary DAG -> EXTERNAL monitor       covers 1, 7 (and 2 for itself)
  H2  each DAG writes a run record;
      a separate job asserts one exists
      per expected interval                covers 2, 3
  H3  a synthetic alert through the REAL
      route, weekly, asserting receipt     covers 4
  H4  a source-volume floor, not a
      success check (Ex. 23.14)            covers 5
  H5  the coverage audit (Ex. 23.24) run
      as a scheduled job, not in CI        covers 6

H2 is the one that generalises and the one people do not build. A run record per expected interval, asserted by a different system, converts "this DAG stopped existing" into a failing check — and it is the same mechanism as a canary, applied per DAG rather than per deployment.

What remains uncovered, honestly: H5 checking itself. The coverage audit is a scheduled job, and a scheduled job that stops running produces no signal — so it needs an H2 record like everything else. The regress terminates at H1, the external monitor, which is the only component outside the system. That is why it has to be external, and it is the whole argument for paying a third party a few dollars a month for a URL.

24.22 An alternative that is correct and intuitive: name the run by its execution moment and pass the interval explicitly.

# runs at 00:15 on 2026-03-18, named "2026-03-18T00:15",
# and receives the window it is responsible for
@task
def load(window_start, window_end):   # 2026-03-17T00:00, 2026-03-18T00:00
    ...

This is intuitive — the run is named when it ran, which is what everyone expects — and correct, because the window is still explicit and the task is still a pure function of it.

The case that breaks it: late-arriving data, and therefore every backfill and every reprocess.

The 2026-03-17 window is loaded on the 18th. On the 22nd, six orders arrive with placed_at on the 17th (§20.7). You must now re-run "the 17th" — and under this scheme there is no such run. There is a run named 2026-03-18T00:15 whose window happened to be the 17th, and you must either re-execute that named run (which now has a name that lies about when it executed) or create a new run named 2026-03-22T09:40 with the 17th's window (so two differently-named runs produce the same partition).

Either way the run's name has stopped identifying its output, which is the property the whole system depends on: retries, backfills, airflow tasks test, and every "which run produced this partition" question.

The deeper point is that there are two clocks — when the work happened and what period the work is about — and any scheme must pick one to be the identity. Airflow picks the period, which is confusing and identifies the output. Picking the moment is intuitive and identifies nothing, because the same period can be processed at many moments. The confusion is the cost of the correct choice, and the mitigation is documentation rather than a redesign.

24.24 (Implementation.)

scheduler killed at 03:14:02
  canary missed its 03:20 ping
  monitor grace 20 min
  notification at 03:40:18                time to notification: 26 min 16 s

metadata volume, before `airflow db clean`
  task_instance      41,208,113 rows      18.4 GB
  xcom                8,102,440 rows       6.1 GB   <- see below
  log                31,004,882 rows      11.2 GB
                                          ────────
                                          38.9 GB of a 50 GB volume  (78%)

after `db clean --clean-before-timestamp <90 days ago>`
                                           4.1 GB  (8%)

Two findings, and the second is the one worth carrying away.

The disk alert at 70% would have fired weeks before the scheduler stopped. It is the cheapest possible control on the most consequential single point of failure in the platform, and it is not enabled by default anywhere.

And 6.1 GB of XCom is a design smell, not a cleanup problem. XCom is for small values (§24.9); six gigabytes of it means something is passing data through the metadata database. db clean makes the symptom go away and leaves the cause, which will refill the volume on the same schedule.

Chapter 25 — Monitoring and Observability

25.1 The three parts: is the data current · is the data right · will it still be current tomorrow.

Most teams do not have the third. The first two are answerable from freshness checks and Chapter 23's assertions; the third requires trends — duration creeping, memory creeping, the margin to the deadline shrinking — and it is where every incident in Chapters 21, 22, and 24 was visible months in advance.

The reason it is missing is structural rather than lazy: the first two questions are about this run and every orchestrator answers them; the third is about the distribution of runs and requires history, a baseline, and someone to look at a line rather than a light. Nothing produces it by default.

25.3 A test and a metric measuring the same quantity want different thresholds because they have different costs of being wrong.

order lines per day:  median 17,753

  the TEST's floor        3,000   -- fires only on catastrophe
  the METRIC's band  ±25% of a trailing median (13,300 - 22,200)

The test's threshold must be one nobody will mute. A test failure stops the build and pages somebody; a false positive at 3 a.m. costs a night's sleep and, on the second occurrence, gets the test turned off (§23.7). So the floor is set well below anything that has ever happened — 3,000 against a median of 17,753 is not timid, it is a threshold chosen to survive.

The metric's band must be one that moves. Nothing stops, nobody is woken, and a value outside the band is a line on a chart that somebody looks at in the morning. It can therefore be tight enough to be informative, and a day at 13,000 lines — 27% low, and invisible to the test — shows up immediately.

The general form: a threshold's tightness is bounded by its consequence. The same number cannot serve both, and trying to make one threshold do both jobs produces either a test that gets muted or a metric that never says anything.

25.5 Because a ratio is scale-free, and a threshold is not.

The four properties that make it work:

1. It survives growth. A job that takes 22 minutes today and 34 next year breaks a fixed 40-minute threshold on a Tuesday with no incident; a 2× ratio against a trailing median never does.
2. It survives seasonality and hardware changes. The baseline moves with the workload, so a faster warehouse or a bigger cluster does not require re-tuning every alert.
3. It works on every job with one number. One rule — "above 2.0× the trailing 14-run median" — applies to a 4-second job and a 3-hour one, so there is nothing to configure per job and nothing to forget when a job is added.
4. It catches the direction people forget. A ratio below 0.5 is as alarming as one above 2.0 — a job that suddenly takes half as long is usually processing half as much data, which is Chapter 19's frozen dimension and Chapter 20's missing rows.

Property 4 is the one that earns it. A duration threshold is inherently one-sided; a ratio is naturally two-sided, and the fast-run alert has caught more silent data loss in this book than the slow-run alert has caught cost regressions.

25.7 The margin is the gap between when the pipeline finishes and when the deadline is — Kestrel's build completes at 04:48 against a 06:00 SLA, so the margin is 72 minutes.

Why 100% SLA compliance can coexist with an outage four months away: compliance is a binary scored after the fact, and it says nothing about how close you came. A pipeline that finished at 05:59 for thirty consecutive days is 100% compliant and has a one-minute margin.

margin, 90-day trend
  day   1   82 min
  day  30   77 min
  day  60   72 min
  day  90   67 min

  slope: -0.167 min/day
  margin reaches zero in 67 / 0.167 = 401 days... at the CURRENT rate.
  But the slope itself is steepening with data volume, and a linear
  extrapolation of the last 30 days gives ~120 days.

The compliance rate cannot show this and the margin trend cannot hide it. That is why §25.23(b) calls the extrapolated date the most important number the platform produces: it is the only metric that is about the future, and it converts "we are fine" into a date somebody can plan against.

25.9 The four requirements: it must be actionable (there is something to do), attributable (it says whose it is), documented (it names a runbook), and received (it reaches a person who is awake and responsible).

Most alerts fail the first. An alert that says a number is unusual, with no accompanying decision, trains its recipient to acknowledge and move on — and Case Study 2's 11% actionable rate is the measured version of that.

The failure is subtle because "actionable" is not the same as "true." A correct alert about a real anomaly that nobody can act on at 03:00 is still an alert that should not have paged. The test is whether the recipient can do something now, and if the honest answer is "look at it in the morning," it is a report, not an alert.

25.12 (Measurement.) The five columns and what health.py finds:

run_id | dag_id | task_id | started_at | ended_at | status | rows_out | cost

SIGNALS
  duration ratio > 2.0 (14-run median)      2 tasks
  duration ratio < 0.5                      1 task     <- investigate first
  margin to SLA, 90-day slope            -0.167 min/day
  extrapolated margin = 0                  ~120 days
  cost ratio > 2.0                          0 tasks
  peak RSS > 0.60 of limit                  3 tasks    (Ex. 22.24)

"If you cannot export it, that is the finding" is the important half of this exercise. A platform whose scheduler cannot produce a per-run, per-task table with start, end, status, and row count cannot answer any of §25.1's three questions except by a human clicking through a UI — and that means the trend question is not merely unanswered, it is unanswerable.

What is usually missing, in order: rows_out (almost never recorded — the orchestrator does not know what the task did), cost (needs a join to the warehouse's query history by a tag nobody sets), and per-task history beyond the retention of the metadata database (§24.24's db clean deletes exactly the history this analysis needs). The fix for all three is §25.23(a): every job writes its own run record, to a table you own, rather than relying on the orchestrator's.

25.14 (Audit.) A worked example:

GREEN INDICATOR:  "Data freshness: OK"

  what number was thresholded?
    max(_ingested_at) on bronze.orders, compared to now() - 6 hours

  plotted for 90 days:
    p50 lag   1h 12m
    p95 lag   4h 41m
    max       5h 52m     <- eight minutes from the threshold, twice
    trend     +4.1 min/week

The indicator was green on every one of those 90 days and it is eight minutes from turning red, with a trend that reaches the threshold in about three weeks.

This is Case Study 1's question and it generalises to every green light in every dashboard. A boolean is a number plus a comparison, and the boolean discards exactly the information needed to know whether it is about to change. Plotting the underlying number is usually a ten-minute exercise and it routinely finds that a green indicator has been drifting toward its threshold for a quarter.

And the useful habit that follows: display the number next to the light. "Freshness OK (5h 52m of 6h)" is the same indicator with the drift visible, and it costs nothing.

25.16 (Measurement.)

last month
  alerts fired                          412
  acknowledged by a human               219      53%
  someone took an action                148      36%
  the action changed an outcome          45      11%    <- actionable

Case Study 2:  11% / 53% / 36%   -- the same three numbers

The gap between 53% and 11% is the whole finding. Half the alerts were acknowledged, which is what a dashboard measures and what a rotation feels like it is doing. One in nine mattered.

The 36%-versus-11% gap is the more uncomfortable one, because those 103 alerts produced an action that changed nothing — an investigation, a re-run, a mute. Work was done, in the middle of the night, in response to a signal that did not need it.

And the number to track going forward is the ratio, not the count. Reducing 412 alerts to 200 by raising thresholds may improve or worsen the ratio; only the ratio says whether the rotation is being paged for reasons.

25.18 (Measurement.)

-- 400 days, because annual reporting exists
SELECT t.table_name,
       max(q.query_date) AS last_read,
       count(*)          AS reads_400d
  FROM information_schema.tables t
  LEFT JOIN query_log q ON q.table_referenced = t.table_name
                       AND q.query_date > current_date - 400
 GROUP BY 1
 ORDER BY reads_400d ASC;

tables:                    214
read by nobody in 400 d:    61      (28%)
  of which, marts:          11
  of which, staging:        29
  of which, one-off/tmp:    21

Investigate three before deleting any, and here is why the exercise insists.

Of three investigated at Kestrel: one was a genuine orphan (a mart for a deprecated dashboard, safe to delete); one was read by a quarterly finance process that had not run inside the window — a 400-day window catches annual reporting and misses an 18-month audit cycle; and one was read by a process that queries through a view, so the query log attributed the read to the view and not to the table underneath.

Two of the three would have been wrong to delete, and the third was fine — which is roughly the hit rate to expect and is exactly why the count is a triage list rather than a deletion list.

The two structural blind spots are worth naming: query logs attribute reads to the object named in the SQL, so views, macros, and SELECT * through a wrapper all hide the real dependency; and any window is shorter than somebody's cycle. The safe procedure is deprecation, not deletion — rename it, wait a cycle, and see who complains (Chapter 30 §30.2).

25.20 (Audit.)

CHAIN                                              covered by
1. scheduler alive                                 H1 canary -> external
2. scheduler + executor + WORKER can run a task    H1 (trivial canary)
3. workers can SCALE to the resource a real
   task needs                                      NOT H1  <-- Kestrel's gap
4. warehouse reachable, credentials valid          H2 resource canary
5. object storage writable                         H2
6. the quality DAG itself runs                     H3 freshness job writes ts
7. the alert route delivers                        H4 synthetic alert, weekly
8. the source system is producing                  H5 source volume floor
9. the external monitor itself is alive            (the monitor's own
                                                    dead-man's switch)

MINIMUM SET
  H1  trivial canary  -> external monitor          1, 2
  H2  RESOURCE-SIZED canary: requests the same
      memory/warehouse as the nightly build,
      writes a row, reads it back                  3, 4, 5
  H3  freshness job writes its own timestamp;
      a check asserts the timestamp moves          6
  H4  synthetic alert through the REAL route,
      asserting receipt                            7
  H5  source volume floor, not a success check     8

Chain 3 is the one Kestrel missed and the reason H2 exists. The trivial canary is a bash echo; it needs one small worker slot and it passed throughout an incident in which the worker pool could not scale to the memory the nightly build required. A heartbeat proves the chain it exercises and nothing else, and a deliberately trivial one proves a deliberately trivial chain.

What remains uncovered, and it should be stated in the design rather than discovered:

Correctness. Every heartbeat above proves that machinery moved. None of them proves the numbers are right, which is Chapter 23's entire subject.

Partial capacity. H2 proves the pool can allocate one large task. It does not prove it can allocate twelve concurrently, which is the shape of the actual failure on a busy night.

And the monitor's own liveness, which is chain 9 and is why a monitoring service with a dead-man's switch is worth choosing over one without.

25.22 (Implementation.)

-- Snowflake: cost per job per run, from QUERY_HISTORY, keyed by a tag
SELECT query_tag                                              AS job,
       date_trunc('day', start_time)                          AS d,
       sum(credits_used_cloud_services
           + total_elapsed_time/1000/3600 * wh_credits_per_hour) AS credits
  FROM snowflake.account_usage.query_history
 WHERE query_tag IS NOT NULL AND start_time > current_date - 30
 GROUP BY 1, 2;

The tag is the whole prerequisite, and it is one line in the dbt profile (query_tag: "{{ model.unique_id }}") or one ALTER SESSION in each job. Without it, cost is attributable to a warehouse and not to a job, which is Chapter 33's 36.4% unattributed share arriving one level down.

Alert on the ratio, not the amount, for §25.5's reasons — and here the case is even stronger, because absolute cost varies with data volume, warehouse size, and concurrency, none of which the job controls.

What the first week finds, in the order it usually appears:

One job is a large share of the total — commonly 30–50% — and it is rarely the one people would have guessed. It is usually a dashboard-backing model refreshing far more often than anyone reads it.

A job's ratio spikes on Mondays, which turns out to be a weekly full-refresh nobody remembered was configured.

And several jobs have a ratio below 0.5 on some days, which is the interesting one: they are processing less data than usual, silently. The cost alert catches data-loss incidents that no quality test caught, which is a genuinely surprising result and the best argument for building it.

25.24 (Measurement.)

margin impact, applied retroactively to the last 10 merged changes
  (measured on staging: median of 5 runs before vs after)

  #4102  add supplier freshness sensor              +3.1 min
  #4108  add order-line grain test                  +0.4 min
  #4115  add a second source freshness sensor       +2.9 min
  #4121  widen fct_order_item lookback 3d -> 7d     +8.2 min
  #4130  add customer dimension snapshot            +4.6 min
  #4133  add a deferrable sensor                    +0.2 min
  #4137  add clickstream sessionization window      +6.8 min
  #4140  add reconciliation query                   +1.1 min
  #4144  add three column-level tests               +0.3 min
  #4151  add a pre-hook assertion                   +0.1 min
                                                   ─────────
                                                   +27.7 min

Twenty-eight minutes, from ten changes, none of which was arguable.

Every one of them is a good change. A freshness sensor, a grain test, a widened lookback that fixed a real completeness bug — reviewing any of these individually and asking "is three minutes worth it?" gets a yes every time, correctly.

The aggregate was invisible at every point where it could have been questioned, which is Case Study 1's sixteen sensors and 55 minutes, reproduced at a smaller scale. The margin is a shared resource that no individual change consumes noticeably, and a shared resource with no per-change accounting gets consumed to zero.

Which is why the line goes on the pull-request template rather than in a quarterly review. Not to reject changes — almost none of these should be rejected — but so that the twenty-eighth minute is visible at the moment it is spent, and so that a change costing 8.2 minutes gets one sentence of thought about whether the lookback needs to be seven days or whether five would do.

Chapter 26 — On-Call for Data

26.1 Four ways, each with a consequence:

1. The failure is usually not urgent, but the decision is. A stale daily_revenue loses nothing until somebody makes a decision on it — 06:15, at Kestrel. Consequence: the incident has a deadline rather than a bleeding rate, so "respond immediately" means something different and a rotation copied from a service team over-pages.

2. The recovery is often waiting. A pipeline frequently cannot be fixed at 03:00 at all, because the source has not produced the data or the fix is a four-hour backfill. Consequence: the on-call job is assessment and communication, not repair — and a rotation designed around repair trains people to do the wrong thing.

3. The blast radius is measured in decisions, not requests. "How many users were affected" has no analogue. Consequence: the impact question is usually unanswerable at 03:00 and sometimes unanswerable ever, so the incident record has to capture what could have been decided on the wrong number.

4. Correctness is discovered later, by someone else. A service outage is known at the moment it happens; a wrong number is known weeks afterwards. Consequence: the incident timeline starts before anyone knew there was an incident, and "time to detect" is the metric that matters rather than "time to resolve."

26.3 Correctness is scored retrospectively because you cannot know a number was wrong until something reveals it — a reconciliation, a downstream complaint, a later investigation. There is no moment at which the system can assert "this figure is correct"; there are only moments at which nothing has yet contradicted it.

What that makes it the only SLI able to measure: the incidents you did not detect.

Freshness and availability are measured from the system's own signals, so they can only ever count failures the system noticed. Correctness is scored from discoveries, which include discoveries the system had no part in — a finance analyst's reconciliation, a customer complaint, an auditor.

And the uncomfortable corollary is Exercise 26.20's: a retrospectively scored measure creates an incentive not to look. A month with no correctness incidents may be a good month or an uninvestigated one, and nothing in the number distinguishes them.

26.5 Burn rate is the rate at which an error budget is being consumed, expressed as a multiple of the rate that would exhaust it exactly at the end of the window. A burn rate of 1 exhausts the budget precisely on schedule; a burn rate of 14.4 exhausts a 30-day budget in about two days.

It needs a short window because the point of the metric is early warning. Computed over the full 30-day window, the burn rate is a slow-moving average that only reaches an alarming value after most of the budget is already gone — by which time the alert is a report of a completed failure.

What happens if you compute it over the full window, concretely:

30-day budget: 43.2 minutes of unavailability

a total outage begins at t=0
  burn rate over a 1-HOUR window     reaches 14.4x within an hour  -> page
  burn rate over the 30-DAY window   reaches 1.0x after 43 minutes,
                                     but the 30-day AVERAGE is still
                                     ~0.03x -- indistinguishable from
                                     normal for days

The standard resolution is multi-window burn-rate alerting: a short window for speed and a long one to suppress false alarms, requiring both to be elevated. The short window makes it fast; the long window makes it trustworthy, and either alone is unusable.

26.7 The seven things a runbook must contain:

1. Is this actually the problem? — the check that confirms or eliminates the hypothesis first.
2. Impact — who is affected and by what deadline.
3. Communication — who to tell, where, and what to say before you know anything.
4. Triage — how to distinguish this cause from the ones that look identical.
5. Immediate action — what to do now, including "buy time."
6. Recovery — how to get back to normal and how to know you are.
7. Escalation — who to wake, and at what threshold.

The two usually missing are 1 and 3.

Number 1 is missing because the author knew. The runbook is written by the person who diagnosed the incident, for whom the diagnosis is obvious — so it starts at step 4 or 5, and the reader at 03:00 follows a procedure for a problem they may not have.

Number 3 is missing because it does not feel like part of the fix. It is the step with the largest effect on how the incident is experienced by everyone outside the rotation, and it is the one an engineer under pressure skips (§26.9).

A third, worth mentioning: a "reader needs" block — access, credentials, and context required before step 1 — which is what Exercise 26.18's drill always finds missing.

26.9 Communication is step 1 because the cost of the incident to everyone outside the rotation is almost entirely a function of how early they were told, and that cost is being incurred while you diagnose.

At Kestrel: the CEO opens the dashboard at 06:15. If they open a stale dashboard with no warning, the incident has cost a loss of confidence in the number that outlasts the outage by months. If they open it having read a message at 04:20, it is a known operational event. The diagnosis has not changed; the cost has.

And a complete message before you have a diagnosis:

04:20 — daily_revenue will not be fresh for 06:00. What: the nightly build failed at 03:52 on the order-lines load. Impact: the revenue dashboard will show data through 30 August, not 31 August. Nothing else is affected. Cause: not yet known — investigating. ETA: next update at 05:00, whether or not I know more. What to do: if you need the 31st's figure before 09:00, tell me and I will produce it manually.

Four properties make it complete without a diagnosis. It says what is wrong and what is not — the scope is the most valuable thing you can give. It states the impact in the reader's terms, not the system's. It names the next update time, which is what stops the follow-up questions. And it offers the manual path, which is often the only thing the reader actually needed.

"Cause: not yet known" is a complete answer, and waiting until you can fill it in is the mistake.

26.12 (Implementation.)

-- SLI 1: FRESHNESS -- was the gold layer current by the deadline?
SELECT count(*) FILTER (WHERE finished_at <= deadline_at)::float / count(*)
  FROM sla_runs WHERE dag_id = 'kestrel_daily' AND d > current_date - 90;

-- SLI 2: COMPLETENESS -- did every source row reach the fact table?
SELECT count(*) FILTER (WHERE missing = 0)::float / count(*)
  FROM (SELECT d, count(*) AS missing
          FROM silver.order_items s
          LEFT JOIN gold.fct_order_item f USING (order_item_id)
         WHERE f.order_item_id IS NULL GROUP BY 1) x;

-- SLI 3: CORRECTNESS -- days with no correctness incident, scored
--         retrospectively (section 26.3)
SELECT 1 - count(DISTINCT affected_date)::float / 90
  FROM correctness_incidents
 WHERE affected_date > current_date - 90;

90-day attainment
  freshness      87 / 90 = 96.7%
  completeness   88 / 90 = 97.8%
  correctness    86 / 90 = 95.6%

Two observations about doing this for real.

Freshness needs a deadline_at column, not a hard-coded time. The deadline is 06:00 America/New_York, which is a different UTC instant twice a year, and an SLI computed against a fixed UTC hour silently misreports on the days either side of a DST transition.

And SLI 3 requires an incident record with an affected_date range, which most teams do not keep. Without it, correctness attainment cannot be computed at all — which is a finding, and it is the reason §26.3's retrospective scoring needs a deliberate artefact rather than a query.

26.14 (Measurement.)

SLO: freshness by 06:00 on 99.0% of days
  window: 90 days
  budget: 1% of 90 = 0.9 days

Restated in currency, the way Ch. 20 CS2 restates a tolerance:

  a day of stale daily_revenue means the 06:15 decisions are made on
  data one day old. Kestrel's inventory reorder and the daily paid-
  marketing allocation both run off it.

  paid marketing allocated daily              ~$41,000/day
  share plausibly misallocated on stale data   ~15%
                                              ─────────
  cost of one stale day                        ~$6,150

  the SLO's budget of 0.9 days is therefore    ~$5,535 per quarter

Stating it in currency changes the conversation in both directions, which is the point of the exercise.

It makes the budget defensible. $5,535 a quarter is a number a business partner can weigh against the engineering cost of a tighter SLO, and 99.0% versus 99.5% stops being a taste question.

And it makes over-tightening visible. A 99.9% SLO on this SLI is a budget of 0.09 days — about $550 a quarter — and buying that last 0.9% would cost far more than $5,000 of engineering. The currency figure is what shows that the expensive SLO is the wrong purchase, which is the argument teams usually cannot make.

26.16 (Implementation.)

# Pre-authorization: what on-call may do without waking anyone

| Situation | Authorized | Not authorized |
|---|---|---|
| A source is late | Delay the build up to 90 min | Publish partial data |
| A quality test fails | HOLD the dashboard at yesterday's data | Publish anyway |
| A backfill is needed | Run it in the `backfill` pool | Run it unpooled |
| Disk on the metadata volume > 85% | Run `db clean` to 30 days | Extend the volume |
| A slot threatens the source DB | Extend the volume; restart the connector | DROP the slot (Ch. 14 section 14.19) |
| Stale versus wrong | **PUBLISH NOTHING. Stale is preferred.** | -- |

Taking the "stale over wrong" row to the business partner is the exercise, and it is worth doing because the answer is not obvious and it must not be the data team's to make.

Agreed with: [Finance Director], 2026-04-08.

Their words, recorded: "If it's late, tell me and I'll wait. If it's wrong
and I don't know, I've already sent it to the board. Late is an
inconvenience; wrong is a correction I have to make in public. Always
choose late -- and I want the message at 04:20, not at 06:10."

Recording it with their name on it does two things. It settles the decision at 03:00, when the on-call engineer would otherwise be guessing at somebody else's risk appetite. And it makes the decision reviewable — if the answer changes, or if a different person holds the role, the record is the thing that gets updated rather than a tacit norm that quietly diverges.

26.18 (Drill.)

runbook: "Replication slot filling the source database disk" (Ch. 14 section 14.19)
reader:  an engineer who joined four months ago
author:  present, silent
clock:   start 14:02

  14:02  step 1, "check pg_replication_slots"      -- BLOCKED, 6 min
         reader has no credentials for the source replica
  14:11  step 3, "restart the connector"           -- BLOCKED, 9 min
         runbook says "restart the connector"; there are four, and the
         command differs between the Kafka Connect REST API and the
         compose stack
  14:24  step 4, "check max_slot_wal_keep_size"    -- BLOCKED, 4 min
         setting was renamed in the version we run
  14:31  completed
                                                    total: 29 minutes
                                                    author's estimate: 8

Classifying the three blocks:

Implicit context (9 min). "Restart the connector" is unambiguous to the person who wrote it and ambiguous to everyone else. This is the most common category and the cheapest to fix — name the connector, paste the exact command.

Missing access (6 min). The reader could not perform step 1. This is the category that is invisible to the author forever, because they have the access. A "Reader needs" block at the top of every runbook fixes it, and writing one requires a drill to discover what belongs in it.

Drift (4 min). The setting was renamed. This is the category that recurs, so the fix is not editing the runbook — it is a review date, and ideally a test that the commands in the runbook still parse.

The 29-versus-8 gap is the finding worth reporting. Not that the runbook was bad — it was a good runbook — but that an author cannot estimate a reader's time, and every runbook in the platform is carrying the same three categories of defect undiscovered.

26.20 (Design.) A measure with the same property: "number of production incidents caused by our team's changes."

It is scored by a subsequent attribution — an incident review decides whose change caused it — so the measure improves if incidents are attributed elsewhere, if root-cause analysis stops at the proximate cause, or if reviews are not held. A team optimising it has three routes to a better number and only one of them involves fewer incidents.

Other examples with the same shape: bugs found in production (improves if you stop looking), data quality incidents (improves if you delete tests — §23.16's mutes), security findings (improves if you stop scanning), and on-call pages per rotation (improves if you widen thresholds).

The counter-incentive: pair every retrospectively scored measure with a measure of the looking.

scored measure                    paired "looking" measure
correctness incidents             reconciliations RUN, and their coverage
                                  (Ex. 23.18: what are they blind to?)
production bugs                   test coverage of the paths that ship
data quality incidents            LIVE (unmuted) assertions per mart
                                  (Ex. 23.24)
pages per rotation                actionable RATE, not count (Ex. 25.16)

The pairing works because the two move in opposite directions under the wrong incentive. Deleting tests improves the incident count and visibly worsens the coverage number. Neither number alone is trustworthy and the pair is, which is a cheaper fix than trying to make a retrospective measure tamper-proof.

And one cultural control that is worth more than either number: reward the finding. A team that reports "we discovered a six-month-old correctness bug" and is scored badly for it will not report the next one.

26.22 (Design.) Labelling a revenue figure that is missing one channel:

BAD -- a footnote nobody reads
  Revenue: $21,945,202          * excludes wholesale

BAD -- a colour change, which is invisible on a printout, in a
       screenshot pasted into a deck, and to some readers entirely

BETTER -- the label is IN THE NUMBER'S NAME, and the number is
          visibly incomplete

  Revenue (WEB + APP ONLY -- wholesale missing)
  $18,864,871
  ⚠ Wholesale ($3,080,331 last month) is NOT included.
     Total will be higher. Do not compare to previous months.

Three properties make the label survive, and each addresses a specific way labels fail.

It is in the metric's name, not beside it. A name travels with the number into a screenshot, a slide, a copy-paste into an email, and a verbal report. A footnote does not.

It states the direction and the magnitude. "Total will be higher" and "$3.08M last month" let a reader decide whether they can proceed. A bare "incomplete" makes the number unusable, which means the reader either discards it or ignores the label.

And it forbids the specific wrong use. "Do not compare to previous months" is the actual danger — a month-over-month decline that is entirely an artefact — and naming it is more effective than any amount of general caution.

Testing it on someone is the exercise, and the test has to be adversarial: show it for five seconds and ask what revenue was. If they say $18.9M without qualification, the label failed, and the usual fix is to make the number itself harder to read as a total — showing it in a different position, or splitting it into the two channels that are present rather than presenting one sum.

This is the dangerous degraded mode precisely because it looks like it is working. Fully down is safe; fully up and correct is safe. Partial and labelled depends on a human reading a label under time pressure, which is a control with a known and poor success rate.

26.24 (Measurement.)

one quarter
  pages                                   142
  pages outside business hours             38
  nights interrupted, per person            9.5   (4-person rotation)
  median time to acknowledge             6 min
  median time to first communication    34 min    <- target 15
  actionable rate                          14%    <- target 60%
  incidents with a runbook that was used   61%

The actionable rate fails, as expected, and it is the number that is paid for in sleep. 14% of 38 overnight pages means roughly five useful wake-ups in a quarter and thirty-three that were not — spread across four people, that is 9.5 interrupted nights each for about 1.2 nights of real work.

Time to first communication is the second failure and the more fixable one. 34 minutes against a 15-minute target is not a diagnosis problem — §26.9's message needs no diagnosis. It is a habit problem, and the fix is a template in the runbook's step 1 rather than anything technical.

And "nights interrupted per person" is the metric to lead with in any conversation about the rotation, because it is the only one on the list that a non-engineer immediately understands the cost of. Pages-per-quarter is an abstraction; 9.5 interrupted nights each is a staffing argument that makes itself.

Chapter 27 — Testing and CI/CD for Data

27.1 Four ways data CI/CD is harder, and the compromise each forces:

1. You cannot run the pipeline on production data in CI. It is too large, too expensive, and often too sensitive. Compromise: accept a sample, and accept that the sample does not exercise everything production will.

2. The "unit" under test is a query whose behaviour depends on data. The same SQL is correct or incorrect depending on what is in the tables, so a passing test says less than it does in application code. Compromise: accept that tests cover shapes of data you thought of.

3. A deploy is not idempotent in the way an application deploy is. Rolling back code does not roll back a table that has been rewritten. Compromise: accept that some deploys are one-way, and design the shapes accordingly (§27.7).

4. The feedback loop is slow. A full build is minutes to hours, so "run the tests" is not a two-second operation. Compromise: accept partial builds — slim CI, state:modified+ — and accept that they can be wrong (Case Study 2).

All four resolve to "accept something." That is the honest summary of the chapter and it is worth stating plainly: data CI/CD is a discipline of choosing which incompleteness you can live with, not of achieving the coverage application teams take for granted.

27.3 Three sources of test data, and what each cannot exercise:

The complementarity is the point: fixtures test logic you specified, samples test shapes you have, and production tests scale you cannot simulate. A CI pipeline using only one of the three has a predictable blind spot, and §27.12's deliberate sample exists to close the gap between the first two.

27.5 Slim CI requires three things operationally:

1. A stored production manifest.json, produced by the nightly run and uploaded somewhere CI can fetch it.
2. A place to build, with production-shaped data — which in practice means --defer plus a read grant on production from the CI role.
3. A decision about what "modified" means when the manifest is missing or stale.

The third is a decision rather than a task, and it is the one Case Study 2 turns on. If the manifest is absent, state:modified+ selects nothing and dbt exits 0 — so the default behaviour is a green build that tested nothing. Choosing what happens instead (fall back to a full build? fail the job? build a fixed safe subset?) is a policy question with cost implications, and it must be answered before the manifest first goes missing rather than after.

And item 2 is a governance decision wearing an operational costume. Granting CI read access to production is a real expansion of who can read production data, and it needs the masking policy, the audit trail, and the reasoning written down (§27.23(c)) — not because it is wrong, but because a grant made silently is a grant nobody reviews.

27.7 Three deploy shapes and the data operation each implies:

The failure of treating a definitional change as additive: the table silently contains two definitions of the same measure, split at the deploy date, with nothing marking the boundary.

And the specific shape of the damage is what makes it worse than a wrong number. Rows before the deploy carry the old rule; rows after carry the new one. Every aggregate spanning the boundary mixes them. A month-over-month comparison shows a step change that is entirely an artefact, and the natural interpretation — "something happened to the business in March" — is wrong in a way that generates action.

Exercise 27.24's _built_by column exists for exactly this, and Kestrel's audit found three models containing more than one definition, one of which had been created while fixing the boundary left by an earlier one.

27.9 Because a definitional change is supposed to produce differences, so a gate of "zero rows differ" fails every time it is used for the one shape that most needs it.

A shadow deploy runs the new model beside the old and compares. For an additive change, zero rows should differ and zero is the right gate. For a definitional change, the differences are the deliverable — the point was to change the number.

So the gate has to be "the differences are the intended ones," which requires the intent to be expressed as a predicate:

-- gate: every differing row differs FOR THE STATED REASON, and no others do
SELECT * FROM new n FULL OUTER JOIN old o USING (order_line_id)
 WHERE n.net_cents IS DISTINCT FROM o.net_cents
   AND NOT (o.sku LIKE 'GC-%')        -- the intended cause: gift cards
-- expect zero rows

And a second gate in the other direction, which people forget: every row that should have changed did.

SELECT count(*) FROM new n JOIN old o USING (order_line_id)
 WHERE o.sku LIKE 'GC-%' AND n.net_cents = o.net_cents;
-- expect zero: every gift-card line must have changed

Two gates, because a definitional change fails in two directions — it can change rows it should not and fail to change rows it should — and only the pair distinguishes "the change worked" from "the change did nothing."

27.12 (Implementation.)

-- fixtures/pathological.sql -- START WITH CLAUSE 3
-- Every row that has ever failed a test, from dbt's --store-failures tables.
CREATE TABLE fixtures.order_items_pathological AS
SELECT DISTINCT s.*
  FROM silver.order_items s
  JOIN (SELECT order_item_id FROM dbt_test_failures.unique_order_item_id
        UNION SELECT order_item_id FROM dbt_test_failures.not_null_customer_sk
        UNION SELECT order_item_id FROM dbt_test_failures.accepted_values_status
        UNION SELECT order_item_id FROM dbt_test_failures.quantity_range
       ) f USING (order_item_id);

rows in the pathological fixture:   1,247

  from grain violations                 412
  from null foreign keys                 88
  from unexpected status values         619
  from out-of-range quantity            128
  (some rows fail more than one)

1,247 rows out of 6.5 million — 0.019% — and they are the only rows that have ever caused a failure.

Starting with clause 3 is the instruction that makes this exercise work, and the reason is that these rows cannot be invented. Nobody writing a fixture by hand produces a status value of awaiting_stock, a quantity of 0, or a customer id that resolves to nothing — they are the rows production produced and a human would not have thought of.

And they compound. Every future test failure adds rows, so the fixture grows in the direction the system actually breaks, which is the property no synthetic fixture set has. That is why §27.23(d) calls it the exercise that carries the chapter.

One operational note: --store-failures writes production rows to a table, so the fixture inherits whatever governance the source has. Mask it at creation, not at use.

27.14 (Reproduction.)

$ rm -f ./prod-manifest/manifest.json
$ dbt build --select state:modified+ --state ./prod-manifest
...
Nothing to do. Try checking your model configs and model specification args
$ echo $?
0

Exit 0, zero models built, and a green CI badge.

The four-line assertion:

N=$(dbt ls --select state:modified+ --state ./prod-manifest --resource-type model | wc -l)
if [ "$N" -eq 0 ]; then
  echo "state:modified+ selected 0 models. Manifest missing or stale. Refusing." >&2
  exit 1
fi

$ ./ci_build.sh
state:modified+ selected 0 models. Manifest missing or stale. Refusing.
$ echo $?
1

The general form is worth stating because it recurs throughout this book: a selector that matches nothing is not the same as a check that passed. Exercise 19.9's comma-versus-space is the same bug with a different cause, and Exercise 23.20's audit is the systematic version.

And note that the fix is not "make the manifest always present." It is "decide what a missing manifest means" (§27.5's third requirement) — because the manifest will be missing, on the day the nightly job fails, which is exactly the day CI is most needed.

27.16 (Measurement.)

fct_order_item, one week of daily builds

  rows                              4,612,881
  _built_by  (git sha, 40 chars)    varchar
  _built_at  (timestamp)            timestamp

uncompressed                         +212.4 MB   (+46 bytes/row)
as Parquet + zstd                      +1.8 MB   (+0.39 bytes/row)
                                     ─────────
                                     118x smaller in Parquet

The 118× is the finding, and it is entirely dictionary and RLE encoding (Chapter 8 §8.3): a week of builds has seven distinct _built_by values and seven distinct _built_at values across 4.6 million rows. The column is a seven-entry dictionary plus a few bits per row.

At the frozen S3 rate, 1.8 MB a week is $0.0000414 a month. The columns are free.

And this is the answer to the objection the change always attracts — "two extra columns on every fact row" sounds expensive and is a rounding error, because the columns are exactly the shape columnar compression is best at. A low-cardinality column on a large table costs nothing, and the intuition that says otherwise comes from row-oriented storage.

27.18 (Audit.)

CI checks, each asked: would its failure be distinguishable from
                       having nothing to check?

check                                    fails open?   distinguishable?
dbt build --select state:modified+          YES            NO   <-- CS2
sqlfluff lint (|| true in the script)       YES            NO
manifest_audit.py --hardcoded               no             yes
the pathological fixture build              no             yes
freshness check (skips if source absent)    YES            NO
a schema-drift check reading a missing
  baseline file                             YES            NO
security scan (advisory, non-blocking)      YES            NO
                                                        ────────
                                             5 of 7 fail the test

Five of seven — which is a normal result and worse than it sounds, because a control that fails open is not merely absent; it is absent while appearing present in every audit (Exercise 23.20).

The two that pass share one property: they emit a positive assertion of work done. manifest_audit prints the number of models scanned; the fixture build prints the number of rows loaded. A check that reports what it examined can be checked; one that reports only failures cannot.

The generalisable fix is to make every control assert a floor on its own scope:

[ "$MODELS_BUILT" -gt 0 ]  || fail "built nothing"
[ "$FILES_LINTED" -gt 0 ]  || fail "linted nothing"
[ "$SOURCES_CHECKED" -eq "$SOURCES_EXPECTED" ] || fail "checked $SOURCES_CHECKED of $SOURCES_EXPECTED"

And || true in a CI script is the single highest-yield thing to grep for. It is always added deliberately, to unblock something, and it is never removed.

27.20 (Audit.) Three requirements of the "do not X; instead Y" shape, and which half exists:

1. "Do not hardcode table names; use ref() instead." The negative half exists as a CI grep (§19.22's check 3). The positive half — a check that every model's dependencies are complete — does not, so a model that uses ref() for four sources and reads a fifth through a macro passes.

2. "Do not page for non-actionable alerts; route them to a report instead." The negative half exists as thresholds. The report does not exist, so the alerts that were removed from paging were simply deleted, and the information is gone rather than relocated. This is the more common outcome than anyone admits.

3. "Do not grant broad warehouse access; grant role-scoped access instead." The negative half is enforced by a review. The positive half — the scoped roles — is half-built, so requests that do not fit an existing role get the broad grant "temporarily." Chapter 30's access review finds these.

The pattern in all three: the negative is a gate and the positive is a project. A gate is cheap, lands in one pull request, and produces an immediate visible improvement. The alternative it points at requires design, ongoing maintenance, and someone's roadmap — so it is scheduled, deprioritised, and forgotten, while the gate stays.

The consequence is that people are blocked from the wrong thing and not offered the right one, which is how a good rule becomes an obstacle. The countermeasure is procedural: do not merge the gate until the alternative exists, and if that is too slow, ship the gate as a warning rather than a failure until it does.

27.22 (Design.) The gate for a definitional change, with intent expressed as a predicate:

# deploy/intent/PR-4188.yml -- committed with the change, reviewed with it
change_shape: definitional
model: gold.fct_order_line
measure: net_revenue_cents
intent:
  description: "Gift-card lines are excluded from revenue (Ch. 38 R3)."
  rows_that_should_change:
    predicate: "sku LIKE 'GC-%'"
    expected_count_range: [28000, 33000]
    expected_direction: decrease
  rows_that_must_not_change:
    predicate: "sku NOT LIKE 'GC-%'"
  aggregate_effect:
    measure: sum(net_revenue_cents)
    expected_delta_range: [-950000000, -870000000]   # cents

-- GATE 1: nothing outside the intended set changed
SELECT count(*) FROM new n JOIN old o USING (order_line_id)
 WHERE n.net_cents IS DISTINCT FROM o.net_cents
   AND NOT (n.sku LIKE 'GC-%');                         -- expect 0

-- GATE 2: everything inside it did
SELECT count(*) FROM new n JOIN old o USING (order_line_id)
 WHERE n.sku LIKE 'GC-%' AND n.net_cents = o.net_cents; -- expect 0

-- GATE 3: the aggregate moved by the predicted amount
SELECT (SELECT sum(net_cents) FROM new) - (SELECT sum(net_cents) FROM old);
-- expect within expected_delta_range

-- GATE 4: the count of changed rows is in range
SELECT count(*) FROM new n JOIN old o USING (order_line_id)
 WHERE n.net_cents IS DISTINCT FROM o.net_cents;
-- expect within expected_count_range

How "intended" is expressed is the whole difficulty, and the answer is: as a predicate over rows, plus a predicted magnitude, both written before the shadow run.

Three properties make it work.

The predicate is the same one the model change uses. If the intent says sku LIKE 'GC-%' and the model implements something subtly different, gates 1 and 2 disagree — which is the most valuable failure this design produces, because it catches a change that does what its author wrote rather than what they meant.

The magnitude must be predicted in advance. A gate that accepts whatever difference occurs is not a gate. Requiring a range forces the author to compute the expected effect, and an author who cannot predict the delta within 10% does not understand the change well enough to ship it.

And gate 4 exists because gates 1 and 2 are both satisfied by "nothing changed." If the deploy silently did nothing — the model was not rebuilt, the selector matched zero — gates 1 and 2 pass vacuously. Every set of gates needs one that fails on the empty case, which is §27.14's lesson in another costume.

27.24 (Implementation.)

-- which code produced this row?
SELECT _built_by, min(_built_at), max(_built_at), count(*)
  FROM gold.fct_order_line
 GROUP BY 1 ORDER BY 2;

-- and the question that matters: does this model contain more than
-- one DEFINITION?
SELECT date_trunc('month', order_date) AS m,
       count(DISTINCT _built_by)       AS distinct_builds,
       count(*)                        AS rows
  FROM gold.fct_order_line
 GROUP BY 1 ORDER BY 1;

model                     definitions   boundary        found by
gold.fct_order_line             2       2026-03-14      gift-card rule
gold.daily_revenue              2       2026-01-31      refund attribution
gold.dim_customer_metrics       3       2025-11-02,     an active-customer
                                        2026-03-14      rule, then a FIX to
                                                        the boundary the
                                                        first one left

Three, and the third is the one to sit with. dim_customer_metrics has three definitions because somebody noticed the boundary from the first change and shipped a correction — which restated rows after a second date and left a second boundary. The fix for a definitional change deployed additively is another definitional change, and deploying that additively compounds the problem.

The correct handling in every case is the same and is not difficult: restate the history, or mark the boundary explicitly with a definition_version column so consumers can filter. What is difficult is knowing the boundaries exist, and before _built_by there was no way to find out — the rows look identical and only the numbers disagree.

Chapter 28 — Infrastructure as Code

28.1 Reproducibility buys three things:

1. Recovery — you can rebuild the platform after a loss.
2. Review — an infrastructure change is a diff somebody reads before it happens.
3. Consistency across environments — staging is genuinely like production, so a test in staging means something.

Ranked by value: review, then consistency, then recovery.

Review is first because it is the one that operates continuously. Every change passes through it, and the value compounds across hundreds of small decisions — the destroy that was caught, the grant that was questioned, the retention that somebody noticed.

Recovery is most often cited and is worth the least, and the reason is worth stating plainly: full recovery from code is rare, and when it happens the code is usually stale anyway because the resources that matter (the data, the state, the grants applied by hand) are precisely the ones the configuration does not fully capture. It is the argument that sells the practice and not the one that justifies it.

28.3 The boundary: Terraform manages resources whose lifecycle is measured in months and whose destruction is consequential. Your pipeline manages resources whose lifecycle is measured in runs.

Concretely: Terraform owns the warehouse, the buckets, the roles, the network, the orchestrator's infrastructure. The pipeline owns tables, partitions, and views.

The argument for putting tables in Terraform: everything is then in one place, with one review process and one audit trail. There is no "some infrastructure is code and some is not" boundary for a new engineer to learn.

The argument against, which wins: a table's schema is defined by the model that populates it, so managing it in Terraform means the definition lives in two places and they will diverge. And a terraform destroy or a resource replacement then deletes data, which is a category of accident that should not be possible — Case Study 1's twelve resources without prevent_destroy are a mild version of the same risk.

The clean statement of the rule: Terraform manages the containers, not the contents.

28.5 Because a tag is a mutable pointer and a digest is content.

python:3.13-slim refers to a different image every few weeks. A build that succeeded on Tuesday and fails on Thursday with no commit between them is almost always this, and the failure is unreproducible by construction: you cannot check out the code from Tuesday and get Tuesday's image.

FROM python:3.13-slim                       # a moving target
FROM python:3.13-slim@sha256:9f2e...c41a    # exactly these bytes, forever

Three consequences beyond reproducibility:

A rollback actually rolls back. Reverting the commit reverts the image, which is not true of a tag.

Supply-chain changes become reviewable. Updating the digest is a diff, which somebody looks at. Updating a tag is invisible.

And the failure moves to a good time. With a digest, a base-image change breaks the build when you choose to update it, in a pull request. With a tag, it breaks at 03:00 on a night you did not choose.

28.7 You adopt it by importing, not by rebuilding. Write the configuration to describe what exists, terraform import each resource into state, and iterate until terraform plan reports no changes. Then, and only then, start making changes through the code.

The "no changes" plan is the milestone, and reaching it is slow — every default you did not write down shows up as a diff, and each one has to be either added to the configuration or deliberately accepted.

The most valuable artifact of the exercise is not the Terraform.

It is the inventory. Almost every team that does this discovers resources nobody knew about: a bucket from a proof of concept, an instance running for eight months with no owner, a role with broad permissions granted for a migration that finished, a snapshot schedule that has been paying for snapshots of a deleted volume. The import forces an enumeration, and the enumeration is the finding.

And the second-most-valuable is the list of things you chose not to import — the resources you looked at and decided to delete instead. That list is usually longer than anyone expects and it is the exercise's immediate return, before a single change has been made through code.

28.9 The trace, with no long-lived secret anywhere:

1. GitHub Actions job starts. The workflow requests an OIDC token from
   GitHub's provider. It is signed, short-lived, and describes the job:
   repo, branch, workflow, environment.

2. The job calls AWS STS AssumeRoleWithWebIdentity, presenting the token.
   The IAM role's trust policy checks the issuer, the audience, and a
   CONDITION on the subject claim:
     "repo:kestrel/platform:environment:production"
   -- so a job on a fork or a feature branch cannot assume it.

3. STS returns temporary AWS credentials. Lifetime: 1 hour. They exist
   only in the job's memory.

4. The job uses those credentials to fetch a SHORT-LIVED warehouse
   credential -- either a key-pair JWT signed with a key from Secrets
   Manager, or (better) the warehouse's own OIDC/workload-identity
   federation, which takes the STS identity directly.

5. dbt reads the credential from an environment variable via env_var()
   and connects. It expires within the hour.

Nothing durable was stored anywhere. There is no secret to rotate,
leak, or find in a git history.

The property that makes this categorically better than a stored key is not the short lifetime — it is the condition in step 2. A long-lived key is a bearer token: anyone who has it is you. An OIDC trust policy binds the credential to a specific repository, branch, and environment, so a leaked token from a pull-request job cannot assume the production role at all.

And the one thing to get right is that condition. A trust policy that checks only the issuer and not the subject claim will accept a token from any repository on that provider, which is a misconfiguration that produces a working pipeline and an open door.

28.12 (Audit.)

stateful resources in the configuration:            31
  with prevent_destroy:                             19
  WITHOUT:                                          12

  of the 12:
    aws_s3_bucket.bronze                     <-- the lake
    aws_s3_bucket.silver                     <-- the lake
    aws_db_instance.metadata                     Airflow's metadata DB
    aws_db_snapshot_schedule.metadata
    snowflake_database.analytics
    aws_efs_file_system.shared
    aws_backup_vault.primary                 <-- the backups
    ... 5 more

Both lake buckets, and the backup vault. That is the finding, and it is the ordinary one — nobody decided these were disposable; prevent_destroy is opt-in and nothing prompts for it.

The lifecycle block is three lines and takes effect immediately:

lifecycle {
  prevent_destroy = true
}

Two things worth knowing about it. It blocks terraform destroy and any plan that would replace the resource — which is the more likely accident, because a replacement is triggered by changing an attribute that forces new, and the plan output says must be replaced in the middle of a long diff.

And it is not sufficient on its own. prevent_destroy can be removed in the same commit that destroys the resource, which is why §28.23(c) asks for two independent controls — the lifecycle block and a plan reviewer that fails on any destroy or replace of a tagged-stateful resource. A control you can disable in the same change it would have blocked is a speed bump.

28.14 (Measurement.)

image                              size      pull (cold)   per-task startup
python:3.13 + pip install          1.14 GB       41 s          6.2 s
multi-stage, slim runtime, digest    248 MB        9 s          2.1 s
                                   ───────                    ──────
                                     4.6x smaller              -4.1 s

per-task startup cost at Kestrel's volume
  tasks per day                      1,840
  saving                          1,840 x 4.1 s = 2.09 hours/day of
                                                  worker time
  at $2.400/node-hour, 1 slot            ~$5.03 / day
                                        ~$1,836 / year

The dollar figure is small and it is not the argument. Two hours a day of worker time is two hours of margin (§25.7) on a pipeline with 72 minutes of it, and on a night when the build is late that is the difference between 05:48 and 06:12.

And the pull time matters more than the steady-state startup, because it is paid on every new worker — which is precisely what happens when the pool scales up under load. A 1.14 GB image makes autoscaling slow at the moment autoscaling is needed, which is a failure mode that does not appear in any average.

28.16 (Audit.) Classify before deciding, and the classification is the exercise:

drift finding                              classification    then what
1. a bucket lifecycle rule added by hand   ADOPT             write it into code;
   during the Ch. 31 erasure work                            it is correct and
                                                             was urgent
2. an IAM policy widened for a migration   REVERT            the migration
   that finished in April                                    finished; this is
                                                             standing access
                                                             nobody needs
3. a tag added by the cloud provider's      LEAVE, note      not ours; adding
   own cost tooling                                          it to code creates
                                                             a permanent diff
4. an instance type changed by an           ADOPT            it was the right
   engineer during an incident                               call; code should
                                                             say so
5. a manual snapshot retained past its      LEAVE, note      it is evidence in
   schedule                                                  an open audit;
                                                             delete after

Why classifying first matters: the instinct on seeing drift is to run terraform apply and make it go away, and that reverts findings 1 and 4 — both of which were correct changes made by competent people under pressure, and reverting them re-breaks whatever they fixed.

Finding 2 is the one drift detection exists for, and it is invisible without it: a widened policy that nobody will ever notice, because nothing fails when access is too broad.

And findings 3 and 5 are why "leave, with a note" must be a category. Without it, every drift check produces the same two findings forever, the report becomes noise, and within two months nobody reads it — which is §23.16's mute problem arriving through a different door. A drift exception needs a reason and an expiry, exactly like a mute.

28.18 (Implementation.)

# post-apply: verify the resource BEHAVES, not that the command succeeded
def assert_bucket_denies_public_read(bucket):
    r = requests.get(f"https://{bucket}.s3.amazonaws.com/probe.txt", timeout=5)
    assert r.status_code in (403, 404), (
        "bucket %s answered %d to an unauthenticated GET" % (bucket, r.status_code))

def assert_versioning_actually_versions(bucket, key="_probe"):
    s3.put_object(Bucket=bucket, Key=key, Body=b"1")
    s3.put_object(Bucket=bucket, Key=key, Body=b"2")
    vs = s3.list_object_versions(Bucket=bucket, Prefix=key)["Versions"]
    assert len(vs) >= 2, "versioning is configured and not versioning"

break it by hand:
  aws s3api put-public-access-block --bucket kestrel-bronze \
      --public-access-block-configuration BlockPublicAcls=false,...
  aws s3api put-bucket-acl --bucket kestrel-bronze --acl public-read

post-apply assertion:
  AssertionError: bucket kestrel-bronze answered 200 to an
                  unauthenticated GET

The distinction the exercise is drawing: terraform apply succeeding means the API accepted the configuration. It does not mean the resource behaves as intended — because a later manual change, a conflicting policy at a higher level, an SCP, or an interaction between two settings can all produce a resource whose configuration says one thing and whose behaviour is another.

And this is the same principle as Chapter 23 §23.3, one layer down: apply succeeded is testing the pipeline; an unauthenticated GET returning 403 is testing the data. The second can fail while the first is completely healthy, which is the definition of a check worth having.

28.20 (Analysis.) Pricing Kubernetes for a four-engineer data team:

OPERATIONAL LOAD
  cluster upgrades, 3-4/year, each ~1 day incl. testing      ~4 days/yr
  node group / autoscaler tuning                             ~3 days/yr
  networking, ingress, DNS, certificates                     ~4 days/yr
  RBAC and secrets integration                               ~2 days/yr
  incidents attributable to the platform, ~4/yr x 0.5 d      ~2 days/yr
  learning curve, one engineer, ongoing                      ~5 days/yr
                                                            ──────────
                                                            ~20 days/yr
                                              = 8% of ONE engineer,
                                                or 2% of the team

ISOLATION LOST WITHOUT IT
  per-task CPU/memory limits          -- available in ECS/Cloud Run
  per-task images                     -- available
  network policy between workloads    -- weaker, and rarely needed here
  bin-packing efficiency              -- real, and worth ~10-20% of
                                         compute on a busy cluster

ALTERNATIVES
  managed Airflow (MWAA/Composer)     -- someone else runs the cluster
  ECS Fargate / Cloud Run             -- per-task isolation, no cluster
  a big VM + Docker Compose           -- honest, and the right answer
                                         below ~10 scheduled jobs

Which of §28.5's three reasons applies: none of them, for a team of four. The reasons are you already run it for other workloads (Kestrel does not), you need a scheduling model nothing else provides (Airflow's executor covers it), and you need workload isolation that a managed runtime cannot give (Fargate can).

The honest answer is therefore managed Airflow or Fargate, and the 20 days a year is the number to put in front of anyone proposing otherwise. It is not a large number in isolation, which is exactly why it gets approved — and it is 20 days that the four-person team does not have, spent on a layer that produces no data.

28.22 (Design.) Distinguishing changes the author requested from changes that come along:

INFORMATION NEEDED
  For each resource change in the plan, the CAUSE:
  (a) an attribute the author edited in this diff
  (b) an attribute changed by a variable/module version bump
  (c) an attribute whose PROVIDER default changed
  (d) drift -- the real resource differs from the last-applied state
  (e) a cascade -- forced by a change to a resource this depends on

WHAT TERRAFORM'S JSON PLAN CONTAINS
  resource_changes[].change.before / .after            yes
  resource_changes[].change.actions                    yes
  resource_changes[].change.before_sensitive           yes
  resource_changes[].action_reason                     yes, PARTIALLY --
      "replace_because_cannot_update", "delete_because_no_resource_config"
  which ATTRIBUTE forced the replacement               NO
  whether the diff came from the author's edit          NO
  provider default changes                              NO
  drift, separately from desired change                 PARTIALLY --
      `terraform plan -refresh-only` isolates it, in a SEPARATE run

So the answer is: Terraform's JSON has (d) if you ask for it separately and (e) partially, and it has nothing at all for (a), (b), or (c). The plan describes what will change, not why, and the "why" is the entire content of the question a reviewer is asking.

The buildable approximation, and it is genuinely useful despite the gap:

1. run `terraform plan -refresh-only` FIRST and record the drift set (d)
2. run `terraform plan` against the merge base -> baseline plan
3. run `terraform plan` against the PR head    -> candidate plan
4. the DIFF of the two plans is (a) + (b): changes attributable to
   this pull request
5. anything in the candidate plan that is NOT in the diff and NOT in
   the drift set is (c) or (e) -- flag it as "came along"

Step 4 is the deliverable and it is what a reviewer needs: a plan showing only what this change does. Step 5 is the safety net, and it is where Case Study 1's twelve unprotected resources would have surfaced — as replacements nobody asked for, in a section labelled "you did not request these."

28.24 (Implementation.)

SPLIT: infra-stateful/  and  infra-stateless/

MOVED to stateful (via `terraform state mv` between states, or import+rm)
  aws_s3_bucket.{bronze,silver,gold,scratch}          4
  aws_db_instance.metadata + its subnet/param groups  3
  snowflake_database.analytics + schemas              5
  aws_backup_vault.primary + plan                     2
  aws_efs_file_system.shared                          1
                                                    ───
                                                     15 resources

CROSS-STATE REFERENCES CREATED                        9
  stateless -> stateful, via terraform_remote_state:
    bucket names (4), db endpoint (1), database name (1),
    KMS key arn (1), vpc/subnet ids (2)

Nine cross-state references, all one-directional — stateless reads stateful, never the reverse. That direction is the design constraint that makes the split work, and it is worth enforcing: a stateful resource that depends on a stateless one recreates the coupling the split was meant to remove.

Why to do it before you need to. Moving a resource between states is terraform state mv across two backends, or state rm plus import — an operation with no dry run, no atomicity, and a failure mode where the resource is in neither state and Terraform proposes to create it. With 15 resources and 9 references, it is a careful afternoon; done under pressure, during an incident, on a Friday, it is how a production bucket gets destroyed.

And the benefit is realised on every plan afterwards: the stateless state can be applied freely, by CI, without a human, because nothing in it can destroy data. That is the property worth having, and it is unavailable while one state contains both.

Part VI — Advanced Topics

Chapter 29 — Streaming and Real-Time Processing

29.1 The question: what decision is made on this data, how often is it made, and what does being late cost?

The four answer clusters:

One of the four needs streaming, and it needs it for a structural reason rather than an impatient one: there is no human in the loop. The decision is made by code, at the moment of the event, and there is nothing to wait for. In the other three a person acts on the number, and a person acting at 09:00 does not care whether the number was computed at 04:48 or at 08:59.

The diagnostic question — "if this arrived thirty minutes later, what would go wrong?" — is worth more than the table, because it moves the conversation from a preference to a consequence, and three of the four clusters answer it with some version of "nothing."

29.3 Lambda's fatal problem was that every piece of logic had to be written twice — once in the batch layer and once in the speed layer, in different systems, with different semantics — and the two implementations drifted. The architecture's whole premise was that the batch layer would eventually correct the speed layer, which required them to agree about what the answer should be, which is exactly what two codebases in two languages do not do.

Kappa's problem was the opposite and equally fatal: it assumed you could reprocess history by replaying the log. That works if the log retains everything you will ever need, if the reprocessing throughput is high enough to catch up, and if the transformation's state fits — and at any real history length, none of the three holds. Replaying three years of Kestrel's clickstream to fix a bug is 5.1 billion events through a stateful job, which is not an operation anyone performs on a Tuesday.

What actually replaced them: a lakehouse with incremental batch, plus streaming only where the latency is genuinely required. One codebase, one set of semantics, the transformation expressed once in SQL, and reprocessing done by rebuilding from bronze (Chapter 34's $198.96) rather than by replaying a log.

The piece of Lambda that survives as a technique is the reconciling recompute. Not as an architectural layer — as a periodic batch job that recomputes a window and corrects the incremental result. Chapter 20's lookback window is exactly this, and so is Chapter 38's month-end restatement. Lambda was right that a fast approximate answer needs a slow correct one behind it, and wrong that this required two systems.

29.5 The three questions a watermark forces you to answer:

1. How late can an event be and still be counted? — the watermark's lateness allowance.
2. What happens to an event that arrives after that? — §29.4's three policies: drop, correct, or side-output.
3. What does "the window is closed" mean to a consumer? — when do you emit, and is the emitted value final or subject to revision?

The third is not a streaming question at all. It is a business question about whether a published number may change after publication — the same question Chapter 38 §38.5 asks about restating a closed month, and the same one Chapter 20's lookback asks about a batch model. Whether a figure is allowed to move is a decision finance makes, not a decision an engineer tunes, and a streaming job that emits corrections into a dashboard nobody warned is a policy violation implemented as a default.

Questions 1 and 2 are technical and are answerable by measurement (§29.23(b)). Question 3 has to be asked of a person, and it is the one that gets skipped because it does not look like a configuration value.

29.7 "Exactly-once delivery" is impossible because of §4.1's third outcome. A sender that transmits and receives no acknowledgement cannot distinguish "not delivered" from "delivered, the ack was lost." Its only options are to resend (risking a duplicate) or not to (risking a loss). No protocol removes that choice, because removing it would require the sender to know something the network did not tell it.

"Exactly-once processing" is achievable because it does not require the message to arrive once — it requires the effect to happen once. Deliver at-least-once, and make the effect idempotent: a merge on a key, a partition replacement, or a deduplication set keyed by an event id. The duplicate arrives and produces no second effect.

Where the guarantee stops: at the boundary of the system that provides it.

Kafka -> processor -> Kafka        transactional; genuinely exactly-once
Kafka -> processor -> S3/warehouse two commits, not one. AT-LEAST-ONCE.

A stream processor's exactly-once guarantee covers its own state and its own sinks that participate in its checkpointing protocol. The moment the sink is an external system that does not — object storage, a warehouse, an HTTP endpoint — the offset commit and the write are two separate commits, and a crash between them is possible. (Exercise 15.21 covers the connectors that close this for specific sinks, and every one of them does it by requiring something of the sink.)

29.9 Because the window decides which events are considered related, and relatedness is a claim about the business, not about the system.

join clickstream events to orders, window = 30 minutes
  -> "a page view within 30 minutes of an order is attributed to it"

window = 24 hours
  -> "browsing yesterday and buying today counts as the same journey"

Those are two different definitions of attribution, and they produce different revenue-by-channel numbers. Neither is more correct as an engineering matter; the choice belongs to whoever owns the attribution model.

Tuning it as a parameter has two specific consequences. Widening the window to "catch more joins" silently changes the meaning of every downstream figure, in a direction nobody reviewed. And narrowing it to reduce state size discards real matches — the state cost is a systems concern and the matches are a business one, and trading the second for the first is a decision an engineer should not make alone.

The practical form: write the window into the contract with a sentence of justification (§17.4's semantics field), the same way Kestrel's 30-minute session gap is written down (§6.21) precisely because it is arbitrary and would otherwise be re-litigated annually.

29.12 (Reproduction.) Case Study 1's failure — an idle partition stalling the watermark:

# the bug: the global watermark is the MINIMUM across partitions,
# so one silent partition holds every window open forever.
def test_idle_partition_stalls_the_watermark():
    h = Harness(partitions=3, allowed_lateness=30)
    h.feed(0, events_for("09:00", "09:05"))
    h.feed(1, events_for("09:00", "09:05"))
    # partition 2 sends NOTHING -- a region with no traffic at this hour
    h.advance_processing_time(hours=4)

    assert h.watermark is None          # <- the bug, captured
    assert h.emitted_windows == []      # 4 hours of data, zero output

# the fix: an idle partition must stop holding the watermark back
def test_idle_partition_does_not_stall_after_idleness_timeout():
    h = Harness(partitions=3, allowed_lateness=30, with_idleness=60)
    h.feed(0, events_for("09:00", "09:05"))
    h.feed(1, events_for("09:00", "09:05"))
    h.advance_processing_time(seconds=90)      # past the idleness timeout

    assert h.watermark == ts("09:05")   # partition 2 no longer counts
    assert len(h.emitted_windows) == 1

The two assertions are the deliverable, and they say different things.

The first captures the bug as an absence: emitted_windows == []. Nothing failed, nothing errored, no metric moved — the job ran perfectly and produced nothing, which is why all twenty-two of Chapter 23's assertions passed. There was no output to assert about.

The second captures the fix as the watermark advancing, not as output appearing. Asserting on the output alone would pass for the wrong reason if the harness emitted on a timer; asserting on the watermark tests the mechanism, which is what will still be correct after the next refactor.

29.14 (Implementation.) The three late-data policies:

# 1. DROP -- the default, and the one that must be counted
h = Harness(allowed_lateness=30, late_policy="drop")
h.feed(0, events_for("09:00", "09:05"))
h.advance_watermark_to("09:36")             # window closed
h.feed(0, [event(ts="09:02", value=100)])   # 34 minutes late
assert h.emitted_windows[0].value == BASE   # unchanged
assert h.metrics["late_events_dropped"] == 1     # <- ASSERT THE COUNTER

# 2. CORRECT -- re-emit the window with the revised value
h = Harness(allowed_lateness=30, late_policy="correct")
... same feed ...
assert h.emitted_windows[-1].value == BASE + 100
assert h.emitted_windows[-1].is_revision is True

# 3. SIDE-OUTPUT -- the late event goes somewhere a human can find it
h = Harness(allowed_lateness=30, late_policy="side_output")
... same feed ...
assert h.emitted_windows[0].value == BASE
assert len(h.side_output) == 1
assert h.side_output[0].lateness_seconds == 2040

Asserting the drop counter is the point of the exercise, and the reason is that dropping is the default in every framework. A stream processor with no late-data configuration drops late events silently — the data is gone, no error is raised, and the only evidence is a metric nobody exported.

Three consequences worth stating. A drop policy without a counter is indistinguishable from having no late data at all. A rising drop count is the signal that the watermark is too tight, and it is the only feedback the choice ever gets. And the counter is what makes the choice reviewable — "we drop events later than 30 seconds, and that is 0.02% of events" is a defensible position; "we drop them" is not.

Policy 2 is the one to check against §29.5's third question, because re-emitting a window means a published number changes, and that is not an engineering decision.

29.16 (Implementation.) Sessionisation in the harness:

def test_streaming_sessionization_has_the_same_boundary_problem():
    """Ch. 18 section 18.9's artifact, in a streaming job."""
    h = Harness(session_gap=1800, allowed_lateness=0)   # no lateness
    h.feed(0, [event("a1", "23:50"), event("a1", "23:58")])
    h.restart()                       # or: the job starts here, mid-session
    h.feed(0, [event("a1", "00:05")])
    sessions = h.emitted_sessions()
    assert len(sessions) == 2                    # <- the artifact
    assert sessions[1].event_count == 1          # a phantom 1-event session

def test_the_watermark_solves_what_the_overlap_read_solved():
    h = Harness(session_gap=1800, allowed_lateness=1800)   # >= the gap
    h.feed(0, [event("a1", "23:50"), event("a1", "23:58")])
    h.advance_watermark_to("00:00")   # does NOT close the session:
                                      # allowed lateness keeps it open
    h.feed(0, [event("a1", "00:05")])
    sessions = h.emitted_sessions()
    assert len(sessions) == 1
    assert sessions[0].event_count == 3

The boundary problem is the same problem in both paradigms, and it is worth naming precisely: a window is closed before all of its events have arrived.

In batch, "closed" means the date range of the run, and the fix is to read an overlap and emit only sessions that started inside the window (§18.9's ⚠️).

In streaming, "closed" means the watermark has passed the window's end plus the allowed lateness, and the fix is to make the allowed lateness at least as large as the session gap.

And the two fixes have the same invariant, which is the transferable part:

batch:      overlap_read  >  session_gap
streaming:  allowed_lateness  >=  session_gap

Chapter 18's assert_overlap_exceeds_gap macro and a streaming job's lateness configuration are the same assertion, and both exist because somebody will widen the gap a year from now without knowing the other number exists.

29.18 (Measurement.)

-- end-to-end latency: from the event happening to it being READABLE
SELECT max(date_diff('second', event_ts, _emitted_at)) AS worst_seconds,
       approx_quantile(date_diff('second', event_ts, _emitted_at), 0.99) AS p99,
       approx_quantile(date_diff('second', event_ts, _emitted_at), 0.50) AS p50
  FROM gold.hourly_sku_sales
 WHERE _emitted_at > current_timestamp - interval '7 days';

p50      41 s
p99   1,204 s      (20 minutes)
max  14,882 s      (4 h 8 m)      <- once, on a Sunday

Was it what I expected? No — and the gap is instructive in a specific direction.

The p50 matches intuition and is the number anybody would quote if asked "how fresh is this."

The p99 is 30× the p50, which nobody predicts, and it is entirely the tail of the arrival distribution rather than anything about the processing.

And the maximum is the one that matters, because a consumer's experience of the pipeline is set by its worst day, not its median one. Four hours on a Sunday is an idle-partition stall (§29.12) or a restart, and it is invisible in every percentile anybody publishes.

The reason to measure emission_time − event_time rather than consumer lag is that lag measures the transport and this measures the promise. Lag can be zero while the job emits nothing (Case Study 1); this SLI cannot. It is the only latency number that is about the data.

29.20 The twenty-third assertion: end-to-end latency on the sink, which is the one property no existing entry in the register measures.

name         gold_hourly_sku_sales__emission_latency
measures     max(_emitted_at - event_ts) over the last 24 hours,
             per output partition
lives        in the QUALITY DAG, against the SINK -- not in the streaming
             job, and not in the consumer's lag metric
threshold    ERROR   above 3,600 s   (the sink promises hourly freshness)
             WARN    above 1,200 s
             ERROR   if NO ROW has been emitted in 90 minutes

The last line is the one doing the real work, and it is the entry Case Study 1 needed.

Every one of Chapter 23's twenty-two assertions is a claim about rows that exist: uniqueness, completeness, range, referential integrity, volume. A job emitting nothing produces no rows to violate any of them, so twenty-two assertions passed while the job was useless. The register was complete in the dimension it covered and had no entry for absence.

Two design choices follow, and both generalise past streaming.

It lives against the sink, not the job. A metric published by the job is a metric that stops when the job does — the same reason §24.18's canary monitor is external.

And it asserts on a timer, not on a row. Every other entry runs when data arrives; this one must run whether or not it does, which means it belongs in a scheduled quality DAG rather than in a post-hook. An assertion that only fires when there is data cannot detect the absence of data, and that sentence is the twenty-third entry's entire justification.

29.22 (Design.) A job where the trade goes the other way — where a stall is preferable to dropping data:

Kestrel's hourly financial reconciliation stream, which joins payment-authorisation events to order events and emits a per-hour settlement figure that finance reads.

# NO with_idleness on this source. Deliberately.
#
# with_idleness trades a STALL for DROPPED DATA: an idle partition stops
# holding the watermark back, so anything that partition later sends for
# a closed window is late and is dropped.
#
# For the clickstream (Ch. 29 section 29.23c) that is the right trade --
# a stalled dashboard is worse than 0.02% of page views.
#
# HERE IT IS NOT. The payment-authorisation topic is partitioned by
# acquirer, and one acquirer is genuinely quiet for hours at a time
# overnight. If we let the watermark advance past its silence, its
# morning batch arrives late and is DROPPED -- and the settlement
# figure is short by a whole acquirer, silently, in a number finance
# publishes.
#
# A stalled settlement figure is an incident somebody notices at 09:00.
# A settlement figure missing one acquirer is a number that reconciles
# to nothing, three weeks later (Ch. 38).
#
# DO NOT COPY THIS SETTING FROM THE CLICKSTREAM JOB, AND DO NOT COPY
# THIS ONE THERE. The right answer differs per source, and it depends
# on whether being LATE or being WRONG is worse for the consumer.
#
# Reviewed with finance, 2026-05-12. Revisit if the acquirer's overnight
# volume becomes continuous.

The form §"The Decision" recommends is visible in the last three paragraphs: it states the trade in both directions, names which side this consumer is on and why, tells the next person explicitly not to copy it, and records who agreed and when.

The general principle: with_idleness is not a performance setting, it is a policy about whether late-and-complete beats prompt-and-partial — and that is §26.16's "stale over wrong" question, arriving as a stream-processing configuration key.

29.24 (Implementation.) The five test cases, with the two that matter:

def test_idle_partition_does_not_stall():        # <- Case Study 1
    """One silent partition must not hold every window open."""
    ...  # Exercise 29.12's second assertion

def test_restart_mid_window_resumes_with_state(): # <- missing operator UIDs
    h = Harness(); h.feed(0, events_for("09:00", "09:20"))
    snapshot = h.checkpoint()
    h2 = Harness.restore(snapshot)                # a real restart
    h2.feed(0, events_for("09:21", "09:30"))
    h2.advance_watermark_to("10:31")
    assert h2.emitted_windows[0].event_count == FULL_HOUR
    #        ^ if operator UIDs are missing, state does not restore and
    #          this is ~10 minutes' worth, not an hour's

def test_late_event_within_allowed_lateness_is_counted(): ...
def test_late_event_beyond_it_is_dropped_AND_counted(): ...
def test_duplicate_delivery_produces_one_effect(): ...

Why those two matter more than the other three.

The idle-partition test encodes an incident that produced no signal. The other three failures are detectable in production — a wrong count, a missing correction, a duplicate — and this one is not, because its output is nothing.

And the restart test catches missing operator UIDs in a test rather than during a deploy, which is the only place it can be caught cheaply. Without explicit UIDs, a job's state is keyed by an auto-generated identifier that changes when the topology changes; the job restarts successfully, with empty state, and produces a plausible smaller number. The deploy succeeds. The alert does not fire. The first symptom is a window whose count is wrong by however much state was lost, weeks later, in a reconciliation — and by then nobody connects it to a deploy.

Note that the restart test must actually round-trip through a checkpoint, not merely construct a new harness. A test that skips the serialisation step tests nothing, because the bug is in what survives serialisation.

Chapters 30–33

Solutions and grading notes for Chapters 30 through 33 are in the instructor companion, not here.

Those chapters' exercises are open-ended by design — an audit of your own catalogue, a privacy scan of a system you actually run, a point-in-time join against your own feature store, a cost attribution against a real bill. They have no single correct answer, and a worked answer would substitute this book's platform for yours, which is precisely the substitution the exercises are designed to prevent.

What is available here instead: every one of those chapters ships a self-checking code artifact that asserts its own figures against its own inputs.

python part-06-advanced-topics/chapter-30-data-governance/code/catalog_audit.py --self-check
python part-06-advanced-topics/chapter-31-privacy-engineering/code/pii_scan.py --self-check
python part-06-advanced-topics/chapter-32-ml-engineering-and-feature-stores/code/pit_join.py --self-check
python part-06-advanced-topics/chapter-33-cloud-cost-optimization/code/cost_model.py --self-check

Read the assertions before you read the code. Each one states a claim the chapter makes, in a form that either holds or does not — and working out why a given assertion is the right one to make is closer to the exercise's intent than any answer key would be.

Part VII — Architecture Patterns

Chapters 34–37

Solutions and grading notes for Part VII are in the instructor companion.

These chapters' exercises are audits and designs applied to a platform you run: a layer-boundary check against your own models, a mesh-readiness assessment of your own organisation, an event-schema evolution plan for your own topics, a migration inventory of your own legacy jobs. A worked answer would replace your platform with Kestrel's, and the substitution is exactly what the exercises exist to prevent.

What is checkable here is the code. Each chapter ships a self-checking artifact:

python part-07-architecture-patterns/chapter-34-the-medallion-architecture/code/layer_check.py --self-check
python part-07-architecture-patterns/chapter-35-data-mesh/code/mesh_readiness.py --self-check
python part-07-architecture-patterns/chapter-36-event-driven-architecture/code/event_lab.py --self-check
python part-07-architecture-patterns/chapter-37-migrating-legacy-pipelines/code/migration_lab.py --self-check

Two of them are worth running before attempting the exercises rather than after.

layer_check.py encodes nine rules about what may live in which layer, and running it against the book's fixture shows you which rules are blocking and which are warnings — a distinction that is the subject of Chapter 34's Exercise set and is much easier to reason about with the output in front of you.

mesh_readiness.py scores an organisation against the four preconditions Chapter 35 argues are non-negotiable. Run it against the fixture, then against your own organisation, honestly. Chapter 35's central claim is that most organisations asking for a mesh fail at least two of the four, and the exercise is only useful if you are willing to produce a low score.

Part VIII — Synthesis

Chapters 38–40

Chapter 38 is the capstone, and its answers are deliberately not here.

The reconciliation figures appear in exactly four places in this book — Chapter 38 itself, Appendix I §I.9, the instructor companion, and capstone.py's assertions — and validate.py fails the build if they appear anywhere else. Compute them yourself before you look at any of the four.

python part-08-synthesis/chapter-38-capstone-the-complete-platform/code/capstone.py --self-check

That command is the answer key. It asserts thirty-eight claims about the capstone's figures against the inputs that produce them, and reading the assertions tells you what the exercise wanted without telling you the numbers first — the assertions name the relationships (the ratio agreements, the per-line cross-check against Chapter 1) rather than the totals.

When you have your own figures, check them in this order, because the order is the lesson:

1. Do your two independent ratios agree? Lines against the annual line count, and revenue against annual GMV. If they do not, one of your four rules is wrong, and the disagreement tells you which.
2. Does your revenue-per-line match Chapter 1's independently derived figure? It was not forced by anything in the reconciliation, so agreement is evidence and disagreement is a finding.
3. Only then compare your closing figures to §I.9.

Steps 1 and 2 are the skill. A reconciliation that ties to a published number because you compared it to the published number has demonstrated nothing.

Chapters 39 and 40 have no answer key, and it would be a category error to write one.

Chapter 39's exercises are interview drills — the value is in saying an answer out loud, badly, and then again. interview_drills.py will score the ones that have a right answer:

python part-08-synthesis/chapter-39-the-data-engineering-interview/code/interview_drills.py --self-check

Chapter 40's are about your own career: a skills inventory, a specialisation decision, a five-year sketch. career_map.py computes the maps; the answers are yours.

python part-08-synthesis/chapter-40-the-data-engineering-career/code/career_map.py --self-check

Grading notes for both, for anyone teaching from them, are in the instructor companion.