Chapter 23: Data Quality: Great Expectations, Testing Pipelines, and Why Bad Data Is Worse Than No Data

"A pipeline that fails wakes someone up. A pipeline that lies gets cited in a board deck."

Overview

Twenty-two chapters have accumulated assertions. Chapter 6 gave you the grain. Chapter 13 gave you watermarks and the four ways updated_at lies. Chapter 17 gave you contracts. Chapters 19 through 22 each ended an incident with a test that would have caught it.

This chapter is where those stop being scattered good ideas and become a system. It covers what to assert, where to put it, what threshold to set, who owns it, and — the question most treatments skip — what happens to the rows that fail.

It also settles a question Chapter 19 deferred: dbt has tests, so what is a data quality platform for? The answer is narrower and more useful than either the dbt community or the data-observability vendors will tell you, and §23.5 gives it.

The organizing claim is in the title, and it is worth stating precisely before defending it: a system that produces wrong data is worse than one that produces none, because the failure mode of "no data" is a phone call and the failure mode of "wrong data" is a decision.

23.1 Why Bad Data Is Worse Than No Data

This sounds like a slogan. It is an argument about who bears the cost and when they discover it, and it survives contact with a business partner if you make it in those terms.

When a pipeline fails, the dashboard is empty or stale. Somebody notices within hours — usually the person who opens it at 06:15 (Chapter 1 §1.7). The cost is the delay, it is bounded, and the person who pays it is the person who can escalate it.

When a pipeline lies, the dashboard is populated and plausible. Nobody notices. Decisions are made on it. The cost is unbounded in time — it compounds until discovery — and the people who pay it are downstream of everyone who could have caught it.

Kestrel's own history, from this book:

Every one of those had every job green. Not one was detected by a pipeline failure, because in not one of them did a pipeline fail.

📐 Design Decision — say it in currency, and say it to the right person

"Bad data is worse than no data" is an engineering sentiment. The version that changes a budget is arithmetic, and Chapter 20 Case Study 2 supplies the shape:

$$\text{reconciliation tolerance} \times \text{annual revenue} = \text{the error budget, in dollars}$$

$$0.5\% \times \$182{,}000{,}000 = \$910{,}000\ \text{a year that can be wrong without anyone looking}$$

That number belongs on the same page as the tolerance, and the conversation it produces is different from the one "we should have more tests" produces.

Then make the trade explicit. The choice is not between correct and broken; it is:

• Block on failure — the mart holds yesterday's data. Cost: staleness, measured in hours.
• Publish anyway — the mart holds today's wrong data. Cost: unbounded, discovered later, paid by someone else.

Frame it as a choice about who is inconvenienced, and the answer stops being an engineering preference. A finance partner asked whether they would rather have yesterday's correct number at 06:00 or today's possibly-wrong one will answer immediately, and their answer is the policy.

Chapter 19 §19.7's version: stale beats wrong, and it is not close.

23.2 The Six Dimensions, and Which You Can Test

The standard taxonomy, with an honest note on each about whether an automated test can reach it.

Accuracy is the one that matters most and the only one you cannot test, and pretending otherwise is the central dishonesty of data quality tooling.

A revenue figure can be unique, complete, valid, consistent, timely, and wrong — because the business logic that produced it was wrong, or because the source system recorded something that did not happen. No assertion about the shape of the data reaches that.

What you can do instead, and all three are in this book already:

Reconcile against an independent source. Chapter 20 Case Study 2's payment-processor comparison. This is the closest thing to an accuracy test that exists, and it works because the second system was built by different people for different reasons.

Compute it twice by different paths. Chapter 18 Case Study 2's shadow model. A measurement with only one implementation has no error bar.

Test the logic, not the data. dbt unit tests (Chapter 19 §19.7) assert that given this input, the model produces that output — which is a claim about correctness that no amount of production data will confirm or deny.

23.3 Testing the Pipeline Versus Testing the Data

The single most useful distinction in this chapter, and the one Kestrel's Chapter 1 incident is about.

Chapter 1's duplicate-rows incident had six monitoring checks and all six were green, because all six measured the pipeline: did the DAG complete, did the task exit zero, was the file written, was it non-empty, did it finish in time, did the row count increase. The row count increasing was the bug.

You need both, and the reason people build only the first is that the first comes free with the orchestrator and the second has to be written per table by someone who understands the table.

🏭 From the Pipeline — the check that measures the pipeline while appearing to measure the data

Three checks that look like data tests and are not:

"The output file is non-empty." A file with one row passes. So does a file with yesterday's rows written again.

"The row count increased." Chapter 1's incident increased the row count, every night, for thirty-one days. A monotonic check on an append-only table cannot distinguish growth from duplication.

"The job finished within its SLA." Chapter 21 Case Study 2's job got 4.2× slower and stayed inside the window for eleven days, reporting nothing.

The test for whether a check measures the data: can it fail while the pipeline is completely healthy? If not, it is a pipeline check with a data-shaped name — which is worse than no check, because it occupies the slot where a real one would go and it contributes to a pass rate somebody is reading as evidence.

23.4 The Six Assertions That Earn Their Keep

Not a taxonomy — a list of the six checks that, in this book's experience, catch nearly everything. Write these before you write anything clever.

1. Grain. One row per what? Chapter 2 promised this in nine lines of dbt YAML, and here it is:

models:
  - name: fct_order_item
    tests:
      - dbt_utils.unique_combination_of_columns:
          combination_of_columns: [order_id, line_number]
    columns:
      - name: order_id
        tests: [not_null]
      - name: line_number
        tests: [not_null]

Nine lines, and it is the highest-value test in this book. A violated grain is a fan-out (Chapter 6 §6.9), and a fan-out multiplies revenue rather than perturbing it. Chapter 20 Case Study 1's 3.4× overstatement is this test not existing.

2. Volume floor. Chapter 19 Case Study 1. Every standard test passes on an empty table, so without this a pipeline that lands nothing is entirely green.

      - dbt_utils.expression_is_true:
          expression: "count(*) >= 3000"
          config:
            where: "ordered_at::date = current_date - 1"

Set the floor far below the expected value — Kestrel's daily average is 17,753 and the floor is 3,000. The aim is catching zero, not catching low, and a tight bound pages someone every public holiday and is disabled within a month.

3. Freshness. Chapter 19 §19.8. Configured on the source and asserted on the mart, because a stalled model mid-DAG is a different failure from a stalled load.

4. Referential integrity — including the unknown member. Chapter 19 Case Study 2's finding: a relationships test passes when unmatched keys resolve to -1, so unknown-member volume needs its own assertion.

      - dbt_utils.expression_is_true:
          expression: "count(*) < 50"
          config: {where: "customer_key = -1 and ordered_at::date = current_date - 1"}

5. Distribution. Not the values, the shape: a null rate, a category mix, a mean within bounds. A column that is 2% null every day and 40% null today is a signal that no per-row test reaches.

6. A business rule that someone would argue about. The one test that requires a conversation: "net revenue never exceeds gross," "a shipped order has a shipment record," "no order line has zero quantity." These are the tests that catch a logic change, and they are the ones nobody writes because writing them requires deciding what is true.

🔎 Read the Plan — six assertions, six chapters, one table

Every one of these came out of a real failure earlier in this book, and the mapping is worth seeing in one place because it is an argument about which tests to write first:

Assertion	The incident it would have caught	Cost of not having it
Grain	Ch. 20 CS1 — SCD2 fan-out	revenue overstated 3.4×
Volume floor	Ch. 19 CS1 — a missing ref()	a month closed $498,630.14 short
Freshness	Ch. 19 CS2 — a frozen dimension	6 days of green on dead data
Unknown-member volume	Ch. 19 CS2 — the COALESCE that defeats relationships	1,082 orders misattributed
Anti-join completeness	Ch. 20 CS2 — a watermark with no lookback	$110,560.14 over 8 months
Reconciliation	Ch. 20 CS2 — the tolerance that hid it	0.5% × $182.0M = $910,000/year

None of them is clever. All six are things a competent engineer would agree with if asked, and the reason they are missing is not disagreement — it is that nothing prompts you to write a test for a failure you have not had yet.

Which is the argument for writing them all on day one, and the argument against "we'll add tests as we find problems": the finding is the cost.

23.5 Great Expectations, and What dbt Tests Cannot Do

Chapter 19 deferred this. dbt tests are excellent and they have a specific shape, and knowing that shape tells you exactly when you need something else.

A dbt test is a SQL query that must return zero rows, run against a table in your warehouse, as part of your dbt DAG. Three constraints hide in that sentence:

It runs in the warehouse. Data that has not landed cannot be tested. A CSV on SFTP, a Kafka message, an API response — all outside dbt's reach, and all places where catching a problem is cheaper.

It is a pass/fail on a query. Expressing "the mean of this column is within two standard deviations of its 30-day trailing mean" is possible and unpleasant.

It has no memory. dbt stores results in run_results.json and does nothing with them. A test that should compare today to last month has to build that itself.

Great Expectations (and its peers) addresses those three:

suite.expect_column_values_to_not_be_null("order_id")
suite.expect_column_values_to_be_between("quantity", min_value=1, max_value=100)
suite.expect_column_mean_to_be_between("net_revenue_cents", 2000, 4000)
suite.expect_column_kl_divergence_to_be_less_than(
    "status", partition_object=BASELINE, threshold=0.1)

Four things it gives you that dbt does not:

It validates data in flight, before it lands — a pandas or Spark DataFrame, mid-pipeline.

A rich vocabulary of statistical expectations, including distributional comparisons against a stored baseline.

Persistent results and Data Docs — a browsable history of validations, which is what makes a trend visible rather than a state.

Profiling. Point it at a table and it proposes a suite from what it finds.

⚠️ Failure Mode — the profiler writes the tests that cannot fail

Automatic suite generation is Great Expectations' most attractive feature and its most dangerous one, for exactly the reason Chapter 19 §19.7 gives.

A profiler observes the data and asserts that it continues to look like that. It produces:

• expect_column_values_to_be_in_set listing every value currently present — which will fail on the first legitimate new value and never catch a wrong one
• expect_column_values_to_be_between(min, max) from today's observed minimum and maximum
• expect_table_row_count_to_be_between around today's count, tight enough to fire on a holiday

These encode the data, not the contract. Chapter 17's distinction exactly: a schema describes what arrived; a contract describes what was agreed.

Use the profiler as a first draft and then delete most of it. For each generated expectation: can you describe a plausible upstream change that makes it fail and that you would want to know about? Two out of three will not survive that question.

The failure mode when you keep them all is not false confidence — it is alert fatigue, which ends with the whole suite being ignored, which is strictly worse than not having run the profiler.

When to use which, and the honest answer is mostly dbt:

The last row deserves scepticism. Automated anomaly detection across a warehouse produces a great many alerts about tables nobody depends on, and the value depends entirely on whether your problem is "we have no tests" or "we have no idea what exists." If it is the first, write the six assertions from §23.4 instead.

23.6 Where a Test Lives

The same assertion has different value depending on where it runs, and the rule is: as early as possible, because the cost of a bad row rises as it moves.

SOURCE          →   BRONZE      →   SILVER      →   GOLD        →   CONSUMER
contract        │   schema      │   grain       │   business    │   reconcile
freshness       │   volume      │   referential │   rules       │   against
(Ch. 17)        │   types       │   dedup       │   distribution│   another system
                │               │               │               │
cheapest ───────┴───────────────┴───────────────┴───────────────┴──── most expensive
to fix                                                              to fix

Four placement rules:

Type and format checks belong at the read. Chapter 22 §22.11 — a declared schema at the boundary where the information still exists.

Grain belongs where the grain is established, which is usually silver, and it belongs there rather than in gold because a fan-out found in gold has already been aggregated.

Business rules belong in gold, because that is where the business logic is.

Reconciliation belongs outside the pipeline entirely — a separate scheduled job comparing your number to somebody else's. Chapter 20 Case Study 2.

📏 Scale Note — a test at the source is a test on one table; a test at the mart is a test on many

There is a real economic argument for the opposite of "test early," and it is worth stating because it is why mature platforms end up with more tests in gold than the diagram suggests.

Kestrel has 13 sources, 5 staging models, 2 intermediate, and 4 marts. A distribution test on every column of every source is 13 tables' worth of tests, most of which will never fire. The same budget spent on the four marts covers everything the business actually reads.

The resolution is not a compromise. It is that the two ends catch different things:

• Source tests catch a change, close to its cause, before it has propagated. They are cheap to act on and most of them never fire.
• Mart tests catch a consequence, and they catch consequences of changes you did not anticipate — including ones from sources you do not have a test on.

Both, and weight toward the mart when the budget is tight, because a mart test's coverage is the whole lineage above it. A source test's coverage is one table.

23.7 Thresholds, Severity, and the Test That Gets Turned Off

Every test you write will eventually fire on something benign. What happens next determines whether the test survives.

tests:
  - not_null:
      config:
        severity: warn       # or error
        error_if: ">100"     # error above 100 failing rows
        warn_if: ">0"

Three levels, and the middle one is what makes a test adoptable:

Error — the build stops, downstream models are not rebuilt (Chapter 19 §19.7). For anything where publishing would be worse than being stale.

Warn above a threshold, error above a larger one. The pattern for introducing a test to a codebase that does not yet pass it, and for a check whose normal state is a small non-zero number.

Warn only — and be honest that a warning nothing consumes is a comment (Chapter 19 Case Study 2). If nothing reads it, delete it.

⚠️ Failure Mode — the threshold that fires on a good day

A test that pages someone on Black Friday gets disabled before Cyber Monday, and it is never re-enabled.

Kestrel's traffic varies 6.28× between an average day and Black Friday (Chapter 1 §1.5). Any volume-based threshold set from a typical week is wrong on both tails.

Four ways to build a threshold that survives, in increasing order of effort:

• Set it for catastrophe, not for anomaly. Floor at 3,000 against an average of 17,753 — two orders of magnitude of headroom, and it still catches zero.
• Make it relative, not absolute. "Within 60% of the trailing 28-day median for this day of week" absorbs seasonality that a constant cannot.
• Exclude known events by date. A holiday calendar in a seed file, joined into the test. Boring, explicit, and reviewable.
• Route by severity rather than suppressing. A holiday breach warns; a Tuesday breach pages.

The metric that tells you a threshold is wrong is not its failure rate. It is whether anyone has muted it, and mutes are usually invisible. Audit them. Kestrel found four tests muted in Slack, two of which had been muted for over a year, and one of which was the only check on a table that had been wrong the whole time.

23.8 Anomaly Detection, Honestly

Statistical anomaly detection over your tables — "this column's mean moved three standard deviations" — is what most data observability vendors sell, and it has a genuine use and two genuine limits.

The use: coverage of things nobody wrote a test for, which in a warehouse of hundreds of tables is most of them. This is a real problem and it is what the tooling is good at.

Limit one: it learns whatever it is shown. Chapter 20 Case Study 2's watermark lost 0.037% of orders for eight months. An anomaly detector trained on those eight months learns that 0.037% loss is normal, and reports nothing. It detects change, and a defect that has been present since before the baseline is not a change.

Limit two: seasonality and precision trade off. A detector loose enough not to fire on Black Friday is loose enough to miss a 5% error, and 5% of $182.0M is $9.1M.

The honest positioning: anomaly detection is a net for the unknown, and the six assertions in §23.4 are controls for the known. It is not a substitute for the six, and a team that buys the tool instead of writing them has bought coverage of everything except the failures this book actually documents.

23.9 What Happens to a Bad Row

Almost every treatment of data quality stops at detection. The operational question is what the pipeline does next, and there are three answers.

Block. The load fails, nothing is written, downstream models hold yesterday's data. Correct for a grain violation or a failed reconciliation — anything where publishing is worse than being stale.

Quarantine. Good rows proceed; bad rows go to a side table with the reason attached.

INSERT INTO quarantine.order_items
SELECT *, 'quantity_out_of_range' AS reason, current_timestamp AS quarantined_at
  FROM staged WHERE quantity < 1 OR quantity > 100;

INSERT INTO silver.order_items
SELECT * FROM staged WHERE quantity BETWEEN 1 AND 100;

Correct when the bad rows are a minority and the good ones are useful without them. And it creates an obligation, which is the part that gets missed.

Flag. Everything loads; bad rows carry a marker; consumers decide.

Correct when you genuinely cannot decide for the consumer — a suspicious-but-possible value, a late arrival, a low-confidence match.

🔁 Idempotency Check — a quarantine table that nobody drains is a deletion with extra steps

Quarantine feels like the responsible option because nothing is thrown away. Nothing is recovered either, unless somebody recovers it, and that somebody has to exist.

A quarantine needs four things and usually has one:

1. A row count that is monitored. Growing quarantine is a growing data loss.
2. An owner who is expected to look. §23.12.
3. A replay path — a documented way to fix a row and re-admit it, which must be idempotent (Chapter 20 §20.3), because re-admitting a row that already made it is a duplicate.
4. A retention policy, so it does not grow forever, with an explicit decision about what dropping a quarantined row means.

Without all four you have implemented WHERE quality_is_bad and told yourself otherwise.

Kestrel's rule, and it is a good one: a quarantine table with rows older than 30 days and no owner fails the build. It forces the decision — drain it, own it, or admit the rows are being discarded and write that down.

23.10 What a Test Costs

A chapter arguing for more tests owes an honest account of what they cost, and the honest account is surprising: the money is negligible and the cost is real anyway.

Compute first, because it is the objection people raise and it does not hold. Kestrel's project runs 313 tests, adding about six minutes to a nightly build on a Snowflake Medium (4 credits/hour at the frozen $2.00/credit):

$$4 \times \tfrac{6}{60} \times \$2.00 = \$0.80\ \text{a night} = \$292\ \text{a year}$$

Against the incidents in §23.1's table, that is not a number worth discussing. Anyone declining a test on compute grounds is either running something pathological or arguing about something else.

What tests actually cost, in descending order:

Attention. Every test is a thing a reviewer reads, a thing that can fire, and a thing someone must understand at 03:00. A project's test suite competes for the same scarce resource as its models.

Pass-rate dilution. A decorative test (Chapter 19 §19.7) contributes to a percentage that people read as evidence. Four hundred tests at 100% is a worse signal than forty at 100%, because the denominator is doing the work.

Alert fatigue, which is §23.7's subject and is the mechanism by which a suite stops being read.

Maintenance when the data legitimately changes. A new region, a new status, a new product family — each one is a legitimate change that fires a test, and a suite full of tests that encode the data rather than the contract turns every business change into an engineering ticket.

And the one nobody counts: the test you did not write instead. Effort spent on not_null for a column the warehouse already declares NOT NULL is effort not spent on the anti-join in §23.4.

💸 Cost Check — Kestrel deleted 31 tests and improved its coverage

The manifest_audit.py --decorative pass from Chapter 19 flagged 47 of 313 tests as probably unable to fail. Each was examined against §23.4's question — describe the upstream change that makes this fail — and:

flagged	47
deleted	31 — could not fail, or encoded the data rather than the contract
kept, with a comment explaining what they catch	11
rewritten into something that can fail	5

The five rewrites are the interesting number. Each was a test whose author had a real concern and expressed it in a form that could not detect it — an accepted_values list where a volume assertion was meant, a not_null where a completeness check was meant. The concern was right and the instrument was wrong.

After: 282 tests, and the six-box coverage matrix went from 17 filled boxes of 24 to 24 of 24, because the review that examined 47 tests also revealed which marts had none of the ones that matter.

Fewer tests, complete coverage, and a pass rate that now means something. The audit took one engineer two days.

The generalizable move: audit your suite against a fixed list of what should exist, not against what does. Counting what you have tells you nothing about what is missing, and it is what everyone does.

Two rules that follow:

A test must be able to fail. Chapter 19 §19.7's question, applied on write rather than on audit.

A test must say what to do. §23.12's failure message. A test that fires and produces no action is a test that will be muted, and the mute will outlive the person who set it.

23.11 Data Quality as a Metric

If quality is a goal, it needs a number, and the useful numbers are not the ones people reach for first.

Not useful: the pass rate. 400 tests at 100% for eight months describes an untested project (Chapter 19 §19.7).

Useful, in order:

Coverage of the six. For each mart: does it have a grain test, a volume floor, a freshness check, a referential check, a distribution check, and a business rule? Six boxes, one row per mart. This is the single most actionable data-quality metric available and it takes an afternoon to build.

Data downtime. Hours during which a table was wrong or stale, whether or not anyone noticed — computed retrospectively when an incident is closed. It is the only metric here that captures the undetected period, which is where all the cost is.

Time to detection. From the first bad row to the first alert. Kestrel's table in §23.1 is 6 days, 11 weeks, 5 months, 8 months, 31 days. That distribution is the argument for everything in this chapter.

Mutes. §23.7. A count of currently-suppressed checks, with ages.

23.12 Whose Test Is It?

A test with no owner is a test that fails at 03:00 and is acknowledged by whoever is on call, who cannot fix it and does not know who can.

The rule: a test's owner is whoever can fix what it reports — which is frequently not the person who wrote it.

Rows one and three are the hard ones, because the owner is outside the data team and cannot be assigned by the data team alone. That is Chapter 17's contract doing its work: the owner and consumers fields exist precisely so that a failing check has somewhere to go.

And the operational version: every test's failure message should name what to do, not what happened.

-- Bad:  "unique_combination_of_columns failed on fct_order_item"
-- Good: "fct_order_item has duplicate (order_id, line_number).
--        Likely cause: int_order_items_deduped lost its tiebreaker
--        (Ch. 18 §18.7). Runbook: docs/runbooks/grain-violation.md.
--        Owner: #data-eng"

23.13 The Register: Every Assertion This Book Has Introduced

Consolidated, because scattered across twenty-two chapters they are advice, and in one place they are a checklist.

Twenty-two assertions and not one of them is difficult. Every one came from a failure that happened, and the total cost of the failures in this table exceeds two million dollars of Kestrel's money.

Assertions 19, 20, and 21 are not about data at all, and they are in the register deliberately: an operational property that degrades silently produces a data failure eventually, and the boundary between "data quality" and "reliability" is a team-structure artifact rather than a real distinction.

🎓 Interview Angle — "how do you know the data is right?"

The most important question in a data engineering interview and the one candidates are least prepared for, because it has no tool-shaped answer.

The weak answer names a tool. "We use dbt tests" or "Great Expectations." That says what you ran, not what you know.

The strong answer draws the distinction and then gets specific:

"There are two different questions and most monitoring answers only the first. 'Did the job run' is about the pipeline; 'is the answer right' is about the data, and the test is whether a check could fail while the pipeline is completely healthy. I've seen an incident where six checks were green for thirty-one days and revenue was 11% high — one of the six was 'did the row count increase', and the row count increasing was the bug. So concretely: a grain test on every fact table, a volume band, a freshness check, and — the one that actually matters — a reconciliation against something the pipeline didn't produce."

Four things that answer does. It names the distinction. It gives a concrete failure with a number. It names the specific assertion that was wrong, which is the detail that makes it credible. And it ends on reconciliation, which is the answer to the question actually asked.

The follow-ups, and what each is checking:

"What do you reconcile against?" — something with an independent derivation. A candidate who reconciles gold against silver has not understood the question, because both come from the same pipeline.

"What does your reconciliation not cover?" — an error that affects both sides identically, a revenue stream that is not in the comparison, and anything below the tolerance. This is the question that separates people who have run one from people who have designed one on a whiteboard.

"How do you stop a test being turned off?" — thresholds that survive, severity levels, and an honest concession that every test you write will eventually fire on something benign, and what happens next determines whether it survives.

And one to volunteer if you get the chance: the sign test. "A tolerance band can't see a small error that's in the same direction every day. Twenty-six same-signed nights is one in thirty-three million under a fair coin, and it needs no threshold at all." Very few candidates say this, and it is the sharpest thing in this chapter.

🧪 Try It — count the tests that cannot fail

Twenty minutes, on your own project, and every cohort finds at least one.

bash cd platform/transform/kestrel_dbt dbt ls --resource-type test --output json | jq -r '.name' | sort

Take ten of them at random and ask §19.7's question of each: what would have to be true in the data for this to fail, and could that actually happen?

text test could it fail? why not ──────────────────────────────────────────────────────────────────────── unique(order_line_id) yes not_null(net_revenue_cents) NO the column is coalesce(x, 0) accepted_values(status) yes -- and this is the one that caught the enum not_null(_dbt_loaded_at) NO the model sets it relationships(customer_sk) yes unique(surrogate_key) NO* generated by the model itself

The starred row is the interesting one, because Exercise 19.20 argues it should sometimes stay: a tautology that guards an invariant somebody might remove is not the same as one that guards nothing. The test is not "can it fail today" but "if it failed, would anyone care."

Then run the second half of the audit, which is the uncomfortable one:

```bash

which tests have EVER failed, in your run history?

jq -r 'select(.status=="fail") | .unique_id' target/run_results.json

...aggregated over the last 90 days of stored run_results

```

A test that has never failed in ninety days is either protecting something stable or is incapable of firing, and the two are indistinguishable from the outside. Cross-reference that list with your answers above — the intersection is the set to delete or fix.

Record the number. "Of 61 tests, 9 cannot fail and 4 more have never fired and I cannot construct an input that would make them" is a coverage figure that means something, and it is the honest version of the number a dashboard would report.

🧭 Version Note — the tooling changed and the argument did not

Data quality has had three tooling eras in a decade, and the useful skill is separating what each tool does from what it claims.

text era the shape what it added ───────────────────────────────────────────────────────────────────────── ~2015 hand-written SQL assertions nothing was standard; everyone in a cron job wrote it once, badly ~2018 Great Expectations a vocabulary, and validation OUTSIDE the warehouse ~2020 dbt tests assertions beside the models, in the same command, in git ~2022 "data observability" products anomaly detection, lineage, and freshness, bought now unit tests (dbt 1.8+), testing LOGIC without data, contracts, model versions which none of the above did

Two of those rows are genuinely new capabilities and the rest is packaging.

Validation outside the warehouse is a real gap that dbt cannot fill (§23.5). A malformed CSV has already been loaded by the time dbt sees it, and the leading-zero postal code (Chapter 22's Case Study 2) is a landing-time failure.

And unit tests are the first mechanism that tests a rule rather than today's data. A gift-card exclusion can be tested against three invented rows, in CI, in milliseconds — and it fails when somebody changes the rule even if no gift card happens to be in the current data. Nothing before could do that.

What has not changed, at all: §23.3's distinction, the six assertions that earn their keep, and the fact that a test which cannot fail is not a test. Every era's tooling has been used to build impressive coverage numbers out of assertions that could never fire.

The reading rule: when you meet a data quality product, ask which of §23.3's two questions it answers. Freshness and volume monitoring answers the first extremely well and is frequently sold as answering the second. Anomaly detection is a genuinely useful third thing (§23.8) and it is not a substitute for an assertion, because it can only tell you a number is unusual and never that it is wrong.

🔐 Privacy & Governance — a failing test writes the failing rows somewhere

--store-failures is one of the most useful flags in dbt and it is a data copy.

text dbt build --store-failures -> a test that finds 412 rows with a null customer_sk writes those 412 ROWS to a table in your warehouse -> the table is named after the test, lives in a `dbt_test__audit` schema, and is retained until the next run overwrites it -> it contains whatever the failing rows contain, including identifiers

Three consequences worth stating.

The audit schema is a copy of your worst data, and nobody classified it. It is created by tooling, named by tooling, and it does not appear in the catalog or the deletion manifest. It is the exact shape of object Chapter 31's generated manifest exists to catch — and it only catches it if the schema is tagged.

In CI, the failures are printed to a log. A test that fails in a pull request prints rows to a build log, which is retained by the CI provider, visible to everyone with repository access, and outside every control you own. Exercise 27.23(f) says to use --store-failures so CI never prints production rows — and the corollary is that the stored table then needs a retention.

And the quarantine has the same shape (§23.9), for the same reason: it exists to hold the rows that failed, which are disproportionately the rows with missing or malformed identifiers — exactly the rows a deletion job keyed on a clean identifier will fail to find.

Three lines that resolve all of it:

text 1. tag the test-failure schema and the quarantine as containing personal data, at creation 2. give both a retention -- 30 days -- and enforce it 3. never print failing rows in CI; store them, and link to the table

The general observation, which recurs in Chapter 24 and Chapter 4: the artifacts that exist to make failures debuggable are the artifacts nobody governs, because they are created by tooling rather than by a design decision.

23.14 The Kestrel Platform

🧱 Kestrel Platform — Increment 23: the quality layer

text platform/quality/ coverage.py ← the six-box matrix, per mart mute_audit.py ← currently-suppressed checks, with ages reconcile_daily.py ← warehouse vs. payment processor, with a variance series great_expectations/ expectations/supplier_inventory.json ← pre-warehouse validation platform/transform/kestrel_dbt/ models/**/_*.yml ← the six assertions per mart tests/ ← singular tests for the register's 7, 8, 10-14

Six things this increment must get right:

1. coverage.py prints the six-box matrix and fails on any mart missing a box. Start with it — it tells you how much of the rest there is to do.
2. Every threshold in the project has a comment stating the gap it sits in — the normal value, the threshold, and the broken value. Chapter 20 Case Study 1's was 400 / 500 / 62,000.
3. mute_audit.py runs weekly and reports any suppression older than 30 days.
4. reconcile_daily.py publishes the variance as a time series, not a pass/fail. Chapter 20 Case Study 2's 0.02% → 0.037% step change is only visible this way.
5. Great Expectations validates the supplier CSV before it lands, which is the one place dbt cannot reach and where Chapter 22 Case Study 2 happened.
6. Every quarantine table has an owner, a monitored count, an idempotent replay path, and a retention decision. Or it does not exist.

The exercise that matters is 23.23(a): run coverage.py against the project as it stands after Chapter 22 and report the number of empty boxes. It will not be zero, and the gap between what this book has told you to do and what the project actually has is the honest starting point.

23.15 Summary

Bad data is worse than no data because of who pays and when. A failure costs a bounded delay borne by someone who can escalate. A lie costs an unbounded, compounding error borne by people downstream of everyone who could have caught it. Every one of Kestrel's five documented incidents had every job green.

Say it in currency. A 0.5% reconciliation tolerance on $182.0M is $910,000 a year that can be wrong without anyone looking, and that number changes the conversation.

Five of the six quality dimensions are testable. Accuracy is not. Reconcile against an independent source, compute it twice by different paths, and unit-test the logic — those are the substitutes, and pretending an assertion about shape reaches truth is the central dishonesty of the tooling.

Testing the pipeline misses every failure where the code worked and the answer is wrong. The test for a real data check: can it fail while the pipeline is completely healthy?

Six assertions earn their keep: grain · volume floor · freshness · referential integrity including unknown-member volume · distribution · a business rule someone would argue about. The grain test is nine lines of YAML and is the highest-value test in this book.

dbt tests cover about 80%. They run in the warehouse, are pass/fail on a query, and have no memory. Great Expectations adds pre-landing validation, statistical vocabulary, and history — and its profiler writes exactly the tests that cannot fail, so use it as a first draft and delete most of it.

Test as early as possible, because a bad row gets more expensive as it moves — but weight toward the mart when the budget is tight, because a mart test's coverage is the whole lineage above it.

Set thresholds for catastrophe, not anomaly, make them relative where seasonality is real, and audit the mutes — the metric that tells you a threshold is wrong is not its failure rate, it is whether anyone has silenced it.

Anomaly detection is a net for the unknown, not a control for the known. It learns whatever it is shown, so a defect present before the baseline is invisible to it — and Chapter 20 Case Study 2's eight months of 0.037% loss is precisely that shape.

Decide what happens to a bad row: block, quarantine, or flag. A quarantine without a monitored count, an owner, an idempotent replay path, and a retention decision is WHERE quality_is_bad with extra steps.

A test's compute cost is negligible — 313 tests cost Kestrel $292 a year — and its real costs are attention, pass-rate dilution, alert fatigue, maintenance, and the test you did not write instead. Kestrel deleted 31 tests and went from 17 filled coverage boxes to 24, because the review that examined the suite also revealed what was missing. Audit against a fixed list of what should exist, not against what does.

Measure coverage of the six, data downtime, time to detection, and mutes. Not the pass rate.

A test's owner is whoever can fix what it reports, which is often outside the data team — and Chapter 17's contract is what makes that assignable.

The register in §23.13 has twenty-two assertions, every one from a failure that happened, and their combined cost exceeded two million dollars. None of them is difficult.

Part V begins with Chapter 24 and orchestration: what decides when any of this runs, what happens when it does not, and why "just use cron" stops working at about the fourth dependency.

Key terms: completeness · uniqueness · validity · consistency · timeliness · accuracy · testing the pipeline · testing the data · grain test · volume floor · freshness check · unknown-member volume · distribution test · business rule · Great Expectations · expectation suite · profiler · severity · threshold · mute · quarantine · replay · data downtime · time to detection