Chapter 17: Schema Evolution and Data Contracts

"They didn't break us. They shipped a good change, on a Tuesday, and nobody had written down that we were reading it."

Overview

Part III has been a catalogue of one problem in different costumes.

A column renamed on a Tuesday afternoon (Chapter 2 §2.2). A sign convention silently inverted (Chapter 2 §2.2). A postal_code widened from integer to text, nulling 12% of customers (Chapter 13 §13.7). A carrier changing a field from a string to an object, dead-lettering 2.1 million messages (Chapter 15's Case Study 2). An API's response shape shifting under a client that had worked for eight months (Chapter 16).

Every one of those was a producer making a reasonable change that a consumer depended on, without either side knowing the dependency existed.

A data contract is the mechanism that makes the dependency explicit, versioned, and enforced. It is not a schema — a schema is one field of it — and it is not primarily a technology. §17.7 argues that contracts fail socially far more often than technically, which is the part most treatments skip and the part that decides whether yours works.

The chapter's central claim is about where a failure lands, not whether one happens. Changes will happen; producers must be able to evolve. The question is whether the incompatibility surfaces:

• at the producer, at write time, as a failed deploy — cheap, immediate, and owned by the person making the change; or
• at the consumer, weeks later, as a wrong number in a dashboard — expensive, delayed, and owned by someone who did nothing.

That is the whole argument. Everything else is mechanism.

In this chapter, you will learn to:

• State the problem in terms of where a failure lands.
• Distinguish backward, forward, and full compatibility and choose a mode.
• Classify a change as compatible or breaking under each.
• Name the seven things a contract contains beyond a schema.
• Place enforcement at four points and say what each catches.
• Design a versioning scheme that permits a breaking change without an outage.
• Explain the social failure modes and what to do about them.
• Write a contract for a source you do not control.

Who needs this chapter: everyone. It is the chapter that converts "they broke us again" from a recurring event into a caught test.

17.1 The Problem, Stated Precisely

Three parties and one missing artifact.

   PRODUCER                    THE DATA                    CONSUMER
   the orders service    ──▶   orders table / topic  ──▶   fct_order_item
                                                            22 dashboards
                                                            2 ML features

   Knows: its own code          Knows: nothing              Knows: what it reads
   Does not know: who reads it                              Does not know: when
                                                            the shape changes

The producer cannot know who depends on what, because nothing records it. The consumer cannot know when the shape will change, because nothing announces it. And the data itself carries no statement of intent — a column is present or absent, and nothing says whether that was a promise.

A contract is the missing artifact. It sits between them, is owned by the producer, is discoverable by consumers, and is enforced by something automated — because a contract that is only a document is a wish.

What it is not

Three common confusions:

A contract is not a schema. A schema says what fields exist and their types. A contract says that, plus what they mean, what is guaranteed, who owns it, how it will change, and what happens when it does. §17.4.

A contract is not a restriction on the producer. It does not say "you may never change this." It says "here is how change happens, and here is what will break if you do it a different way." A contract that forbids change gets routed around.

A contract is not primarily a technology. A schema registry enforces one field of it. The rest — ownership, meaning, deprecation process — is written down and agreed, and that half is where they fail.

17.2 Compatibility: Backward, Forward, Full

The technical core, and the terms are used inconsistently enough to be worth pinning down.

The question is always: can this reader read that data?

The mnemonic that actually works: the direction names what the new schema is compatible with. Backward compatible = compatible with data written backward in time. Forward compatible = compatible with readers from forward in time, i.e. readers you have not upgraded yet.

What each mode permits

Two rows deserve comment because they surprise people.

Adding a required field is backward compatible and not forward compatible. A new reader reading old data finds the field missing — which is fine only if it has a default, and a required field by definition does not. In practice, adding a required field is a breaking change in every mode and the table's ✅ is a technicality about how Avro resolves it.

Adding an enum value is forward and not backward compatible. An old reader encountering a new value it does not know about handles it (forward ✅); a new reader is fine with old data (backward ✅ in principle) — but many implementations reject unknown symbols, so adding an enum value is frequently breaking in practice while being compatible on paper. This is the single most common "it should have worked" incident with registries.

Choosing a mode

Backward is the default and the right one for most event topics. It means you can deploy a new consumer against a history of old data — which is what replay (Chapter 15 §15.1) requires.

Forward is right when producers must move first and consumers cannot be coordinated — an event emitted by a mobile app whose users upgrade on their own schedule.

Full is right for anything long-lived and multi-consumer, and it is restrictive: it permits adding and removing optional fields and essentially nothing else.

Transitive variants — BACKWARD_TRANSITIVE, and so on — check compatibility against every previous version rather than only the last. Use the transitive form if consumers may read arbitrarily old data, which for a replayable log they can.

17.3 The Schema Registry

A registry stores schemas by id, assigns ids, and — the part that matters — refuses to register a schema that violates the configured compatibility mode.

producer                        registry                     consumer
   │  1. register schema  ──────▶  check compatibility
   │                              ┌─ compatible → return id 4182
   │                              └─ NOT → 409 CONFLICT ← THE POINT
   │  2. write [id 4182][payload]
   ▼
 topic ─────────────────────────────────────────────────────▶ │
                                                   3. read id 4182
                                 ◀───── 4. fetch schema ──────┤

Step 1's rejection is the entire value. A producer attempting an incompatible change gets a 409 at deploy time, in CI, with a message naming the incompatibility. The bad data never exists.

Compare the alternative from Chapter 11 §11.5: without a registry, the same change succeeds, the data is written, and the failure surfaces in a consumer weeks later — as a null column, a cast error, or a dashboard that is quietly wrong.

Three practical points:

The message carries an id, not a schema. Five bytes, not a kilobyte of JSON per message. At Kestrel's 14 million events a day that is the difference between negligible and significant.

The registry is a runtime dependency of every consumer. It caches aggressively, and a registry outage during a consumer restart with a cold cache is a real failure mode. Run it with the availability you would give a database.

A registry only enforces the schema field of the contract. Ownership, meaning, freshness, and deprecation process are not schema and the registry knows nothing about them.

And a registry is not the only place this check can live. For files, the table format does the same job:

mechanism                enforces at        rejects
──────────────────────────────────────────────────────────────────────────
schema registry          producer deploy    an incompatible schema, in CI
Iceberg / Delta          write time         a type change; a dropped required column
bare Parquet             nothing            nothing -- it stores what it was given
dbt on_schema_change     model build        depends entirely on the setting (ch20 §20.5)

Iceberg and Delta enforce schema on write, which is the registry's rejection moved to a different moment: a job writing a string into an int column fails the write rather than producing a file that a reader will choke on next Tuesday. This is one of the stronger arguments for a table format over bare Parquet (Chapter 10), and it is rarely the one people lead with.

Bare Parquet enforces nothing. Each file embeds its own schema, and one directory can hold files whose schemas disagree. The reader discovers this — as an error on some engines, as a silent null on others, and sometimes differently depending on which file it happened to read first.

The shape is the same in all three cases: something must be able to say no at write time. Where you put that "no" decides who gets paged.

🧭 Version Note — BACKWARD and BACKWARD_TRANSITIVE are different, and the default is the weaker one

Registries default to BACKWARD, and BACKWARD checks a new schema against the latest version only. BACKWARD_TRANSITIVE checks it against every version ever registered.

Three changes that each pass, and a chain that does not:

```text v1 {order_id, discount: string} "10%" v2 {order_id} discount removed BACKWARD vs v2's parent: ok v3 {order_id, discount: int} re-added, as int BACKWARD vs v2: ok

a v3 reader over v2 data discount defaults fine a v3 reader over v1 data int reader, string on disk -> FAILS ```

Every step was legal and the chain is not. Delete a field, re-add it with a different type, and the registry never sees the two schemas that are actually incompatible.

If your consumers can read arbitrarily old data — and for a replayable log (Chapter 15) they can — you want the transitive mode, and you want it set before the first schema is registered rather than discovered during a replay eighteen months later.

Check the default in your registry's current version. It differs between products, it has changed, and it is settable per subject as well as globally — so the global setting you verified may not be the one your topic is using.

17.4 What a Contract Contains

Seven things beyond the schema. The schema is the easy part and the least likely to be the source of a dispute.

# platform/contracts/orders.v2.yml
contract: kestrel.orders
version: 2.1.0
status: active                    # draft | active | deprecated | retired

# 1. OWNERSHIP -- a team, not a person, and a channel that is monitored
owner:
  team: checkout-engineering
  slack: "#checkout-eng"
  escalation: "#checkout-oncall"

# 2. CONSUMERS -- the thing the producer could not otherwise know
consumers:
  - {name: "data-platform / silver.orders", team: data-engineering,
     criticality: high, contact: "#data-eng"}
  - {name: "finance-reporting", team: finance-systems, criticality: high}
  - {name: "fraud-scoring", team: ml-platform, criticality: medium}

# 3. THE SCHEMA
schema:
  format: avro
  registry_subject: kestrel.orders.cdc.v1-value
  compatibility: BACKWARD_TRANSITIVE

# 4. SEMANTICS -- what the fields MEAN. The half no schema can express.
semantics:
  order_id:    "Immutable. Never reused, including after deletion."
  status:      "One of pending|paid|picked|shipped|delivered|cancelled|refunded.
                Transitions are forward-only EXCEPT refunded, reachable from
                delivered. New values require a MINOR version and 30 days notice."
  placed_at:   "When the customer submitted, UTC. NOT when payment authorised."
  total_cents: "INTEGER CENTS. Merchandise + shipping + tax, minus discount.
                Excludes refunds -- see the returns contract."

# 5. GUARANTEES -- what a consumer may rely on
guarantees:
  freshness_p99_seconds: 60       # from source commit to topic
  completeness: "every state transition is emitted; at-least-once"
  ordering: "per order_id, guaranteed"
  volume_per_day: {typical: 6575, peak: 41300}   # so consumers can size
  availability: "99.9% monthly"

# 6. CHANGE POLICY -- how change happens, not whether it may
change_policy:
  additive_fields: "no notice required"
  semantic_change: "MINOR version, 30 days notice in #data-eng"
  breaking_change: "MAJOR version, new topic, 90 days dual-write, then retire"
  emergency: "notify #data-eng within 1 hour; retrospective within 5 days"

# 7. VALIDATION -- what is checked, and where
validation:
  producer_ci:  ["schema compatibility", "required fields present"]
  producer_run: ["schema id resolves", "status in the declared enum"]
  consumer_run: ["freshness within SLA", "volume within 3x trailing band"]

Field 2 is the one that changes behavior. The producer's central problem in §17.1 is that they cannot know who depends on them. A consumer list — kept current, because being on it is what entitles you to notice — solves exactly that.

Field 4 is the one that prevents the worst incidents. Chapter 2 §2.2's discount_cents sign inversion was a semantic change with no schema change at all. No registry can catch it. A written statement of meaning, reviewed when the field's producer changes, can.

17.5 Enforcement: Four Places a Contract Can Bite

A contract that is only a document is a wish. Four enforcement points, and each catches something the others cannot.

1. Producer CI — before merge. Schema compatibility against the registry, run as a pull-request check. The cheapest and most valuable point: the change never merges, the author sees it in context, and no data is written.

- name: Check schema compatibility
  run: |
    curl -sf -X POST \
      "$REGISTRY/compatibility/subjects/kestrel.orders.cdc.v1-value/versions/latest" \
      -H "Content-Type: application/vnd.schemaregistry.v1+json" \
      -d @schema.json | jq -e '.is_compatible == true'

2. Producer runtime — before write. The serializer registers or resolves the schema and refuses incompatible data. Catches what CI missed: a code path that constructs a message differently, a configuration difference between environments.

3. Consumer runtime — after read. Assert the fields you depend on are present and typed as expected. This is Chapter 14's shape assertion and Chapter 16's contract test, generalized: it catches a producer who bypassed the contract, and a source you do not control.

4. Scheduled — against the guarantees. Freshness, volume, and distribution against the contract's declared numbers. This is the only point that catches a guarantee being violated without any schema change at all — a producer that is compliant and has stopped, or slowed, or halved its volume.

📐 Design Decision — Enforce at the producer, or validate at the consumer?

Both, and the interesting question is which you build first when you can only build one.

Producer enforcement is strictly better when available: the bad data never exists, the failure lands on the person making the change, and it is immediate. If you own the producer, do this.

Consumer validation is the only option when you do not own the producer — a third-party API, an acquired system, a team that will not adopt a registry. It is worse in every respect: the bad data exists, the failure lands on you, and it is delayed by however long your pipeline takes.

And consumer validation is what you build first anyway, for a reason that is organizational rather than technical: you can build it unilaterally, this afternoon, without anyone's agreement. Producer enforcement requires the producer team to adopt something, which is a conversation, a roadmap slot, and a negotiation.

The practical sequence: consumer validation now, to stop the bleeding and to generate evidence — a log of every time the producer changed something and broke you is the single most persuasive input to the conversation about producer enforcement.

What consumer-first costs: you absorb the failures in the meantime, and you may build validation that a later registry makes redundant. Both acceptable.

17.6 Versioning

Semantic versioning, adapted:

A MAJOR version is a new topic or table, not a modification of the existing one:

kestrel.orders.cdc.v1   ← v1 consumers keep reading, undisturbed
kestrel.orders.cdc.v2   ← the producer dual-writes for the migration window

Expand-contract

The pattern that lets a breaking change happen without an outage, and it is the whole answer to "how do we ever rename a field."

  1. EXPAND    add the new field alongside the old. Both populated.
               Backward compatible. Nobody breaks.

  2. MIGRATE   consumers move to the new field, at their own pace,
               over the notice period. The contract's consumer list
               is how you know when everyone has.

  3. CONTRACT  remove the old field. A MAJOR bump, and by now
               nothing reads it.

Step 2 is the one that takes the time, and it is where the consumer list earns its place: without it, step 3 is a guess. Kestrel's policy is 90 days of dual-write for a MAJOR change, and step 3 does not happen until every consumer on the list has confirmed migration or been removed.

The cost is real: for the migration window the producer writes both fields, the contract carries a deprecated field, and consumers see two ways to get the same thing. That is the price of never having a coordinated outage.

17.7 The Social Half

Contracts fail socially more often than technically, and this section is the one that decides whether yours works.

Four failure modes, and the fixes are not technical.

1. The contract is written once and never updated. A stale contract is worse than none, because consumers rely on it. Fix: the contract lives in the producer's repository, next to the code, and changing the schema without updating the contract fails CI. Proximity is what keeps documents alive.

2. Nobody knows the contract exists. Fix: it is discoverable from the data — the catalog entry for silver.orders links to it, the registry subject references it, the dbt model documents it. A contract in a wiki nobody links to is a wish.

3. The producer sees it as an imposition. This is the important one, and it is usually earned: a contract presented as "you may not change your system without asking us" will be resisted, and should be. Fix: frame it as what it is — a statement of what the producer already provides, so they can change things without being surprised by an angry data team. The consumer list is the producer's benefit, not the consumer's: it is how they find out who to tell.

4. Violations have no consequence. A contract nobody enforces is a document. Fix: §17.5's four points, and — critically — a violation is an incident with a retrospective, not a Slack message.

🏭 From the Pipeline — The contract that changed a relationship

A data team introduced contracts for three upstream services. Two teams adopted them within a quarter. The third refused, politely and firmly: they had a heavy roadmap and no capacity for "documentation for another team's benefit."

The data team did not escalate. They did three things instead:

1. They wrote the contract anyway, from observation, and marked it status: observed rather than agreed — an explicit statement that this describes what the producer does, not what they have promised.

2. They enforced it at the consumer (§17.5 point 3) and logged every violation.

3. Eight months later they brought the log to a conversation. Nineteen violations. Six had caused incidents. Four had cost the producer's own team time, because the data team's investigation had pulled them in to answer questions.

The producer team adopted the contract that quarter, and the reason they gave was the fourth point: they had been spending time on incidents caused by their own changes, and had not connected the two.

The lesson: an observed contract is a legitimate artifact and it generates the evidence that makes the agreed one possible. Do not wait for agreement to start. And note what did not work: asking, twice, with a good argument.

17.8 Contracts for Sources You Do Not Control

A third-party API, an acquired system, a vendor database. You cannot make them agree to anything.

Write the contract anyway, marked status: observed, and enforce it at the consumer:

contract: carrier-b.tracking
version: 1.0.0
status: observed          # ← NOT agreed. This describes behaviour, not a promise.
owner:
  team: carrier-b         # external
  note: "No agreement exists. This documents observed behaviour and is
         maintained by data-engineering. Support: support@carrier-b.example"

observed_guarantees:      # ← measured, with the date, not promised
  freshness_p99_seconds: 340      # measured 2025-11-01..2025-11-30
  volume_per_day: {typical: 98000, min_observed: 71000, max_observed: 141000}
  retroactive_change_window_days: 30   # measured -- Ch. 16 §16.8

validation:
  consumer_run: ["required fields present", "types unchanged",
                 "volume within observed band", "freshness within 2x p99"]
  scheduled:    ["nightly contract test -- Ch. 16 §16.9",
                 "monthly reconciliation vs. carrier summary"]

Three things this gives you even with no counterparty:

A place to record what you have measured. The observed guarantees are the numbers from Chapter 16's re-fetch window measurement, written down where the next engineer will find them.

A definition of "broken." Without a stated expectation, "the carrier changed something" is a feeling. With one, it is a check.

Evidence. The §17.7 log. If the vendor relationship ever becomes negotiable, you have nineteen dated violations rather than an impression.

17.9 The Kestrel Orders Contract

The contract in §17.4, in production, with the parts that matter in practice.

What it changed, measured over eighteen months:

The two that still broke are worth naming, because they are the residual and they show the limits:

1. A semantic change with no schema change. placed_at began being set at payment authorization rather than at submission, following a checkout refactor. The schema was identical. The contract's semantics field said what it should mean, and nothing enforced that — it was caught in review of the contract, three weeks later, by a human.

§17.4's field 4 documents the meaning; nothing validates it. That gap is real and this book does not have a good automated answer. The best available is a distribution check: placed_at moving systematically later relative to paid_at is detectable (Chapter 25 §25.4), and detecting it requires having thought to look.

2. An enum value added. status gained awaiting_stock. Backward compatible by the registry's rules, and the consumer's CASE statement had no branch for it, so those orders fell into ELSE and were counted as cancelled.

This is §17.2's enum footnote, in production. The contract's semantics field said "new values require a MINOR version and 30 days notice" and that policy was followed — a MINOR bump, notice given, in a channel, which nobody read.

The fix was a consumer-side assertion, not a process change:

-- Fails the build on an unknown status. Ch. 17 §17.5 point 3.
SELECT status, COUNT(*) FROM silver.orders
 WHERE status NOT IN ('pending','paid','picked','shipped','delivered',
                      'cancelled','refunded')
 GROUP BY 1;
-- expect zero rows

Nine lines, and it converts a silent miscount into a failed build. The general form — assert against the declared enumeration rather than trusting it — belongs on every categorical column whose values a contract enumerates.

🧪 Try It — break the contract three ways

bash cd part-03-ingestion/chapter-17-schema-evolution-and-data-contracts/code python contract_check.py --self-check

Then edit the contract fixture and re-run after each change:

1. Widen postal_code from int to string (Chapter 13 §13.7's incident). Which compatibility modes accept it, and which reject it?
2. Add awaiting_stock to the status enum. Watch the compatibility check pass, then watch the consumer assertion above fail. The second failure is the useful one, and it is the whole argument of §17.5's point 3.
3. Change placed_at's semantics line and nothing else. Nothing fails. That is this section's first residual incident, reproduced in ten seconds, and it is the honest limit of tooling.

17.10 When Contracts Are Not Worth It

Four cases.

1. One producer, one consumer, same team. The contract is a conversation. Write down the semantics; skip the ceremony.

2. The data is genuinely exploratory. A dataset someone is prototyping with, which may not exist next month. Contracts on exploratory data slow down the exploration and rarely survive it.

3. You cannot enforce it anywhere. A contract with no enforcement point (§17.5) is a document, and a document that people believe is worse than nothing. Either find an enforcement point or be explicit that it is observed.

4. The producer changes daily and the consumer is tolerant. A log stream consumed by a schema-on-read exploration tool. The cost of a contract exceeds the cost of the breakage.

Notice what is not on the list: "the producer is a different team," "the producer is external," and "the producer will not agree." All three are reasons to write an observed contract, not reasons to skip it.

17.11 Starting From Nothing: The First Ninety Days

Most teams that need contracts have none, no registry, and no authority over the producers. The literature describes the end state and skips the transition, which is the part that is actually hard.

The sequence that works starts at the consumer and ends at the producer — the opposite of how contracts are usually described.

Weeks 1–2: write down what you already depend on. Not a contract; an inventory. For each upstream table or topic, the columns you actually read, the values you assume, and the freshness you assume. Most teams cannot produce this list, and producing it is the most valuable artifact of the first month. Kestrel's took four days and turned up nine columns that three pipelines read and nobody had ever mentioned to the producer.

Weeks 3–4: turn the inventory into assertions. §17.5's point 3, consumer-side. They live in your CI, they cost the producer nothing, and they convert a class of silent wrong answers into loud failures immediately — before any negotiation and with nobody's permission.

Weeks 5–8: publish the inventory to the producers. Not as a demand: as information they do not have. "Here is what we read from you, and here is what breaks if it changes." This is the step that changes the relationship (§17.7), and its entire content is §17.4's consumer list — delivered by the consumer, because nobody else is in a position to write it.

Weeks 9–12: get one contract agreed, for one dataset. Pick the one where a break has already hurt, because that argument is already won. Do not attempt a programme. One contract, one enforcement point, one owner, one deprecation window.

Three things not to do first, all of them common:

Do not start with a registry. A registry with no agreed contracts enforces a schema nobody has discussed, and it will be routed around the first time it inconveniences a producer at 2 a.m.

Do not start with a template. A seven-field template circulated to eight teams produces eight half-filled documents and a durable shared sense that this was bureaucracy.

Do not start with the most important dataset. Start with the one that has already broken. The important one is where you will be asked the hardest questions and where you have the least evidence.

💸 Cost Check — the incident is the budget

Contract work is funded by incidents and by nothing else, so price the last one before asking for the time.

Kestrel's postal_code widening (Chapter 13 §13.7): 12% of customers nulled, discovered nine days later by a marketing team whose regional segmentation had quietly stopped working.

text engineering: root cause, fix, backfill 3 people x 4 days = 12 person-days nine days of regional campaign spend, mistargeted ~$41,000 re-running the affected sends ~$6,200 analyst hours spent doubting the segmentation real, unmeasured

The consumer-side assertion that would have caught it on day one is four lines, and it existed in the repository within a week of the postmortem — which is the usual pattern and the frustrating one.

Lead with days-to-detection, not with dollars. "Nine days" is the number that gets the work approved. A dollar figure invites an argument about the dollar figure, which you will lose, because the person arguing knows the marketing budget better than you do.

🎓 Interview Angle — "how would you introduce data contracts here?"

The weak answer describes the end state: a registry, a template, CI enforcement, ownership. It is correct, it is what the candidate read last week, and it tells the interviewer nothing.

The strong answer starts at the consumer, because that is the only place you have authority on day one:

"I'd start by writing down what we already depend on — the columns we read, the values we assume — and turn that into assertions in our own CI. That's free and needs nobody's agreement. Then I'd take that list to the producers, because it's information they don't have: they can't know who reads them. The contract comes after that conversation, not before it, and I'd want the first one on a dataset that's already broken, so I'm not arguing hypotheticals."

The follow-up is always a version of "what if the producer won't agree?" The answer is §17.8: you write an observed contract, you enforce it on your side, and you label it observed rather than agreed — because a document that looks agreed and is not is worse than having none.

🔐 Privacy & Governance — a contract is where classification becomes enforceable

A schema says a column is a string. A contract can say it is a person's email, and that difference is what makes automated privacy tooling possible at all.

yaml fields: email: type: string classification: personal # <- the field this callout is about lawful_basis: contract retention_days: 730 may_leave_region: false anonymous_id: type: string classification: personal # NOT obvious, and it is the point note: > A first-party cookie id. It is not a person and it identifies a device, which is personal data under every reading we have. value_cents: type: integer classification: none

Four things become mechanical once that field exists, and none of them is possible from a schema alone:

The deletion manifest generates itself (Chapter 31). Every dataset whose contract declares a personal field joins the manifest automatically, including the ones nobody remembered.

Retention becomes checkable. A scheduled job compares the declared retention_days against what the lifecycle rules actually enforce, and disagreements are findings rather than surprises.

Residency becomes enforceable at the boundary. may_leave_region: false is a rule a pipeline can assert on, at the moment of a cross-region write, rather than a policy discovered in an audit.

And classification becomes reviewable. A pull request that adds a field with classification: none to a dataset carrying identifiers is a diff somebody can question — which is the only point in the lifecycle at which questioning it is cheap.

The observation that makes this worth the effort: classification decays. A tag applied once, in a catalog, is correct on the day it is applied and drifts as the data changes. A classification in a contract is reviewed every time the contract is — and the contract is reviewed because the schema changes, which is the same event that would have invalidated the tag.

For an observed contract (§17.8) the field is even more valuable, because you are classifying data from a source that will never classify it for you. The carrier's tracking response contains an address; writing classification: personal next to it is the only record that will ever exist.

📏 Scale Note — how many contracts you should have

Not one per table. The most common way a contract programme dies is that somebody decides every dataset needs one, produces two hundred half-filled templates, and the whole idea is discredited within a quarter.

Contracts are for trust boundaries, and the number of trust boundaries is much smaller than the number of tables.

text team size datasets TRUST BOUNDARIES contracts worth having ──────────────────────────────────────────────────────────────── 4 (Kestrel) ~60 3 3-5 20 ~300 ~8 8-12 100 ~2,000 ~40 40-60

A trust boundary is any place where the producer and the consumer cannot coordinate by talking. Inside one team, a schema change is a conversation and a shared repository; across a boundary it is a surprise, and that is where a contract earns its cost.

Kestrel's three: orders and customers from the commerce team, clickstream from the web team, and two observed contracts for sources with no counterparty at all.

Two ratios worth carrying:

Roughly one contract per five to eight datasets crossing a boundary, because several datasets usually share a producer and a change process.

And roughly 8–12 assertions per contract, which is the number that actually fails builds. The count that matters is never the contract count; it is the assertion count (§17.12's 🧱), because a contract with no assertion behind it is a paragraph.

The failure mode at each size is different and worth naming. At four engineers, the risk is that nobody writes any, because everything feels like a conversation — and the two observed contracts are the ones that matter, since the external sources are the boundaries you cannot talk across. At a hundred, the risk is the template flood, and the discipline is to write a contract only where a specific break has happened or is clearly imminent.

🔎 Read the Plan — what the registry will tell you before you ship

Compatibility is checkable from the command line, before a deploy, in about two seconds — and almost nobody does it locally, which is why the 409 arrives in CI instead.

```bash

1. what is the subject's CURRENT compatibility mode?

curl -s $REG/config/kestrel.orders.v1-value | jq

{"compatibilityLevel":"BACKWARD"} <- note: NOT transitive (§17.3)

2. what does the GLOBAL default say? They can differ, and the

subject-level one wins.

curl -s $REG/config | jq

3. WILL this schema be accepted? Ask before registering.

curl -s -X POST $REG/compatibility/subjects/kestrel.orders.v1-value/versions/latest \ -H "Content-Type: application/vnd.schemaregistry.v1+json" \ -d @new-schema.json | jq

{"is_compatible": false, "messages": ["READER_FIELD_MISSING_DEFAULT_VALUE: ..."]}

4. what versions exist, and how far back can a consumer replay?

curl -s $REG/subjects/kestrel.orders.v1-value/versions | jq ```

Command 3 is the whole callout. It is a dry run of the registration, it costs nothing, and it turns "my deploy failed" into "my schema change is incompatible, here is the field."

Command 1 is the one that surprises people. The subject's mode may not be the global default — somebody set it once, for a reason, and a subject on BACKWARD when you believed the project was on BACKWARD_TRANSITIVE is the gap Exercise 17's version note is about. Check per subject.

And command 4 answers a question the compatibility check cannot. With non-transitive compatibility, your new schema is checked against version N only; the version list tells you how many earlier versions exist that nothing has verified you can read. If bronze retains two years and there are nine versions, you have nine schemas' worth of history and one check.

The habit: run command 3 in a pre-commit hook. Two seconds, no network dependency at deploy time, and it moves a class of failure from CI to the editor.

🔁 Idempotency Check — a contract should say whether a replay is safe

Every consumer of a stream will eventually replay it, and the contract is the only place the producer can say what that means.

yaml guarantees: replay: safe: true from: "the topic's retention window, 7 days" semantics: > Events are immutable once published. Replaying an offset range reproduces the same events. We do NOT rewrite history. caveat: > A corrected event is published as a NEW event with a later timestamp and the same business key. A consumer replaying MUST apply last-write-wins on (order_id, event_ts, offset), not first-write-wins.

The caveat is the field that earns the section. "We publish corrections as new events" is a statement about semantics that no schema can express and that changes every consumer's code — and a consumer that does not know it will produce a different answer on replay than it did live.

Three questions a contract should answer about replay, and most contracts answer none:

How far back can I replay? The retention, stated as a duration. A consumer designed against "forever" and deployed against seven days bootstraps with a silent gap.

Is the stream immutable, or is history rewritten? A compacted topic (§15.9) does rewrite history — earlier versions of a key disappear — so a replay from offset 0 is not the same as the original sequence. That is a very different guarantee and it is invisible from the schema.

And is the ordering the same on replay as it was live? Per partition, yes. Across partitions, no — and a consumer that merged two partitions by arrival order live will merge them differently on replay.

The general point: idempotency is usually discussed as a property of the consumer's write. For a stream it is also a property of the producer's publication, and the contract is the only artifact that can record it.

17.12 How Contract Programmes Fail

Four failure modes. None of them are technical.

The contract nobody enforces. Written, reviewed, agreed, filed — and with no enforcement point (§17.5) it drifts from reality inside a quarter. It is then actively harmful, for §17.10's reason: a document people believe is worse than no document, because belief removes the vigilance that absence would have preserved.

The registry everyone routes around. A producer blocked at 2 a.m. by a 409 will find the path that does not check — a second topic, a direct write, a "temporary" bypass flag that outlives everyone involved. The bypass is a signal, not a crime. If a compatibility rule is circumvented weekly, the rule is wrong or the legitimate path is too slow, and the fix is to the path.

The contract written by the consumer and never read by the producer. Very common, and it produces a specific pathology: the consumer believes there is an agreement, the producer has never heard of it, and the first breaking change is received as a betrayal by one party and as a normal Tuesday by the other. An observed contract is fine. An observed contract labelled agreed is a lie with a paper trail.

The deprecation window honoured on paper. §17.9's second residual, exactly: a MINOR bump, thirty days' notice, posted to a channel, and the consumer's CASE statement still counted the new status as cancelled. Notice was given and notice was not received. The countermeasure is not a better channel. It is the consumer-side assertion, which requires nobody to read anything.

⚠️ Failure Mode — the contract that documents the bug

A contract is usually written by reading the current behaviour and writing it down. If the current behaviour is wrong, the contract has now blessed it, and the next person to fix the bug is in breach.

Kestrel's orders contract froze refund_cents as a positive number, because that is what the producer emitted. It had been positive since a sign inversion three years earlier (Chapter 2 §2.2) that nobody had corrected, and by the time it was written down, four consumers had compensating ABS() calls and one did not.

Writing the contract made the bug permanent for another eleven months, because fixing it now required a MAJOR version and the coordination of five consumers rather than one commit in one repository.

The discipline: mark every field you write down as intended or observed. They are different claims, they are indistinguishable six months later, and only one of them is safe to build on.

🧱 Kestrel Platform — contracts/ after eighteen months

text platform/contracts/ ├── orders.yml agreed producer: #commerce enforced: producer CI ├── customers.yml agreed producer: #commerce enforced: producer CI ├── clickstream.yml agreed producer: #web enforced: registry ├── supplier_b_feed.yml OBSERVED producer: external enforced: our CI only ├── payments_export.yml OBSERVED producer: #finance-ops enforced: our CI only └── check_contracts.py runs in both CIs; 41 assertions

Five contracts in eighteen months, two of them observed. That rate looks slow and it is the honest one: each agreed contract took a negotiation, and each observed contract took an afternoon and delivered most of the protection.

The number that matters is not five. It is 41 — the assertions, which are the thing that actually fails a build. A contract with no assertion behind it is a paragraph.

17.13 Summary

Part III has been one problem in five costumes, and every instance was a producer making a reasonable change that a consumer depended on, with neither side knowing the dependency existed.

The argument is about where a failure lands, not whether one happens. Producers must be able to change. The question is whether the incompatibility surfaces at the producer, at write time, as a failed deploy owned by the person making the change — or at the consumer, weeks later, as a wrong number owned by someone who did nothing.

Three parties and one missing artifact. The producer cannot know who reads them; the consumer cannot know when the shape will change; the data carries no statement of intent. A contract is not a schema, not a restriction on the producer, and not primarily a technology.

Compatibility has three modes: backward (a new reader can read old data — the default, and what replay requires), forward (an old reader can read new data — right when producers must move first), and full (both — restrictive, and right for long-lived multi-consumer data). Use the transitive variant if consumers may read arbitrarily old data, which for a replayable log they can. And two rows surprise people: adding a required field is breaking in practice whatever the table says, and adding an enum value is compatible on paper and frequently breaking in practice, which is the most common "it should have worked" registry incident.

A registry's entire value is step 1's rejection: an incompatible schema gets a 409 at deploy time, in CI, and the bad data never exists. It carries an id rather than a schema in each message, it is a runtime dependency of every consumer — a registry outage during a cold-cache restart is a real failure — and it enforces exactly one field of the contract.

A contract contains seven things beyond the schema, and two of them do most of the work. The consumer list solves the producer's central problem — they cannot otherwise know who depends on them — and being on it is what entitles you to notice. The semantics field is what catches the worst incidents, because a sign inversion or a redefined timestamp is a change with no schema change at all, and no registry can see it.

Four enforcement points: producer CI (cheapest and most valuable — the change never merges), producer runtime, consumer runtime (the only option for sources you do not own), and scheduled checks against the guarantees — the only point that catches a compliant producer that has stopped or halved its volume. Build consumer validation first, because you can build it unilaterally, this afternoon, without anyone's agreement, and because it generates the evidence that makes the producer conversation possible.

Version semantically, and a MAJOR is a new topic, not a modification. Expand-contract is the answer to "how do we ever rename a field": add alongside, migrate at each consumer's pace, remove when the consumer list says nobody reads it. Step 2 takes the time and step 3 is a guess without the list.

Contracts fail socially more often than technically. Four failure modes: written once and never updated (fix: it lives in the producer's repo and CI fails without it), nobody knows it exists (fix: discoverable from the data), the producer sees it as an imposition (fix: frame it as what they already provide — the consumer list is the producer's benefit), and violations have no consequence (fix: a violation is an incident with a retrospective).

Write contracts for sources you do not control, marked status: observed. It records what you measured, defines "broken" so it is a check rather than a feeling, and generates evidence. One team's eight-month log of nineteen violations — six of which had cost the producer's time — achieved in one conversation what two years of asking had not.

Kestrel's contract cut breaking changes from 11 to 2 and detection time from nine days to zero. The two residuals show the limits: a semantic change with no schema change, which nothing automated caught; and an added enum value that was policy-compliant and silently miscounted orders until a nine-line consumer-side assertion against the declared enumeration converted it into a failed build.

Four cases where a contract is not worth it — one team, exploratory data, no enforcement point available, or a tolerant consumer. And three that are not on that list: a different team, an external producer, or a producer who will not agree. All three call for an observed contract rather than for skipping it.

What's next

Part III ends here. Part IV is transformation — where raw data becomes something a person can trust, where most of your code lives, and where the silent failures happen. Chapter 18 opens it with advanced SQL for data engineering: window functions, CTEs, recursion, and pivots, and the set-based thinking that separates a query that runs in four seconds from one that runs in forty minutes.