Chapter 36: Event-Driven Architecture

"The database said the order was shipped. The event log said it had never been placed. Both had been written by the same function, four lines apart."

Overview

Chapters 14, 15, and 29 have been approaching the same idea from three directions. Change data capture turns a database's writes into a stream. Kafka gives that stream a durable, replayable home. Streaming processing consumes it. Each chapter treated the log as a transport for data whose real home was a table.

This chapter inverts that. What if the log is the home, and every table is a view of it?

That inversion is worth taking seriously and is not always right, and the chapter is organized around four things you can measure rather than around an argument:

The dual write loses data silently (§36.5). Writing to a database and then publishing an event is not atomic, and code/event_lab.py measures the loss: 24 of 10,000 writes, with the database correct, the service returning 200, and nothing reconciling the two.

State is a fold over events (§36.6), and the payoff is not elegance. It is that you can build a projection that did not exist when the events were written — two of the lab's four projections were computed retroactively, and against a state table they are not slow, they are unanswerable.

Ordering matters for some projections and not others (§36.9). Shuffle the event stream and every sum stays correct while 49% of order statuses go wrong. Most teams buy ordering machinery for the whole stream when they need it for one projection.

And the log costs almost nothing (§36.11): 10× the storage of a state table, and 27 cents a month more.

One framing to carry throughout. "Event-driven" covers three different things — event notification, event-carried state transfer, and event sourcing — which have different costs and are constantly conflated. §36.2 separates them, and most teams need the first and adopt the vocabulary of the third.

36.1 What an Event Is

An event is an immutable statement that something happened, at a time, expressed in the past tense.

{"type": "OrderShipped", "order_id": 889201, "version": 7,
 "occurred_at": "2026-03-14T09:12:44Z", "carrier": "ups"}

Four properties, and the third is the one that gets violated:

Past tense. OrderShipped, not ShipOrder. A command can be rejected; an event has already happened. If your event names are imperative, you have a command queue, which is a different thing with different guarantees.

Immutable. You never update an event. A mistake is corrected by appending a compensating event — OrderRefunded, not an edit to OrderPlaced — and the history of the mistake is part of the record.

Self-contained enough to be useful. How much context to include is §36.8's whole subject and the hardest design question in this chapter.

Identified and ordered within its aggregate. An order_id and a monotonic version. The version is what makes ordering checkable rather than assumed, and the lab asserts contiguity from 1 for every aggregate.

And what an event is not: a row. A row is a statement about the present, mutable, and it answers what is true now. An event answers what happened, and the difference is the entire chapter.

36.2 Three Things Called "Event-Driven"

Conflating these is the most common source of over-engineering in this territory.

Event notification. "Order 889201 changed." The consumer fetches what it needs. Cheap, loosely coupled, and it creates a synchronous dependency back on the producer — which is the trade.

Event-carried state transfer. "Order 889201 now looks like this: {...}." The consumer can act without calling back. This is what CDC produces (Chapter 14) and what most "event-driven" data platforms actually run on.

Event sourcing. The event log is the system of record. There is no order table to fall back on; the order is its events, and every table is derived. This is a much bigger commitment and it is the one the vocabulary is borrowed from.

⚠️ Failure Mode — adopting event sourcing's vocabulary with event notification's design

The pattern, seen repeatedly: a team adopts events, uses the words event store, projection, and replay, and builds event notification — small events with ids, a database that remains the source of truth, and consumers that call back.

That is a fine architecture. The problem is that the vocabulary creates expectations the design cannot meet:

• "We can replay." You can replay the notifications. They do not contain the data, and the database they would call back to has since changed, so replaying reconstructs today's state with yesterday's triggers.
• "We have full history." You have a history of when things changed, not what they changed to. §36.6's retroactive projections are unavailable.
• "State is a fold over events." It is not; state is in the database, and the events are a notification about it.

The failure surfaces eighteen months later when somebody asks for a retroactive projection and discovers the log cannot answer it — which is exactly when the argument for having built it that way is least available.

The fix is vocabulary discipline, and it is free. Call it what it is. A team that says "we use event notification" has an accurate shared model; one that says "we're event-sourced" has a shared model that will be wrong in a way nobody discovers until it matters.

36.3 Kestrel's Order Events

Five event types, one aggregate, and this is the model the lab uses:

OrderPlaced      order_id, customer_id, occurred_at
ItemAdded        order_id, sku, cents
ItemRemoved      order_id, sku, cents
OrderShipped     order_id, carrier
OrderRefunded    order_id, cents, reason

8,480 events across 2,000 orders — 4.24 per order. That ratio matters for everything downstream: it determines the log's size (§36.11), the fold's cost, and how much a replay costs.

Note what is not in the list. There is no OrderUpdated and no OrderStatusChanged. A generic event carrying a diff is a row update wearing an event's clothes, and it destroys the property that makes events useful: you cannot tell, from OrderUpdated, what happened or why.

36.4 State Is a Fold

The core idea, in one function:

def project_order_state(events):
    state = {}
    for e in sorted(events, key=lambda x: (x["order_id"], x["version"])):
        s = state.setdefault(e["order_id"], {"status": None, "cents": 0})
        t = e["type"]
        if t == "OrderPlaced":    s["status"] = "placed"
        elif t == "ItemAdded":    s["cents"] += e["cents"]
        elif t == "ItemRemoved":  s["cents"] -= e["cents"]
        elif t == "OrderShipped": s["status"] = "shipped"
        elif t == "OrderRefunded": s["status"] = "refunded"
    return state

state = fold(reducer, events). The order table is not a thing you maintain; it is a thing you compute.

Three consequences:

A projection is a pure function of the events. Same events, same output, always. The lab asserts this, and it is what makes a rebuild trustworthy.

You can have many projections of the same events. An order-state table, a daily revenue rollup, a per-customer aggregate — all correct, all derived, none authoritative over another. This is CQRS's read-side, and it arrives here as a consequence rather than as a pattern to adopt.

A projection can be wrong and fixed without data loss. Change the reducer, replay, done. Compare a state table, where a bug that has been running for three months has overwritten the evidence of what the right answer was.

36.5 The Dual Write

The most important practical problem in this chapter, and it is four lines of ordinary code:

def place_order(order):
    db.execute("INSERT INTO orders ...")   # commits
    db.commit()
    kafka.publish("OrderPlaced", order)    # <- and here the process dies

There is no transaction spanning both. The database commits; the process dies before publishing; the order exists and the event does not, forever.

--outbox measures it at a 0.2% crash rate — the probability of the process dying in that window across deploys, OOM kills, and node preemption:

DUAL WRITE (commit, then publish)
    rows in the database        10,000
    events in the log            9,976
    LOST, silently                  24   (0.24%)

TRANSACTIONAL OUTBOX (one transaction, then relay)
    rows in the database        10,000
    events published            10,000
    LOST                             0
    duplicates                      14   <- consumers must be idempotent

⚠️ Failure Mode — the dual write is silent, and reversing the order does not help

Nothing about this failure is visible. The database is correct. The service returned 200. The customer's order exists and will ship. Twenty-four orders simply never reached the log, and the only thing that would notice is a reconciliation nobody wrote.

The instinctive fix — publish first, then commit — is worse, and it is worth being explicit about why:

text commit then publish: crash -> order exists, no event (missing event) publish then commit: crash -> event exists, no order (PHANTOM event)

A phantom event is harder to deal with than a missing one. Downstream consumers act on an order that does not exist: the warehouse picks it, the analytics count it, the customer is emailed about it. A missing event under-reports; a phantom event causes work.

The second instinctive fix — retry the publish — helps and does not solve it. A retry loop inside the process is defeated by the process dying, which is the case that matters. You cannot retry your way out of a crash between two non-atomic writes.

The third — a background job comparing the table to the log — is a real mitigation and is essentially the outbox, discovered the hard way, without the outbox's guarantee that the comparison can be made cheaply.

The transactional outbox is the standard fix and it is simple:

BEGIN;
  INSERT INTO orders (...) VALUES (...);
  INSERT INTO outbox (aggregate_id, type, payload, created_at)
       VALUES (..., 'OrderPlaced', ..., now());
COMMIT;                       -- ONE transaction. Both or neither.

A separate relay reads unsent outbox rows, publishes them, and marks them sent.

📐 Design Decision — the outbox trades loss for duplication, and that is the right trade

The outbox does not give you exactly-once. A crash after publishing and before marking the row sent produces a duplicate on the next relay pass — 14 of them in the lab's run.

So the choice is not between broken and perfect. It is between two failure modes:

text dual write: events LOST -- silent, permanent, unfixable outbox: events DUPLICATED -- loud if you look, and fixable by the consumer

Duplication is strictly better and the reason is Chapter 20. An idempotent consumer — one keyed on (aggregate_id, version) — makes a duplicate a no-op. There is no consumer design that makes a missing event a no-op, because the consumer never learns of it.

This is the same asymmetry as at-least-once versus at-most-once delivery (Chapter 15 §15.7), and it resolves the same way: build for at-least-once and make consumers idempotent. The version number in §36.1 is what makes that cheap — a consumer that has seen version 7 discards a second version 7 in one comparison.

And the honest cost: every consumer must now be idempotent, forever, including the ones written by people who have not read this chapter. Kestrel enforces it with a shared consumer base class that does the dedup, which is the only version of "every consumer must" that survives contact with a growing team.

Two alternatives worth knowing. CDC on the outbox table (Chapter 14) removes the relay entirely — Debezium reads the outbox from the write-ahead log. And CDC on the business tables directly skips the outbox, at the cost of coupling your event schema to your database schema, which §36.8 argues against.

36.6 The Retroactive Projection

This is the payoff, and it is the argument that justifies the cost.

The lab computes four projections from the same 8,480 events:

PROJECTION A: order state          (what a state table holds)
    placed 129 | shipped 1,804 | refunded 67

PROJECTION B: daily revenue        (designed for)
    90 days, 52,777,245 cents

PROJECTION C: time to ship         <- NOBODY DESIGNED FOR THIS
    1,869 orders; p50 2 days, p90 5, max 18

PROJECTION D: orders with a removed item   <- ALSO RETROACTIVE
    5.90%

C and D were not anticipated. No column was added, no schema changed, no instrumentation deployed. They were computed from history that already existed.

🏭 From the Pipeline — "unanswerable" is stronger than "slow"

Against a state table, projections C and D are not expensive. They are impossible.

Time to ship requires placed_at and shipped_at. A state table has status and updated_at, and updated_at was overwritten when the order shipped. The placement time is gone, unless someone thought to keep a column for it — which is the whole point: somebody had to anticipate the question.

The removed-item rate is worse. A state table records the order's current lines. An item that was added and then removed leaves no trace at all — the row was deleted, and nothing anywhere records that it existed.

Kestrel's actual experience with C: the operations team asked, in a quarterly review, "what is our p90 time to ship?" The p50 was known — it is on a dashboard. The p90 was not, and the tail is what the question was about: p50 is 2 days and the maximum is 18, and it is the eighteen-day orders that generate support tickets.

With events, the answer took twenty minutes. Without them it would have required adding a column, waiting a quarter, and answering a question about last quarter with next quarter's data.

The generalizable claim, and it is the one worth arguing for internally: an event log is an option on questions you have not thought of yet. Most of them you will never exercise. The ones you do exercise, you exercise at a moment when the alternative is waiting a quarter.

36.7 Reading the Log to Debug

The retroactive projection is the argument that wins a design review. This is the benefit you get every week, and it is the one practitioners cite when asked what they would miss.

A support ticket: "order 889201 says refunded and the customer says they never asked for a refund."

Against a state table, the investigation is: the row says refunded, updated_at says 09:14 on the 14th, and that is everything the database knows. Who did it, why, and what the order looked like before are all gone. The rest of the investigation is application logs, if they were retained, and guesswork.

Against the log, it is a SELECT:

SELECT version, type, occurred_at, payload
  FROM events
 WHERE aggregate_id = 889201
 ORDER BY version;

1  OrderPlaced     2026-03-11T14:02Z  customer_id=903418
2  ItemAdded       2026-03-11T14:02Z  KS-00291, 8400
3  ItemAdded       2026-03-11T14:03Z  KS-00147, 3200
4  ItemRemoved     2026-03-11T14:05Z  KS-00147, 3200
5  OrderShipped    2026-03-13T09:40Z  carrier=ups
6  OrderRefunded   2026-03-14T09:14Z  cents=8400, reason=carrier_damage

The reason field answers the ticket in one line, and events 3 and 4 answer a question nobody asked: the customer added an item and removed it two minutes later, which a state table never recorded happened at all.

🏭 From the Pipeline — the debugging benefit is the one that survives a change of team

Kestrel's team ranked the benefits of event sourcing after two years, and the order surprised them:

text 1. "why is this row in this state?" answered in one query 2. retroactive projections 3. rebuilding a projection after a bug, with no data loss 4. audit trail for finance and privacy

The retroactive projection is the better argument and the rarer event. Kestrel has built two in two years. The debugging query runs several times a week, by support engineers and analysts as well as by the data team.

Why the ranking matters: the debugging benefit accrues to people who did not choose the architecture. A support engineer who joined last month gets it without knowing what event sourcing is, which means the pattern keeps paying after the people who advocated for it have moved on — and that is a property very little architecture has.

The corollary is a warning. If your event log is not queryable by ordinary people — if it lives only in Kafka, behind a consumer API, with no table anyone can SELECT from — you have the architecture and not the benefit. Kestrel projects every event into a plain events table in the warehouse, partitioned by day, for no reason other than that people can query it, and that table is read more than three of the four projections.

36.8 Designing the Event

The hardest question in this chapter: how much does an event carry?

Thin events carry an id and a type. Small, stable, and every consumer must call back, which recreates the coupling events were supposed to remove and puts synchronous load on the producer.

Fat events carry the full state. Consumers are independent, and now your event schema is a public API that changes whenever the entity does.

Three rules that resolve most cases:

Carry what the event is about, not the entity's whole state. OrderShipped carries the carrier and the time, not the customer's address. If a consumer needs the address, that is a different question.

Never carry a computed value that a consumer could compute differently. Chapter 30 §30.8: put cents in the event and let consumers sum it. Putting order_total in OrderShipped creates a second definition of the total.

Version the schema and never break it. New fields are optional. Removed fields are deprecated for a window (Chapter 17). An old consumer must survive a new producer, which is the whole of §36.10.

📐 Design Decision — the event schema is not the database schema, and coupling them is the mistake

The cheapest way to emit events is CDC on your business tables (Chapter 14). It requires no application change and produces a complete change stream.

It also makes every consumer depend on your table layout, and that is the trap:

• A column rename becomes a breaking change for every consumer. The refactor you wanted to do in your own service is now a cross-team migration.
• The events carry your normalization decisions. A consumer receives orders and order_lines as two separate streams and must rejoin them — reconstructing an aggregate you had and threw away.
• There is no place for intent. A row change from status='placed' to status='cancelled' does not say whether the customer cancelled or the fraud system did. The intent was in the application and the CDC stream never saw it.

The third is the one that matters most and is the least visible, because you only discover the missing intent when someone asks a question that needs it — and by then the history that would have carried it is years long.

So: CDC is an excellent way to replicate a database and a poor way to publish a domain event. Kestrel uses CDC for exactly what Chapter 14 built it for — keeping bronze in sync — and the outbox for domain events, and the two are deliberately different streams with different schemas.

The honest cost of that decision: the application team must write the outbox insert, which is work they would not otherwise do, and it is the reason most organizations end up with CDC-as-events.

36.9 Ordering: Which Projections Actually Care?

Ordering machinery is expensive — partition keys, sequence numbers, buffering, watermarks (Chapter 29) — and most teams apply it to their whole stream.

--order shuffles the event stream and re-runs the projections:

daily revenue (a SUM)
    identical after shuffle?  YES   <- commutative

order line total (a SUM)
    orders with a wrong total   0   <- commutative

order status (LAST WRITE WINS)
    orders with wrong status  245 of 500  (49.0%)

The same shuffle that leaves every sum exactly correct corrupts nearly half the statuses.

🔎 Read the Plan — sort your projections by commutativity before buying ordering

The reducer's algebra decides, and it is checkable in about ten seconds per projection:

Reducer	Commutative?	Needs ordering
sum, count	✅ yes	no
min, max	✅ yes	no
set union, distinct count	✅ yes	no
last-write-wins (status)	❌ no	yes
first-write-wins	❌ no	yes
a state machine	❌ no	yes
anything using a running total	❌ no	yes

Two consequences that change what you build:

You need per-aggregate ordering, not global ordering. Kestrel's status projection needs order 7 to arrive after order 6 for the same order — it does not care about order 889202. Partitioning by order_id gives that for free in Kafka, and it is dramatically cheaper than any global ordering scheme.

And for non-commutative projections, the version number removes the need for ordering entirely. A consumer that stores (order_id, last_version_seen) and discards anything not greater is correct under arbitrary reordering — it does not need the events in order, it needs to know which is newer. That is a per-consumer change of about six lines against a platform-wide ordering guarantee.

The general move: check the algebra before buying the infrastructure. Kestrel found that two of its four projections were commutative and needed nothing, one needed per-partition ordering, and one was fixed with a version check — and the global-ordering design that had been proposed was needed by none of them.

36.10 Schema Evolution and Replay

An event log is forever, which means every schema you have ever emitted is still in it.

Three rules, and the third is the one that gets skipped:

Additive changes only. New optional fields. A consumer written last year must still parse an event written today.

Never reuse a field name for a different meaning. Chapter 30 §30.8's problem in a stream that cannot be rewritten.

Upcast on read, in one place. When a schema genuinely must change, write a function that converts old events to the new shape at read time, and put it in the deserializer rather than in every projection.

def upcast(e):
    if e["type"] == "OrderRefunded" and "reason" not in e:
        e["reason"] = "unknown"        # v1 had no reason field
    return e

🧭 Version Note — the replay you will actually do

Replay is the property everyone cites and few have exercised, which is Chapter 34 §34.15's warning in a new place: a capability you never use is a capability you do not have.

Kestrel's rebuild of all four projections from 8,480 events takes under a second in the lab, and at production scale — 30.5 million order events over three years — it is minutes, not hours, because a fold is a single pass with no joins.

What actually goes wrong in a replay, from three real attempts:

• An old event type nobody handles. ItemPriceCorrected was emitted for four months in 2024 and removed. The reducer had no branch for it and silently ignored it, which was correct in 2024 and wrong on replay, because the corrections were real.
• A projection that calls an external service. A reducer that enriched events with a live API lookup produced today's enrichment for 2024's events. A projection must be a pure function of the events — the lab asserts this, and it is the property that replay depends on.
• A schema version nobody remembered. The upcaster handled v1→v2 and the log contained v0.

All three are found by replaying, and only by replaying. Kestrel replays into a scratch schema quarterly and diffs against the live projection — the same exercise as Chapter 34's full rebuild, for the same reason, and it has found something twice.

36.11 What It Costs

Three years of Kestrel's order events against the equivalent state table:

event log       13.0 GB   $0.30 / month
state table      1.3 GB   $0.03 / month
ratio           10.0x     extra $0.27 / month

Twenty-seven cents a month.

💸 Cost Check — the storage argument against event sourcing is not an argument

"An event log is ten times the size of a state table" is true, is the objection usually raised, and is worth $0.27 a month at Kestrel's order volume.

The reason is arithmetic nobody does: 2.4 million orders a year, 4.24 events each, ~420 bytes per event, three years. That is 30.5 million events and 13 GB — and Chapter 33 §33.4 established that storage is 2.1% of the bill. Ten times a rounding error is a rounding error.

What the log actually costs is not storage:

• Every consumer must be idempotent (§36.5), forever, including future ones.
• The application team writes the outbox insert (§36.8), which is real work they do not otherwise do.
• Every projection is code that must be maintained and replayed, and a projection with a bug is a projection that has been wrong since it was written.
• Debugging requires reading a log rather than a row. "Why is this order in this state" becomes a fold rather than a SELECT, which is more informative and less immediate.

Those are the real costs and they are all engineering time. Argue the trade there, and if somebody raises storage, do the arithmetic in front of them — it takes a minute and it removes the objection permanently.

The honest counterweight: clickstream is different. 14 million events a day, and the same 10× multiplier is 4.19 TB a year rather than 13 GB. Event volume, not the pattern, decides whether storage is part of the conversation.

36.12 Retention: The Log That Cannot Be Forever

Event sourcing's premise is that the log is complete. Two things make that impossible, and the resolution is the same in both cases: snapshot, then truncate.

Privacy. Chapter 31: an erasure request must reach every copy of a person's data, including the events. An immutable log and a deletion obligation are in direct conflict, and the conflict is real rather than a technicality.

Volume. At Kestrel's order rate the log is 13 GB over three years and the retention question is theoretical. At clickstream volume it is 4.19 TB a year and it is not.

🔐 Privacy & Governance — how an immutable log survives an erasure request

Three approaches, and only two are honest.

Crypto-shredding (§31.5). Each subject's event payloads are encrypted with a per-subject key; deleting the key makes every event about them unreadable without modifying the log. The events remain, the sequence remains, the projections that already ran remain — and a replay produces an order whose customer fields are unrecoverable, which is exactly the intended state. This is the best available answer and it is a substantial commitment: per-subject key management on the read path, and a re-encryption story for rotation.

Snapshot and truncate. Fold the events older than a boundary into a snapshot, apply the deletion to the snapshot, and drop the events. You lose the history before the boundary — which is a real loss of the property this chapter argues for, and is often acceptable: Kestrel keeps two years of raw events and snapshots beyond that.

And the dishonest third: rewriting the log. Deleting or editing individual events. It breaks immutability, invalidates every version number after the deletion, and makes replay non-reproducible — and it is what teams do when they have not planned for this, under time pressure, with a statutory deadline.

The design decision that makes either honest option available: keep personal data OUT of the event payload where you can. OrderPlaced carries customer_id, not the email, the name, or the address. Then erasure is a matter of the customer dimension (Chapter 31's manifest) and the event log holds an identifier that no longer resolves — which is pseudonymized, still in scope, and dramatically easier to handle than a log full of addresses.

Kestrel got this right by accident, because §36.8's rule — carry what the event is about — happened to exclude personal data from every event type. The rule was adopted for coupling reasons and paid off for privacy reasons, which is worth noting because the reverse also happens.

Compaction is the other tool and it is not the same thing. A compacted topic retains the latest event per key and discards the rest — which is correct for a state-transfer stream and destroys an event-sourced log, because the earlier events are the data. Compaction and event sourcing are incompatible on the same topic, and confusing them is a real failure: Kestrel's order topic is retention-based, and the CDC topics that mirror tables are compacted.

36.13 When Not To

Four conditions, and the first two disqualify most systems.

When nobody needs history. If the current state answers every question you have and can imagine, the log's option value is zero and its costs are not. A configuration table does not need events.

When you have no aggregate boundary. Event sourcing needs a thing that events are about, with a version. A cross-cutting entity that everything touches has no clean aggregate, and the version becomes a contention point.

When your team will not maintain projections. A projection is code. Four projections is four things that can be wrong, and a team that struggles to maintain three dbt models will not maintain twelve reducers.

When you need to query state ad hoc. "Find all orders where the customer's third order was over $200" is a SELECT against a state table and a full fold against a log. You will build the state table anyway — which is fine, and is what CQRS says — but it means you are running both.

And the partial adoption that is usually right: events for the aggregates where history is the question — orders, payments, subscriptions, anything with a lifecycle — and plain tables for everything else. Kestrel event-sources orders and does not event-source the product catalog, and the catalog is the larger table.

🎓 Interview Angle — "how do you get data out of a service you don't own?"

A question about the dual-write problem, usually without saying so.

The weak answer is "they publish an event." That is the desired end state and it skips the question of how the event and the database row stay consistent.

The strong answer names the problem before the solution:

"The naive version is that the service writes its row and then publishes an event, and there's an interval between those two commits where the process can die — so you get a row with no event, or an event with no row, and no client-side code closes that gap. The outbox pattern fixes it: the service writes the domain row and an outbox row in the same transaction, and a separate relay reads the outbox and publishes. That gives you at-least-once publication of every committed change, in per-key order. What it doesn't give you is global ordering, and consumers have to be idempotent because the relay can publish twice."

Four things that answer does. It states the failure and why it has no local fix. It gives the mechanism in one sentence. It names what the guarantee is. And it names what the guarantee is not, which is the part that separates people who have run one.

The follow-ups:

"What if you can't change the service?" — CDC (Chapter 14), reading the outbox the database already keeps: its write-ahead log. The good answer notes that this needs nothing from the service team, which is Chapter 14 §14.2's first point and is the strongest argument for CDC.

"How does the relay know what it has published?" — a marker column, or a delete after acknowledgement. And the good answer prefers the delete, because a TTL as a safety valve turns a falling-behind relay into silent data loss.

"What's in the event?" — and this is where §36.2's three kinds matter: a notification, a state transfer, or an event-sourced fact. Most systems want the middle one and describe it as the third.

And a detail worth volunteering: the event's schema version goes in the payload, not in a header or a topic name, because both of those are lost on replay from storage.

🧱 Kestrel Platform — what is event-driven, and what deliberately is not

text flow event-driven? why ───────────────────────────────────────────────────────────────────────── clickstream -> bronze YES it is already events; there is no table to extract orders -> silver NO -- CDC the source is a database and the WAL is a better outbox than one we would ask for order status -> the support tool YES, outbox an operational consumer needs the transition, not the state inventory movements -> the on-hand projection YES a projection rebuildable from the log; ch 29's 🏭 is why it is a MICRO-BATCH nightly gold build NO it is a schedule, not an event, and pretending otherwise buys nothing the daily revenue figure NO finance wants a number as of a boundary, not a stream

Two rows carry the chapter's judgment.

Row 2 is the one people argue about. Asking the commerce team to publish order events is the "correct" event-driven answer and CDC needs nothing from them (Chapter 14 §14.2) — so Kestrel reads the log the database already writes, and the events it produces are as good as the ones a hand-built outbox would emit. The outbox pattern is for a service you control; CDC is for one you do not.

And the last row is the boundary worth defending. A revenue figure is a statement about a closed period, produced once, restated deliberately (Chapter 38 §38.5). Making it a continuously-updating projection would mean the number changes while somebody is reading it, which is §29.5's third question and is finance's decision rather than an architectural preference.

The count: three event-driven flows out of six, and the three that are not are not compromises. Event-driven is a good answer to "how does something know that something happened" and a poor answer to "what was the total."

📏 Scale Note — event volume decides whether this is cheap or expensive

The same pattern costs three orders of magnitude differently depending on what it is applied to, and the arithmetic is worth doing before the design discussion.

text domain events/day at ~400 bytes annual, retained ───────────────────────────────────────────────────────────────────── order state changes 33,000 13 MB/day 4.8 GB inventory movements 120,000 48 MB/day 17.5 GB clickstream 14,000,000 5.6 GB/day 2.0 TB

The first two rows make event sourcing nearly free. 4.8 GB a year is $0.11 a month, a full replay takes seconds, and the "keep every event forever" property that sounds extravagant is genuinely costless.

The third row is a different decision entirely. Two terabytes a year of retained events, replayed through a stateful projection, is a real system with real operational weight — and it is the same pattern, applied to something 400× larger.

Three consequences:

Event-source the small, high-value aggregates and not the large, low-value streams. Order state is worth every event; a page view is worth a retention window.

Replay cost scales with retention, not with the aggregate's importance. A projection over 4.8 GB rebuilds in seconds; over 2 TB it is a scheduled operation with a cost, which means it is no longer a debugging tool.

And the retention decision is a privacy decision at the third row's volume (Chapter 15's 🔐). Bounded retention on a clickstream is straightforward; unbounded retention on an event-sourced order aggregate is a design commitment to keeping personal data forever, and crypto-shredding is the only mechanism that resolves it.

The rule: apply event sourcing where the log is smaller than the state it produces. Order events compress to a state; clickstream events are the state, and there is nothing to gain.

🧪 Try It — kill the process between the two commits

bash cd part-07-architecture-patterns/chapter-36-event-driven-architecture/code python event_lab.py --self-check # 39 assertions python event_lab.py --demo dual-write --kill-after-commit python event_lab.py --demo outbox --kill-after-commit

The first demo writes a row, commits, and dies before publishing. Look at the two systems:

text the database the order exists, status='paid' the broker no event the consumer has no idea the order exists ...and there is NO CLIENT-SIDE CODE that closes this

Then try to fix it with a try/except — students always want to — and observe that it changes nothing, because the failure is between two commits rather than inside one. That five minutes is worth more than the explanation.

The second demo writes both rows in one transaction and dies before the relay runs.

text the database the order exists, AND an outbox row exists the broker no event -- yet restart the relay the broker the event, published

Then three things worth doing while the lab is open:

Run the relay twice over the same outbox rows. It publishes twice. At-least-once is the guarantee, and the consumer's idempotency is what makes it usable — which is Chapter 15's 🔁, and the lab's consumer demonstrates it.

Publish two events for different keys and check the order they arrive in. Per key, ordered; across keys, not. Say out loud what that means for a consumer that merges them.

And delete the outbox row before the acknowledgement rather than after. Kill the relay in the window. The event is lost, permanently, and nothing reports it — which is why §36.5's ordering is not a stylistic preference.

🔁 Idempotency Check — a projection must be rebuildable and a consumer must be repeatable

Two different properties, both required, and they are frequently conflated.

text property means tested by ───────────────────────────────────────────────────────────────────────── the CONSUMER is processing the same event replay a range twice, idempotent twice has one effect diff both directions the PROJECTION is rebuilding from offset 0 rebuild into a scratch rebuildable gives the same result target, EXCEPT both ways

A consumer can be idempotent and the projection still not rebuildable — if the projection depends on the order events arrive in and the replay order differs across partitions (§36.9's commutativity question), or if it reads anything outside the log.

Three things that break rebuildability and are easy to write:

A projection that reads the current time. WHERE occurred_at > now() - 30 days gives a different answer on every rebuild. It is Chapter 18's 🔁, in a stream.

A projection that enriches from a mutable table. Joining an event to today's dim_product produces yesterday's events with today's product names — which may be what you want and is a decision, not a default (Chapter 32's point-in-time question).

And a projection that is not commutative, over a log whose cross-partition order is not reproducible. "Last status wins" is order-dependent; it needs an explicit ordering key in the event rather than reliance on arrival, and that key is the same total-order problem as §18.7's tiebreak.

The test is one command and it belongs in CI:

bash rebuild_projection --from 0 --into scratch_verify diff_both_directions production scratch_verify # expect empty

A projection that has never been rebuilt is a projection you cannot fix. Chapter 34's argument, applied to a log rather than to a lake.

36.14 The Kestrel Platform

platform/events/
  schema/                 # one file per event type, versioned, additive only
  outbox_relay.py         # reads unsent, publishes, marks sent
  consumer.py             # base class: dedups on (aggregate_id, version)
  upcast.py               # old -> current, in ONE place
  projections/
    order_state.py        # last-write-wins  -> needs ordering
    daily_revenue.py      # sum              -> commutative
    time_to_ship.py       # built retroactively, 20 minutes
    removal_rate.py       # built retroactively
  replay.py               # rebuild into a scratch schema; quarterly

The row worth reading twice is the second-to-last. A global ordering design had been proposed and costed. Two of four projections turned out to be commutative, one needed only per-partition ordering, and one was fixed with a six-line version check — and the proposal was withdrawn.

36.15 Summary

Chapters 14, 15, and 29 treated the log as transport for data whose real home was a table. This chapter asks what happens when the log is the home — and answers it with four measurements rather than an argument.

⚠️ Three things are called "event-driven" — notification, carried state transfer, and sourcing — with different costs. Adopting sourcing's vocabulary with notification's design creates expectations the architecture cannot meet, and the failure surfaces eighteen months later when someone asks for a retroactive projection. Call it what it is.

An event is past-tense, immutable, self-contained enough, and versioned within its aggregate. A generic OrderUpdated carrying a diff is a row update wearing an event's clothes.

⚠️ The dual write loses data silently: 24 of 10,000, with the database correct and the service returning 200. Publishing first is worse — a phantom event causes work where a missing one merely under-reports — and you cannot retry your way out of a crash between two non-atomic writes.

📐 The transactional outbox trades loss for duplication, which is the right trade, because an idempotent consumer keyed on (aggregate_id, version) makes a duplicate a no-op and no consumer design makes a missing event a no-op. Enforce idempotency with a shared base class — the only version of "every consumer must" that survives a growing team.

🏭 State is a fold, and the payoff is retroactive projections. Two of the lab's four were computed from history that already existed, and against a state table they are unanswerable, not slow — updated_at was overwritten, and a removed line item left no trace at all. Kestrel's p90 time to ship took twenty minutes; the alternative was adding a column and waiting a quarter. An event log is an option on questions you have not thought of yet.

📐 The event schema is not the database schema. CDC-as-events couples every consumer to your table layout, makes a column rename a cross-team migration, and — the invisible one — has no place for intent: a row going to cancelled does not say who cancelled it.

🔎 Check the algebra before buying ordering. A shuffle leaves every sum exactly correct and corrupts 49% of statuses. You need per-aggregate ordering, not global — and for non-commutative projections a stored last_version_seen removes the need for ordering entirely, in six lines. Kestrel withdrew a global-ordering proposal that none of its four projections needed.

🧭 Replay is a capability you do not have until you exercise it. Three real failures: an event type nobody handles, a projection that calls an external service (a projection must be a pure function of the events), and a schema version nobody remembered. Replay quarterly into a scratch schema and diff.

💸 The storage objection is not an objection. 10× the state table and $0.27 a month. The real costs are engineering time — idempotent consumers forever, the outbox insert, projections that must be maintained, and debugging that becomes a fold. Argue the trade there. (And note that clickstream, at 14M events a day, is a different conversation — volume decides whether storage is in it.)

🏭 The benefit that survives a change of team is debugging, not retroactive projections. Kestrel built two retroactive projections in two years and runs the "why is this row in this state" query several times a week — by support engineers who do not know what event sourcing is. If your log is not queryable by ordinary people, you have the architecture and not the benefit: project every event into a plain table people can SELECT from.

🔐 An immutable log and an erasure obligation are in direct conflict. Crypto-shred, or snapshot and truncate — and never rewrite the log, which is what teams do under a statutory deadline when they have not planned for it. The decision that makes either option available is keeping personal data out of the payload: carry customer_id, not the address. And compaction is not retention — a compacted topic keeps the latest event per key, which destroys an event-sourced log, because the earlier events are the data.

Do not event-source when nobody needs history, when there is no aggregate boundary, when your team will not maintain projections, or when your access pattern is ad hoc query. The partial adoption is usually right: events for aggregates with a lifecycle, tables for everything else. Kestrel event-sources orders and not the product catalog — and the catalog is the larger table.

Chapter 37 is migrating legacy pipelines, which is the work most data engineers are actually hired to do, and where every pattern in Part VII meets a cron script nobody has credentials for.

Key terms: event · aggregate · projection · fold · event notification · event-carried state transfer · event sourcing · dual write · transactional outbox · idempotent consumer · commutativity · partition key · sequence number · upcasting · CQRS · replay