Case Study 2: The Ordering Guarantee Nobody Needed

"The proposal was eleven pages and cost two quarters. The counter-proposal was a spreadsheet with four rows and one column headed 'commutative?'"

Executive Summary

Kestrel proposed a global ordering guarantee for its event stream — a single partition, sequence numbers assigned by a coordinator, and a buffering consumer framework. Two quarters of engineering, plus a permanent throughput ceiling of one partition's worth of writes.

The proposal was withdrawn after a four-row analysis.

projection            reducer              commutative?   needs ordering
daily_revenue         sum                       YES             no
order_line_total      sum                       YES             no
customer_ltv          sum                       YES             no
order_state           last-write-wins           NO              yes

Three of four projections do not care about order at all. The fourth was fixed with a stored last_version_seen and a six-line comparison — not with ordering, but with the ability to recognize which of two events is newer.

The measurement that made it undeniable: shuffle the entire event stream and every sum is exactly correct, while 49.0% of order statuses are wrong.

Cost of the adopted solution: one afternoon, plus partitioning the topic by order_id, which Kestrel was doing anyway.

Skills applied: commutativity as a design tool (§36.9); per-aggregate versus global ordering; and the general move of checking the algebra before buying the infrastructure.

Background

The proposal was well-motivated and the motivating incident was real.

A consumer had produced a wrong order status. OrderShipped (version 5) had been processed before OrderPlaced (version 1), because the two events had landed in different Kafka partitions — the producer had been partitioning by a round-robin default rather than by order_id, which nobody had noticed because it had never mattered before.

The status projection saw OrderShipped first, set status = 'shipped', then saw OrderPlaced and set status = 'placed'. The order sat in placed forever, and the operations team noticed when it never appeared in the daily fulfillment report.

Forty-one orders were affected over three weeks.

The proposal that followed was thorough, and its thoroughness is why it was nearly approved:

1. one partition, so all events are totally ordered
2. a coordinator assigning a global sequence number
3. a consumer framework that buffers out-of-sequence events
   and releases them in order
4. a gap detector that alarms on a missing sequence number

Estimated at two quarters, with a stated and accepted cost: a single partition caps write throughput, which at Kestrel's 17,753 order lines a day is fine and at clickstream volume would not be.

The Problem

The counter-question came from an engineer who had read the reducers.

"Which of our projections would actually change if events arrived out of order?"

Nobody had asked. The incident had involved one projection, and the proposal generalized from one projection to the whole stream without anyone checking whether the generalization held.

🔎 Read the Plan — the four-row spreadsheet

The analysis took an afternoon and is entirely mechanical. For each projection, look at the reducer and ask whether f(f(s, a), b) == f(f(s, b), a):

text projection reducer commutative? ───────────────────────────────────────────────────────────── daily_revenue state += cents YES order_line_total state += cents YES customer_ltv state += cents YES order_state state.status = <this event's> NO

Three of four are addition, which is commutative, associative, and does not care. The fourth assigns, which is neither.

And the empirical confirmation, from event_lab.py --order: shuffle 500 orders' events completely and re-fold.

text daily revenue identical after shuffle? YES orders with a wrong line total 0 orders with a wrong STATUS 245 of 500 (49.0%)

The same shuffle that corrupts half the statuses leaves every sum exactly correct — not approximately, not within tolerance. Exactly.

This is why the argument was over in one meeting. The proposal's premise was "our event stream needs ordering." The measurement says one projection needs ordering and the other three are provably indifferent, and building a platform-wide guarantee for one projection's benefit is a category error — the same error as Chapter 33's optimizing for the wrong meter, arriving in reliability engineering.

The Analysis

Once the question was one projection rather than four, three cheaper options appeared.

Option A: partition by order_id. Kafka guarantees ordering within a partition, so all events for one order arrive in order. This alone fixes the original incident, costs a producer config change, and preserves parallelism across orders.

Option B: version checking in the consumer. Store last_version_seen per aggregate; discard anything not greater.

def handle(self, e):
    seen = self.versions.get(e["order_id"], 0)
    if e["version"] <= seen:
        return                      # older or duplicate -- ignore
    self.versions[e["order_id"]] = e["version"]
    self.apply(e)

Option C: the original proposal.

📐 Design Decision — A and B are not alternatives; they solve different problems

The team's first instinct was to choose between A and B. They are complementary, and untangling why took a whiteboard.

Partitioning (A) makes events arrive in order — under normal operation. It does not survive:

• A partition rebalance, where a consumer group reassignment can replay from a committed offset.
• A replay (§36.10), where you are deliberately re-reading history.
• A producer retry, which can reorder within a partition unless idempotent production is enabled.

Version checking (B) makes the consumer correct regardless of arrival order. It does not care what the transport did.

So A is an optimization and B is the correctness property. A reduces how often B has to discard anything; B is what makes the projection right.

Kestrel adopted both, and the framing that made the decision easy: A is a performance measure and B is a correctness measure, and you do not substitute one for the other. A team that adopts only A has a projection that is correct until the first rebalance, which is a failure mode that appears months later during an unrelated incident and is attributed to the incident.

And the six lines of B are the whole cost. No coordinator, no buffering, no gap detector, no throughput ceiling.

And the honest limitation of version checking, which the team wrote down:

It discards. An event that arrives late is dropped rather than applied. For a last-write-wins status that is exactly right — a later version has already superseded it. For a projection that needs every event, it is wrong, and such a projection needs buffering after all.

Kestrel has no such projection. Every non-commutative projection it runs is last-write-wins. Writing that down mattered, because the next projection might not be, and the note says what to check.

The Decision

Withdraw the proposal. Adopt A and B. One afternoon plus a config change.

producer:   partition by order_id                    config
consumer:   base class stores last_version_seen      ~6 lines
projections: the three commutative ones unchanged    0 lines
docs:       "if you write a non-commutative,
             non-last-write-wins projection, talk
             to us" -- with the reason              1 paragraph

And one thing the proposal had right, kept: the gap detector. Not for ordering — for loss.

🏭 From the Pipeline — the one part of the rejected proposal that was worth building

A gap detector alarms when version 7 arrives for an aggregate whose highest seen version is 5.

In the original proposal it existed to trigger buffering. With version checking, buffering is unnecessary — but the gap is still information, and it is information about something completely different: an event that has been lost (Case Study 1) rather than an event that is merely late.

So the semantics changed:

text proposal: a gap means "wait, more is coming" -> buffer adopted: a gap that persists past 5 minutes means "an event is MISSING" -> alarm

It has fired four times in eighteen months. Three were transient — a rebalance, and two producer retries — and resolved within the window. The fourth was a genuine loss, from a producer bug that dropped events for one order type, and it was caught in eleven minutes.

The generalizable point: a rejected proposal usually contains one component worth keeping, and it is worth going through the rejected design looking for it rather than discarding the whole document. The gap detector was the second-cheapest thing in an eleven-page proposal and the only part with value independent of the proposal's premise.

What Happened

Zero out-of-order status errors in eighteen months, and the version check has discarded 1,847 events in that period — almost all of them duplicates from the outbox relay (Case Study 1), which is the mechanism working as designed for a completely different reason than it was built.

Two things the team recorded as caveats.

The commutativity analysis must be redone when a projection is added. It is in the pull-request template for a new projection: "is this reducer commutative? if not, is it last-write-wins?" Four projections have been added since; three were sums and one was last-write-wins.

And the clickstream stream is a different case. Sessionization (Chapter 29) is a state machine, is neither commutative nor last-write-wins, and does need watermarks and buffering — which Chapter 29 built. The lesson is not "ordering is never needed"; it is that the answer differs by projection and by stream, and generalizing from one to all of them is what cost two quarters of nearly-committed engineering.

Lessons

1. The proposal generalized from one projection to the whole stream without anyone checking whether the generalization held. The counter-question — "which of our projections would actually change?" — had never been asked.

2. 🔎 The analysis is mechanical and takes an afternoon. For each reducer, ask whether f(f(s,a),b) == f(f(s,b),a). Three of four were addition.

3. The empirical confirmation is stronger than the algebra. Shuffle the stream: every sum exactly correct, 49.0% of statuses wrong.

4. Building a platform-wide guarantee for one projection's benefit is a category error — the same shape as optimizing for the wrong cost meter.

5. 📐 Partitioning and version checking are complementary, not alternatives. Partitioning is a performance measure; version checking is the correctness property, and it survives rebalances, replays, and producer retries, which partitioning does not.

6. A team that adopts only partitioning has a projection that is correct until the first rebalance — a failure that appears months later during an unrelated incident and gets attributed to it.

7. Version checking discards, which is right for last-write-wins and wrong for a projection that needs every event. Kestrel has none — and wrote that down, so the next person knows what to check.

8. 🏭 A rejected proposal usually contains one component worth keeping. The gap detector survived, with different semantics: not "wait, more is coming" but "an event is missing." It caught a genuine loss in eleven minutes.

9. The version check has discarded 1,847 events in eighteen months, almost all duplicates from the outbox relay — the mechanism working as designed for a reason it was not built for.

10. Redo the analysis when a projection is added. It is one line in a pull-request template.

11. The lesson is not "ordering is never needed." Sessionization is a state machine and genuinely needs watermarks. The answer differs by projection and by stream, and generalizing is what cost two quarters.

Questions for Discussion

1. Nobody asked which projections cared, for three weeks, while an eleven-page proposal was written. What would make that question routine at the start of a design?

2. The empirical shuffle test is more convincing than the algebra to most audiences. Is it more rigorous? What could it miss that the algebra would not?

3. Version checking discards late events. Construct a projection where that is wrong, and design what it needs instead.

4. The gap detector's semantics changed from "buffer" to "alarm." Go through a proposal your team rejected and find the component worth keeping.

5. Kestrel's PR template asks whether a new reducer is commutative. Who answers it, and what happens if they answer wrong?

6. The single-partition ceiling was "fine at Kestrel's volume." At what volume does it stop being fine, and would anyone notice before it did?

7. The original incident was caused by round-robin partitioning that "had never mattered before." How many defaults in your producers have never mattered yet?