Case Study 2: Two Point One Million Messages Nobody Read

"The dead letter queue worked perfectly. It caught every bad message, kept it safely, and told absolutely nobody."

Executive Summary

A team implemented dead-lettering correctly — original bytes preserved, failure reason in headers, a dedicated topic, the consumer continuing past bad messages. Textbook.

They did not implement two things: an alert on the dead-letter rate, and a replay tool.

Fourteen months later the dead-letter topic held 2.1 million messages. A schema change eleven months earlier had made a category of events unparseable, and every one had been quietly diverted. Worse: the DLQ had inherited the cluster's default seven-day retention, so the oldest eight months of diverted messages had already aged out and were gone.

This case study is §15.10's ⚠️ callout at full length. It is here because the failure is entirely a failure of the surrounding practice rather than of the mechanism, and because the two missing pieces are a three-line alert and a fifty-line script.

Skills applied: dead-letter design (§15.10); retention as a recovery window (§15.6); alerting on rate rather than existence (§15.10); "a check nobody reads" (Chapter 1, Case Study 2).

Background

The pipeline. A consumer reading a partner-integration topic — inbound shipment events from three carriers — and writing them to bronze.

The dead-letter implementation, written in month one and reviewed:

try:
    event = deserialize(msg.value())
    validate(event)
    write_to_bronze(event)
except (DeserializationError, ValidationError) as exc:
    producer.produce(
        topic="carrier.events.dlq.v1",
        key=msg.key(),
        value=msg.value(),              # ✅ original bytes
        headers={
            "dlq.reason": type(exc).__name__,   # ✅ the reason
            "dlq.detail": str(exc)[:500],
            "dlq.source_offset": str(msg.offset()),
        })

Three of §15.10's four properties are present: original bytes, reason in headers, and the consumer continues.

The fourth is absent, and so is the alert:

    # metrics.increment("dlq", ...)     ← never written

And the topic was created with defaults:

kafka-topics --create --topic carrier.events.dlq.v1 --partitions 3
# no --config retention.ms
# → cluster default: 7 days

The Problem

Month 3. Carrier B changed a field from a string to a nested object, in a minor release, without notice — Chapter 13 §13.7's schema drift, arriving over a wire rather than in a table.

The event was still valid JSON. It failed validate() because a required field was now the wrong type.

Every Carrier B "delivered" event began dead-lettering. Roughly 4,900 a day.

Nothing alerted, because nothing was measuring. The consumer continued, as designed. Bronze kept receiving Carrier A and Carrier C events, and its volume dropped by about 31% — which would have been visible to a volume monitor and which the team did not have on that table.

Month 14. An operations analyst asked why delivery-time analysis showed no Carrier B shipments completing.

⚠️ Failure Mode — The mechanism worked and the outcome was total data loss

This is the uncomfortable part, and it is worth stating precisely.

The dead-letter queue did exactly what it was built to do. It caught every unparseable message, preserved the original bytes, recorded the reason, and let the consumer continue. If you audited the code you would pass it.

And the outcome was worse than crashing. Had the consumer crashed on the first bad message, lag would have grown, an alert would have fired within an hour, and someone would have investigated on day one. The bad events would have been sitting in the source topic, replayable.

Dead-lettering converts a loud failure into a quiet one, deliberately. That is its purpose and it is correct — if and only if something turns the quiet back into loud. Without the alert, you have chosen to fail silently, and Chapter 2's Case Study 1's principle applies unchanged: the cost of loud is hours; the cost of plausible is months.

The rule this produced at Kestrel: a dead-letter path may not be merged without an alert on its rate. Not on its existence — a DLQ with messages in it is normal — on the rate, compared to a baseline. It is three lines and it is the difference between this incident and a same-day fix.

The Analysis

The investigation, day 1, took two hours:

# How many, and since when?
kafka-run-class kafka.tools.GetOffsetShell --topic carrier.events.dlq.v1 --time -1
# partition 0: 701,204   partition 1: 698,881   partition 2: 700,915
#                                                     total: 2,101,000

2.1 million. Then the reasons:

# read the DLQ and group by header
from collections import Counter
reasons = Counter()
for msg in read_topic("carrier.events.dlq.v1", from_beginning=True):
    reasons[headers(msg).get("dlq.reason", "unknown")] += 1
print(reasons.most_common())

[('ValidationError', 2_098_412), ('DeserializationError', 2_588)]

One reason accounts for 99.9%. A single systematic cause, running for months, which is exactly the shape a rate alert detects on day one.

Then the second finding, which was worse:

kafka-configs --describe --topic carrier.events.dlq.v1
# (no retention.ms override — cluster default applies)
kafka-configs --describe --entity-type brokers --describe | grep log.retention
# log.retention.hours=168      ← 7 days

Seven days. The oldest message in the topic was from eight days earlier.

2.1 million messages was not fourteen months of accumulation. It was eight days of it. Roughly eight months and 1.2 million messages had already aged out.

🔎 Read the Plan — Two questions to ask of any queue-like thing

The team had asked how many are in here? and got a number that was misleadingly reassuring — it sounded like everything was still available.

The two questions that matter, and the second is the one people skip:

```bash

1. How many?

kafka-run-class kafka.tools.GetOffsetShell --topic X --time -1

2. How OLD is the oldest one, and what is the retention?

kafka-run-class kafka.tools.GetOffsetShell --topic X --time -2 # earliest offset kafka-configs --describe --topic X | grep retention ```

If the earliest offset is greater than zero, you have already lost messages. Offset 0 is where the topic began; anything above it means retention has deleted the beginning.

This generalizes well beyond Kafka: for any store with a retention policy, "how much is in here" and "how far back does it go" are different questions, and the second one tells you what you have already lost. It applies to log retention, to time-travel windows (Chapter 10 §10.6), to replication slots (Chapter 14 §14.6), and to bronze layers with a lifecycle policy.

The Decision

Four changes.

1. An alert on dead-letter rate, compared to a trailing baseline:

metrics.increment("dlq_messages",
                  tags={"topic": msg.topic(), "reason": type(exc).__name__})

- alert: DeadLetterRateElevated
  expr: |
    sum(rate(dlq_messages[15m])) by (topic, reason)
      > 3 * sum(rate(dlq_messages[15m] offset 1d)) by (topic, reason)
  for: 30m
  annotations:
    summary: "DLQ rate for {{ $labels.topic }} ({{ $labels.reason }}) is 3x
              yesterday. A systematic cause is usually a schema change upstream.
              Read the dlq.detail header on a sample before anything else."

Three times the trailing rate, sustained for thirty minutes. A trickle is normal; a step change is an upstream event.

And a second, absolute alert: any DLQ receiving more than 1% of its source topic's volume, at any rate. The relative alert misses a failure that was present from day one, because there is no baseline to compare against.

2. DLQ retention set explicitly, and longer than the source.

kafka-configs --alter --topic carrier.events.dlq.v1 \
  --add-config retention.ms=2592000000     # 30 days

Source topic: 7 days. DLQ: 30. The reasoning stated plainly in the topic-creation runbook:

The DLQ contains, by definition, the messages nobody has dealt with yet. It needs more time than the source, not less. The default is shorter, which is backwards.

3. A replay tool, written that week.

# platform/ingest/stream/dlq_replay.py
#
# Written BEFORE it is needed. Reading a DLQ, fixing, and re-producing is a
# routine operation and it must not be improvised during an incident.
#
#   dlq_replay.py --topic carrier.events.dlq.v1 --summarize
#   dlq_replay.py --topic ... --reason ValidationError --dry-run
#   dlq_replay.py --topic ... --reason ValidationError --since 2025-11-01 --replay

Three modes, and --summarize is the one used most: group by reason, show counts, show the earliest and latest offsets, and show the earliest timestamp against the retention so the "what have I already lost" question is answered without being asked.

4. Volume monitoring on the bronze table. The 31% drop in bronze volume would have been visible from month 3 to any freshness-and-volume check (Chapter 2's Case Study 1's control), and that table did not have one.

📐 Design Decision — Should the consumer have crashed instead?

The retrospective seriously considered removing the DLQ and letting the consumer crash on an unparseable message.

The case for crashing: it is loud, it is immediate, and the messages stay in the source topic where they are replayable. This incident does not happen.

The case against, which won: one bad message from one carrier stops ingestion for all three. A single malformed event — a genuine one-off, which happens — would halt the pipeline until someone intervened, at any hour. Over the fourteen months there were 2,588 DeserializationError messages that were genuine one-offs, and each would have been an incident.

The resolution is not either/or. Dead-letter, and alert on the rate:

  • A trickle (one-off malformed messages) → diverted, continue, no alert. Correct.
  • A step change (a systematic cause) → diverted, continue, and page someone. Also correct.

The DLQ handles the individual message; the alert handles the pattern. Choosing between them is a false dichotomy, and it is the shape of the choice in Chapter 2's Case Study 1 as well — the answer there was "fail loudly," and the answer here is "fail quietly and tell someone," which is the same principle applied to a case where availability genuinely matters.

What Happened

The 900,000 surviving messages were replayed after the validator was updated to accept both the old and new field shapes. The 1.2 million that had aged out were partially recovered — Carrier B was able to re-send four months of events from their own archive, which the team described, again, as luck.

Since then:

  • The rate alert has fired five times. Four were upstream schema changes caught within an hour. One was a genuine spike in malformed data from a carrier having an incident of their own — useful information the team would not otherwise have had.
  • The absolute alert (>1% of source volume) has fired once, on a newly built pipeline where the DLQ was receiving 8% from day one. There was no baseline, so only the absolute alert could catch it — which is exactly why both exist.
  • --summarize is run roughly weekly by whoever is on call, as a two-minute habit rather than an investigation.

The retention audit that followed found four other topics with default retention where an explicit value had been intended, including one changelog topic that should have been compacted and was silently losing state after seven days.

Lessons

  1. A dead-letter queue with no alert is a slower way to lose data. The mechanism worked perfectly; the outcome was worse than crashing.

  2. Dead-lettering deliberately converts a loud failure into a quiet one. That is correct only if something turns it back into loud.

  3. Alert on the rate, not the existence. A DLQ with messages in it is normal. And add an absolute alert too, because a relative one cannot catch a failure that was present from day one.

  4. The DLQ's retention must be longer than the source's, because its contents are what nobody has dealt with yet. The default is shorter, which is backwards.

  5. "How many are in here" and "how far back does it go" are different questions. If the earliest offset is above zero, you have already lost messages. This applies to every retention-governed store.

  6. Write the replay tool before you need it. Reading, triaging, and re-producing is routine and must not be improvised during an incident.

  7. --summarize as a weekly habit beats an investigation. Two minutes, and it makes the DLQ visible.

  8. Crash-versus-divert is a false dichotomy. The DLQ handles the individual message; the alert handles the pattern.

  9. A 31% drop in bronze volume was visible from month 3. The volume monitor from Chapter 2 would have caught this independently, and that table did not have one.

Questions for Discussion

  1. The DLQ implementation passed review with three of four properties. Write the review checklist item that would have caught the fourth.

  2. The team's alert uses 3× the trailing rate for 30 minutes, plus an absolute 1% threshold. Critique both numbers. What would you use, and what would a carrier's own bad day do to your choice?

  3. Eight months of messages aged out under a default. How many other defaults in a typical streaming deployment are load-bearing? Design the audit, and estimate how often it should run.

  4. The surviving messages were replayed after fixing the validator; the rest were partially recovered by luck. What would a deliberate safety net look like, and what does it cost?

  5. The 📐 callout calls crash-versus-divert a false dichotomy. Find another apparent dichotomy in this book that dissolves the same way.

  6. --summarize is run weekly by whoever is on call. Is a manual habit an acceptable control, or should it be automated? What is lost by automating it?

  7. This incident and Chapter 1's Case Study 2 both feature a correct mechanism that nobody was watching. Write the single question to ask of any new control before it ships.