Case Study 1: The Slot That Stopped the Storefront

"A data pipeline took down checkout. Not slowed it — stopped it. And the component responsible had been switched off eleven weeks earlier."

Executive Summary

A company evaluated Debezium, created a replication slot, decided not to proceed, and shut down the connector. Nobody dropped the slot.

Eleven weeks later the primary database's disk filled with retained write-ahead log. PostgreSQL stopped accepting writes at 02:14 on a Tuesday. Checkout returned errors for fifty minutes. The fix is one SQL statement and it took forty of those fifty minutes to find.

This case study is §14.6 as an incident. It is included because the failure mode is counter-intuitive — a component that is not running caused an outage — and because the three practices that prevent it are trivial and almost never in place before the first occurrence.

Skills applied: replication slots and WAL retention (§14.6); max_slot_wal_keep_size (§14.6); blast radius (Chapter 3 §3.6); ownership as a precondition for existence (Chapter 9 §9.7).

Background

The evaluation. In April, a data engineer spent two weeks evaluating Debezium against the company's PostgreSQL 15 primary. The evaluation was competent: a slot, a publication, a connector, a test topic, and a written comparison against the existing batch extract.

The conclusion was to defer — the batch extract was adequate, and the team did not have capacity to operate Kafka Connect. A correct decision, and exactly the reasoning §14.10's fourth condition recommends.

The shutdown. The connector was stopped, the Kafka topics deleted, the Connect worker torn down.

What was left behind:

SELECT slot_name, plugin, active FROM pg_replication_slots;

   slot_name    | plugin  | active
----------------+---------+--------
 debezium_poc   | pgoutput| f

One row. Inactive, retaining WAL, owned by nobody, named after a thing that no longer existed.

Why nobody noticed for eleven weeks. The database's WAL volume is a fraction of its total disk, and disk usage grew slowly and monotonically — which is what disk usage does. The disk alert was set at 90%, and the growth from 34% to 89% took ten weeks.

The Problem

week 0   connector shut down. slot retains from LSN X.  WAL 4 GB, disk 34%
week 2   WAL 31 GB   disk 41%
week 5   WAL 96 GB   disk 58%
week 8   WAL 178 GB  disk 74%
week 10  WAL 241 GB  disk 89%   ← disk alert fires
week 11  Monday 23:50  WAL 268 GB  disk 97%
         Tuesday 02:14 DISK FULL
                       ↓
         PANIC: could not write to file "pg_wal/...": No space left on device
         → the database cannot write WAL
         → it cannot commit
         → EVERY WRITE FAILS

The disk alert had fired, at week 10, and had been triaged as "disk usage is growing, we should provision more" and put on the backlog. That triage was reasonable given the information available — disk usage growing steadily on a database that is growing steadily is not obviously alarming.

What nobody did was ask what was growing. The pg_wal directory was 241 GB of a 270 GB volume, and the tables were a small fraction of the total.

⚠️ Failure Mode — Monotonic growth is not obviously a problem

A disk filling steadily over ten weeks looks exactly like a database with a growing dataset, which is what everyone expects a database to be.

The distinguishing question is not "is disk usage growing?" but "which directory?" — and it takes one command:

bash du -sh /var/lib/postgresql/data/* | sort -h | tail -5

text 1.2G ./pg_stat 8.4G ./base 241G ./pg_wal ← 96% of the volume, and it should not be

pg_wal should be roughly constant. It is a circular buffer sized by max_wal_size and checkpoint frequency; it grows unboundedly only when something is preventing recycling — a replication slot, an archiving failure, or a stalled standby.

The alert to add is not "disk is 90% full." It is "pg_wal exceeds N × max_wal_size." That fires in week two rather than week ten, it is unambiguous about the cause, and it names the mechanism rather than the symptom.

This generalizes: a symptom alert tells you something is wrong; a mechanism alert tells you what. Chapter 7's Case Study 1 reached the same conclusion with dead-tuple ratio versus query latency.

The Analysis

02:14. Writes fail. Alerts fire across every service.

02:16. On-call — a backend engineer, not on the data team — is paged. The error names a disk-space problem, so they look at the disk.

02:19. Disk is full. They start looking for what to delete, which is the correct instinct and the wrong target: nothing in pg_wal can be deleted safely by hand, and the temptation to try is significant and dangerous.

⚠️ Failure Mode — The recovery that destroys the database

An engineer facing a full pg_wal under pressure will consider deleting WAL files. Do not.

Deleting WAL files that the database still needs makes it unrecoverable — the database cannot replay to a consistent state and will refuse to start, and you are now restoring from backup instead of dropping a slot.

The safe actions, in order:

  1. Find out what is retaining WAL. SELECT slot_name, active, pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) FROM pg_replication_slots; Also check archive_command failures — a failing archive command retains WAL identically.
  2. Drop the offending slot if it has no owner or the consumer is gone: SELECT pg_drop_replication_slot('...'); WAL is recycled within seconds.
  3. Add disk if you cannot identify the cause. Buying time is legitimate.
  4. Never delete files from pg_wal by hand.

Put those four lines in the runbook. The person who meets this will not be a data engineer — it presents as a database problem — and the difference between step 2 and the wrong instinct is fifty minutes versus a restore.

02:19–02:54. Thirty-five minutes of investigation. The on-call engineer had never heard of a replication slot. They escalated at 02:41; a database-experienced engineer joined at 02:49 and asked about slots immediately.

02:56. The slot is found. debezium_poc — a name that meant nothing to anyone still on the team. The engineer who created it had changed teams in June.

Two minutes of hesitation, which is worth recording: is it safe to drop? Nobody could immediately confirm that nothing was consuming it. active = false was the evidence, and it took a moment for someone to be confident that inactive plus eleven weeks meant abandoned.

02:58. SELECT pg_drop_replication_slot('debezium_poc');

03:02. WAL recycled, disk at 39%, writes resumed.

Total: 48 minutes of no writes. For an e-commerce business at 02:14 on a Tuesday, that is the cheapest possible timing, and it was luck.

The Decision

Four changes, and the first would have prevented it entirely.

1. max_slot_wal_keep_size set on every environment.

ALTER SYSTEM SET max_slot_wal_keep_size = '64GB';
SELECT pg_reload_conf();

A slot exceeding 64 GB of retained WAL is invalidated. The CDC stream breaks and requires a re-snapshot. That is the correct trade, made deliberately: a broken pipeline is a data team problem; a full disk is a company problem.

The team debated the value. 64 GB at their WAL generation rate is about 40 hours — enough to survive a weekend outage of a consumer, not enough to survive a two-week holiday. They chose weekend coverage on the reasoning that a consumer down for two weeks should break.

2. Slots are named with an owner and a purpose.

cdc_orders_dataeng. Not debezium_poc, not test, not slot1. The name is the only metadata a slot carries, and at 02:56 it is the only thing available.

Enforced by a naming check in the weekly report (change 3), not by a constraint — PostgreSQL has no mechanism to enforce it.

3. A weekly slot report, sent to engineering leadership:

SELECT slot_name,
       active,
       COALESCE(split_part(slot_name, '_', 3), 'UNKNOWN') AS owner_team,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn))
         AS retained,
       CASE WHEN NOT active THEN 'INACTIVE — investigate' ELSE 'ok' END AS state
  FROM pg_replication_slots
 ORDER BY pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) DESC;

Any slot with no identifiable owner is dropped after one week's notice. This is Chapter 9's Case Study 2's archive discipline — ownership as a precondition for existence — applied to a different resource, and it works for the same reason.

4. Two alerts, on the mechanism rather than the symptom:

  • pg_wal size exceeds 4 × max_wal_size — fires in week two, unambiguous about the cause.
  • Any slot active = false for more than one hour — fires at the crash.

5. The runbook, with the four safe actions from the ⚠️ callout, filed under "database disk full" rather than under "CDC," because the person who meets this will not be a data engineer.

📐 Design Decision — Should a data pipeline be able to break the source database at all?

The incident prompted a real architectural question: is it acceptable for the data platform to have a failure mode that stops the storefront?

The case for eliminating it: the blast radius is unacceptable. Use trigger-based CDC, or batch, or replicate to a dedicated standby and run CDC against that — so the worst case is losing the replica.

The case for accepting it, with controls, which won: every one of the alternatives has costs that are certain, against a risk that is now bounded. Trigger-based CDC doubles the write cost of the source tables, permanently. A dedicated standby is another instance to operate and pay for. Batch cannot see deletes.

With max_slot_wal_keep_size set, the worst case is no longer an outage — it is an invalidated slot and a re-snapshot. The risk was not eliminated; it was converted into a different, smaller risk, deliberately, with the trade written down.

What that gives up: the possibility of a CDC stream breaking at an inconvenient moment, requiring a re-snapshot of a large table. That has since happened once, took four hours, and nobody outside the data team noticed — which is the outcome the trade was designed to produce.

What Happened

In the two years since:

  • max_slot_wal_keep_size has fired once, when a Connect worker was down over a long weekend. The slot was invalidated, orders was re-snapshotted over four hours, and no other system was affected.
  • The weekly slot report has found three abandoned slots — two from testing, one from a decommissioned service. All dropped before they retained more than a few gigabytes.
  • The pg_wal size alert has fired twice, both times correctly: once for the long-weekend outage, once for a failing archive_command, which is the other cause of unbounded WAL growth and which the alert caught because it watches the mechanism rather than the actor.

That last one is the most satisfying result. An alert designed for one cause caught a different one with the same mechanism, which is what distinguishes a mechanism alert from a symptom alert.

One thing did not change, and the team's note about it is honest:

"We still cannot say, at 02:14, whether a given inactive slot is safe to drop without asking someone. The naming convention helps and it is not proof. We accepted that."

Lessons

  1. A component that is not running can cause an outage. The slot outlived the connector by eleven weeks.

  2. A PostgreSQL primary with a full WAL disk stops accepting writes. Not degraded. Stopped.

  3. Monotonic growth is not obviously a problem, which is why the week-10 disk alert was reasonably triaged as capacity planning.

  4. Alert on the mechanism, not the symptom. "pg_wal exceeds 4 × max_wal_size" fires in week two and names the cause; "disk 90% full" fires in week ten and names nothing. It also caught a completely different problem — a failing archive command — for free.

  5. max_slot_wal_keep_size converts an outage into a broken pipeline. Set it before the first connector. It is the single control that would have prevented this.

  6. Never delete files from pg_wal by hand. It converts a fifty-minute incident into a restore. Put the four safe actions in the runbook.

  7. File the runbook under the symptom, not the cause. The person who meets this is a backend engineer looking at a full disk, not a data engineer looking for a slot.

  8. A slot's name is the only metadata it carries, and at 02:56 it is all you have. Owner and purpose, enforced by a report rather than by hope.

  9. Ownership as a precondition for existence works here as it does for datasets. Any slot with no identifiable owner is dropped after notice.

Questions for Discussion

  1. The decision to defer Debezium was correct and the cleanup was incomplete. Design the checklist that accompanies a "we evaluated it and decided no" outcome. What else besides slots does it need to cover?

  2. The week-10 disk alert was triaged as capacity planning. Was that wrong? What information would have changed the triage, and whose job is it to provide it?

  3. max_slot_wal_keep_size was set at 64 GB — about 40 hours, chosen to cover a weekend. Argue for a week instead, and for four hours. What does each optimize for?

  4. The team accepted that they still cannot confirm a slot is safe to drop at 02:14. Design the mechanism that would let them. What does it cost, and is it worth it?

  5. The 📐 callout accepts a bounded risk to the source database rather than eliminating it. When would you eliminate it instead? What kind of business or regulatory context changes the answer?

  6. The pg_wal alert caught a failing archive command it was not designed for. Find another mechanism alert in this book with the same property, and say what makes an alert generalize.

  7. Forty of fifty minutes went into diagnosis by someone who had never heard of a replication slot. Is the fix a runbook, training, or a different alert? Defend one.