Further Reading: Change Data Capture

Sources are tagged Tier 1 (confident it exists, recommended without reservation) or Tier 2 (real and worth seeking, but confirm the current edition, version, or URL yourself).

CDC has better primary documentation than most subjects in this book, because the projects involved are open-source and the failure modes are severe enough that people write them down.

The primary sources

• The Debezium documentation, especially the PostgreSQL connector page. Read three sections properly before your first connector: "How the connector works" (the snapshot and streaming phases), "Setting up PostgreSQL" (wal_level, publications, and — the section people skip — REPLICA IDENTITY), and "WAL disk space consumption," which is §14.6's incident described in advance by the people who built the thing that causes it. Tier 2 — versioned; check the selector.

• The Debezium FAQ on REPLICA IDENTITY. Short, and it is the fix for this chapter's second case study. If one person on your team reads one page from this list before configuring a connector, make it this one. Tier 2.

• The PostgreSQL documentation, "Logical Decoding" and "Logical Replication." Recommended in Chapter 7 and mandatory here. The "Replication Slots" section explains WAL retention precisely, and max_slot_wal_keep_size is documented with the invalidation behavior spelled out. Read it before you set the value, not after. Tier 1.

• The PostgreSQL pg_replication_slots view documentation. Every column, particularly restart_lsn, confirmed_flush_lsn, wal_status, and safe_wal_size. These are the fields the monitoring in §14.6 reads, and knowing what each means is what lets you interpret the number at 02:56. Tier 1.

On the snapshot problem

• Andreas Andreakis and Ioannis Papapanagiotou, "DBLog: A Watermark Based Change-Data-Capture Framework" (2019), Netflix. The design Debezium's incremental snapshot is based on. The watermark technique — writing low and high watermark events into the change stream itself to resolve conflicts between a snapshot chunk and concurrent changes to the same keys — is genuinely clever and is worth understanding rather than trusting. About ten pages. Tier 1.

• The Debezium blog post on incremental snapshots. The implementation write-up alongside the DBLog paper, with the configuration options and their trade-offs. Read after the paper. Tier 2 — a vendor blog; find the current URL.

On the wider pattern

• Martin Kleppmann, Designing Data-Intensive Applications, Chapter 11 ("Stream Processing"), particularly the sections on change data capture and event sourcing. The clearest available explanation of why a change log is a more fundamental representation than a table — the idea that a table is a derived view of a stream of changes rather than the other way around. It reframes Chapter 36's event-driven architecture before you get there. Tier 1.

• Martin Kleppmann, "Turning the database inside-out with Apache Samza" (2015), and the accompanying talk. The argument taken further: if the log is primary, what does a data platform look like? Provocative, and useful as a lens even where you disagree with the conclusion. Tier 2 — a talk and transcript; findable by title.

• Jay Kreps, "The Log: What every software engineer should know about real-time data's unifying abstraction" (2013). Recommended in Chapter 1 and directly load-bearing here. It is the essay that explains why an append-only ordered log sits underneath databases, replication, and stream processing alike, which is exactly what §14.1 asserts in two paragraphs. Tier 2 — has moved.

On the alternatives

• The MySQL documentation on the binary log, particularly binlog_format and why ROW is the only format usable for CDC (STATEMENT logs the SQL, not the resulting row changes, so a non-deterministic statement produces a log you cannot replay into a table). Read if your source is MySQL; the distinction is the equivalent of REPLICA IDENTITY in importance. Tier 2 — versioned, and 5.7 versus 8.0 differ.

• The Oracle GoldenGate and SQL Server CDC documentation, if you meet them. SQL Server's built-in CDC is genuinely good and works differently from log-decoding — it populates change tables that you then query, which puts it somewhere between log-based and trigger-based in §14.3's taxonomy. Tier 2.

• The MongoDB change streams documentation, covered in Chapter 12 §12.3 and relevant here as the best non-relational analogue. Comparing its resume-token model to Debezium's offset model is instructive: they solve the same problem with different guarantees about how long you can be down. Tier 2.

On the operational side

• The Kafka Connect documentation on offsets, errors.tolerance, and dead-letter queues. Debezium runs inside Connect, and Connect's failure semantics are Debezium's failure semantics. The errors.tolerance page is the source for §14.11's insistence on none. Tier 2.

• Any postmortem you can find of a replication-slot disk-full incident. They exist, they are written by people who have had a bad night, and the consistency of the story across organizations is the finding. Search for "replication slot disk full postgres postmortem." Tier 2 — blogs and incident reports, quality varies, and the shared shape is the point.

If you only read one thing

Read the Debezium PostgreSQL connector's "WAL disk space consumption" section. It is perhaps two pages, it describes exactly the incident in this chapter's first case study, and it is written by the people who built the tool that causes it.

Then set max_slot_wal_keep_size before you create your first slot. The whole of §14.6 reduces to that one action, and it is the difference between a broken pipeline and a stopped storefront.

If you have a second hour, read the DBLog paper. It is the only place the initial-snapshot problem — which decides whether CDC is feasible on a large table and which every tutorial skips — is treated as the hard problem it is.