Part III — Ingestion
Chapters 13–17
Ingestion is where you meet systems you do not control.
Everything up to this point has been about decisions you get to make. Ingestion is about a transactional database that was designed for checkout latency and not for you, a clickstream from mobile clients that go offline in tunnels and resend, a carrier API that rate-limits you at a threshold documented nowhere, and a product team that will rename a column on a Tuesday afternoon without malice and without notice.
This is the part of the lifecycle that generates the most incidents. Not because it is intellectually hard — the code is often simple — but because it is the boundary where your assumptions meet someone else's reality.
Chapter 13 is batch ingestion: extracting from databases, files, and legacy systems on a
schedule. Full loads versus incremental, watermarks and their failure modes, why the naive
updated_at > last_run pattern loses rows, and how to extract from a production database without
becoming an outage.
Chapter 14 is change data capture: reading the database's own replication log with Debezium instead of querying the tables. CDC solves the deletes problem and the hard-deletes problem and the "row changed twice between polls" problem, and it introduces an operational burden that this chapter is honest about.
Chapter 15 is Apache Kafka: producers, consumers, topics, partitions, offsets, consumer groups, and delivery semantics. Kafka is the current best answer to a durable question — how to decouple producers from consumers with a replayable, ordered log — and this chapter teaches the question first.
Chapter 16 is API ingestion, the least glamorous and most universally required skill in the part. Pagination that changes shape at page 50, rate limits expressed in four incompatible ways, authentication that expires mid-job, and retries that must distinguish "try again" from "stop immediately."
Chapter 17 is schema evolution and data contracts: the mechanism by which a producer and a consumer agree on what will be sent, versioned, enforced, and negotiated. This is the chapter that turns "they broke us again" from a recurring event into a caught test.
What you will have built by the end of Part III: a watermarked batch extractor, a Debezium CDC
pipeline into Kafka, a clickstream producer and consumer with a dead-letter path, an API ingester
that survives rate limits, and a schema registry with an enforced contract on orders.
The single idea to carry out of this part: every ingestion pipeline will run twice. Retries, manual reruns, backfills, duplicate scheduler triggers, and consumer rebalances all guarantee it. Design for that first and the rest of ingestion becomes ordinary engineering.