Further Reading: Batch Ingestion

Sources are tagged Tier 1 (confident it exists, recommended without reservation) or Tier 2 (real and worth seeking, but confirm the current edition, version, or URL yourself).

Batch ingestion has less written about it than it deserves, precisely because the code looks simple. Most of what follows is either about the source systems you extract from or about the properties — idempotency, restartability — that make an extract survivable.

On the source side

• The PostgreSQL documentation, "Concurrency Control" (Chapter 13 of the manual). Recommended in Chapter 7 and directly relevant to §13.4's first lie: the distinction between the value an application writes and the moment a transaction becomes visible is a snapshot-isolation property, and reading it once makes the whole watermark discussion mechanical rather than mysterious. Tier 1.

• The PostgreSQL pg_stat_replication and pg_last_xact_replay_timestamp documentation. The lag guard in §13.12 is built on these. Worth reading the distinctions between write, flush, and replay lag — they mean different things, and the one you want for an extract guard is replay. Tier 1.

On idempotency and reproducibility

• Maxime Beauchemin, "Functional Data Engineering — a modern paradigm for batch data processing" (2018). Recommended in Chapter 2 and most directly applicable here. Its central argument — that batch pipelines should be pure, idempotent functions over immutable partitions — is the theoretical justification for §13.10's backfill rules and §13.11's restartability. The essay's practical advice about partition-based reprocessing is the design this chapter's extractor implements. Tier 2 — Medium; has moved.

• The Airflow documentation on "Data Interval," "Catchup," and "Backfill." Even before Chapter 24, worth reading for the vocabulary: Airflow's model of a data interval separate from a run time is exactly §13.4's distinction between the range you are extracting and the moment you are extracting it, and having the vocabulary makes the distinction easier to hold. Tier 2 — versioned, and Airflow 2 versus 3 differ here.

On the patterns

• Gregor Hohpe and Bobby Woolf, Enterprise Integration Patterns (Addison-Wesley, 2003). Old, and the patterns are undiminished: Idempotent Receiver, Guaranteed Delivery, Message Sequence, Dead Letter Channel, and Claim Check are all in this chapter under different names. The book predates every tool in this chapter and describes every problem in it. Tier 1.

• Ralph Kimball and Joe Caserta, The Data Warehouse ETL Toolkit (Wiley, 2004). The companion to the modelling book. Dated on tools and unusually thorough on exactly the material in this chapter: change data capture strategies, deduplication, error handling, and the "34 subsystems of ETL," which is a genuinely useful completeness checklist even now. Read it as a catalogue rather than a tutorial. Tier 2 — out of print in places; check availability.

• Martin Fowler's writing on the Outbox pattern and on dual writes. Directly relevant to §13.5's conclusion that when a source cannot tell you what changed, the application should. Chapter 36 §36.4 covers it properly; Fowler's treatment is the clearest short version. Tier 2.

On files and legacy sources

• The Singer specification (singer.io) and the Meltano project's documentation. Singer defines a JSON protocol for taps (extractors) and targets (loaders), and its STATE message is a standardized watermark. Worth reading even if you never use it: it is a well-considered formalization of exactly the state management §13.4 describes, and seeing someone else's design sharpens your own. Tier 2 — the ecosystem has shifted; the spec is stable.

• The Airbyte protocol documentation. A more recent formalization of the same problem, with a different take on incremental state and schema evolution. The comparison with Singer is instructive. Tier 2 — actively changing.

• IBM's documentation on COBOL copybooks and packed decimal, if you meet a mainframe. Deeply unglamorous and the only way to parse a fixed-width EBCDIC extract correctly. You will know if you need this. Tier 2.

On the failure modes

• Kyle Kingsbury's Jepsen reports (jepsen.io). Recommended in Chapter 4 and relevant here for a narrower reason: several reports demonstrate sources whose timestamp or version semantics differ from their documentation, which is §13.4's second lie with empirical evidence. Tier 1.

• The "birthday problem" in any elementary probability text. The mechanism behind this chapter's first case study: the probability of two rows sharing a timestamp second scales roughly with the square of the write rate, which is why a 6.28× peak produces far more than 6.28× the collisions. Understanding this once makes §13.8's "what breaks at 10×?" question answerable rather than rhetorical. Tier 1.

On testing what you cannot observe

• Any material on load testing and on chaos engineering, particularly the principle of deliberately generating the conditions you cannot otherwise observe. This chapter's first case study concluded that a load-scaled failure can only be found by generating the load, and the chaos-engineering literature is where that idea is developed properly. Netflix's writing on Chaos Monkey and the "Principles of Chaos Engineering" statement are the usual entry points. Tier 2.

If you only read one thing

Read Beauchemin's "Functional Data Engineering." Twenty minutes.

Its idea — that a batch job should be a pure function from an immutable input partition to an immutable output partition — is the single frame that makes idempotency, restartability, backfills, and reproducibility stop being four separate concerns and become one. Every rule in §13.10 and §13.11 follows from it, and once you have the frame you can derive the rules rather than remembering them.

Then, before you write your next extract, read the Singer STATE message specification — three paragraphs — and notice that a well-designed protocol makes the watermark an explicit, transmitted, inspectable object rather than a row in a table someone might forget to commit.