Further Reading: Incremental Processing and Slowly Changing Dimensions

Sources are tagged Tier 1 (confident it exists, recommended without reservation) or Tier 2 (real and worth seeking, but confirm the current edition, version, or URL yourself).

This chapter's two halves have very different literatures. Slowly changing dimensions are thoroughly, permanently documented — Kimball settled the vocabulary in the 1990s and it has not moved. Incremental processing is not, and most of what exists is vendor documentation plus practitioner writing that assumes a particular engine. The gap is worth knowing about: if you find yourself unable to locate a canonical treatment of watermark correctness, it is not because you are searching badly.

On slowly changing dimensions

Ralph Kimball and Margy Ross, The Data Warehouse Toolkit (3rd ed., Wiley, 2013), Chapter 5. The source of the type numbering, and still the best treatment. Read it for the reasoning rather than the taxonomy — Kimball is explicit that the type is chosen per attribute according to how the business wants history reported, which is the point §20.8 makes and which almost every secondary summary loses in the process of tabulating the types. The book also covers Types 4, 5, 6, and 7, which this chapter dismisses fairly briskly; read Kimball before deciding this chapter was right to. Tier 1.
Kimball Group Design Tips, the archived newsletter. Short, specific, and several of them address exactly the problems in this chapter's Case Study 1: what to do when a dimension's attributes change faster than expected, and when a mini-dimension is the answer rather than a symptom. The mini-dimension tips are worth reading precisely because §20.8 is dismissive of Type 4 and you should hear the other case. Tier 2 — the archive has moved more than once; search by title.
The dbt documentation on snapshots. The check versus timestamp strategy discussion, the check_cols configuration, and — new in 1.9 — dbt_valid_to_current and the YAML configuration format. The documentation offers check_cols: all without a warning attached, which is worth noticing as you read: Case Study 1 is what that option does in a real project, and the documentation's neutrality about it is exactly the kind of gap a practitioner has to fill from experience. Tier 1 — versioned; read the 1.9+ page.

On incremental correctness

Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017), Chapters 7 and 11. Chapter 7 on transaction isolation is the why behind §20.7: the reason a commit can become visible after a row with a later timestamp is snapshot isolation working correctly, not a bug in your database. Chapter 11 on stream processing covers exactly-once semantics and the relationship between idempotency and delivery guarantees. This is the single most useful book for understanding why incremental processing is hard, and it never mentions dbt. Tier 1.
The Apache Beam documentation on watermarks and triggers. Beam's model is the most careful public treatment of "how do I know I have seen everything for a window," and its vocabulary — event time versus processing time, watermarks, allowed lateness — is the vocabulary this chapter's lookback window is a crude approximation of. Reading it will make you slightly dissatisfied with INTERVAL '3 days', which is the correct reaction. Chapter 29 §29.4 returns to this properly. Tier 1.
Tyler Akidau, Slava Chernyak, and Reuven Lax, Streaming Systems (O'Reilly, 2018). The book version of the Beam model, and the chapters on watermarks are the definitive treatment of late data. Chapters 2 and 3 are the ones relevant here, and they are readable independently of the streaming material. Tier 1.
The dbt documentation on incremental models, including incremental_strategy, incremental_predicates, on_schema_change, and microbatch. Read the microbatch page even if you are not going to use it, because it names the assumptions a hand-written incremental model makes implicitly, and naming them is most of the value. Tier 1 — 1.9+ for microbatch.

On the specific failures

Anything careful on clock skew in distributed systems. Kleppmann's Chapter 8 ("The Trouble with Distributed Systems") covers monotonic versus time-of-day clocks and why timestamps from different machines cannot be ordered. Case Study 2 is a single-database instance of the general problem, and the general problem is worse. Tier 1.
The PostgreSQL documentation on transaction isolation and on logical replication slots. The mechanism in Case Study 2's diagram — a transaction visible only at commit, carrying a timestamp from when it began — is documented behaviour, and the logical-decoding documentation explains why a replication slot delivers changes in commit order rather than timestamp order. That ordering is the fix Chapter 14 §14.5 recommends, and the reason an LSN is a better watermark than a timestamp whenever you can get one. Tier 1.
Google's Site Reliability Engineering, on alerting and on error budgets. Case Study 2's argument that a reconciliation tolerance is a budget for permanent error is the error-budget idea applied to data quality rather than availability, and the SRE treatment is where that framing comes from. The useful transfer is the insistence on stating budgets numerically and revisiting them, rather than treating a threshold as a fact of nature. Free online. Tier 1.

Practice

code/scd_lab.py in this chapter. Runs every failure here against SQLite in about a second — the append that is not idempotent, the watermark that loses a row, the lookback that duplicates, the three invariants, the NULL sentinel, and the check_cols explosion. Standard library only, no warehouse required. Modify the fixtures rather than only running it.
Build an SCD2 dimension by hand once, without a snapshot macro. Exercise 20.15. Snapshot tools are good and they hide exactly the mechanics that go wrong, and there is no substitute for having written the UPDATE ... SET valid_to yourself and then discovered the row you forgot to close.

A note on what to be skeptical of

Tutorials showing WHERE updated_at > (SELECT MAX(updated_at) FROM {{ this }}) without a lookback. This is the most-copied incremental pattern in existence, it appeared in dbt's own documentation for years, and Case Study 2 is what it does. It is not wrong as an illustration of the mechanism; it is wrong as a thing to run.

The general form of the warning: incremental processing advice is almost always written from the perspective of cost and runtime, and almost never from the perspective of completeness. When you read a piece on making a model faster, ask what it assumes about arrival order — and note that the question is usually not addressed at all, rather than addressed badly.