Further Reading: Distributed Systems Foundations

Sources are tagged Tier 1 (confident it exists, recommended without reservation) or Tier 2 (real and worth seeking, but confirm the current edition, version, or URL yourself).

This chapter compressed a large field into one chapter and refused several topics outright. This list is where the refused material lives.

The book this chapter is a compression of

• Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017). If you read one book alongside this one, read this. Chapter 5 (Replication), Chapter 6 (Partitioning), Chapter 7 (Transactions), Chapter 8 (The Trouble with Distributed Systems), and Chapter 9 (Consistency and Consensus) are §4.2 through §4.5 of this chapter, done properly and at four times the length. Chapter 8 in particular is the best available treatment of unreliable clocks and why "process pauses" break assumptions you did not know you had made. Tier 1.

A note on the second edition: a substantially revised edition has been in preparation. Check which one you are buying — the first edition remains excellent and the material in the chapters above is stable, but the sections on stream processing and cloud-native storage are the ones most changed by a decade.

On partitioning and skew

• Giuseppe DeCandia et al., "Dynamo: Amazon's Highly Available Key-value Store" (2007), SOSP. The paper that made consistent hashing, quorums, and eventual consistency part of mainstream engineering vocabulary. The section on the $W + R > N$ quorum condition is the source of §4.3's treatment, and the paper is unusually candid about the operational consequences of the choices — including that eventual consistency pushed real complexity onto application developers. Tier 1.

• The Apache Spark documentation on adaptive query execution (AQE) and skew join optimization. Modern Spark can detect and split skewed partitions at runtime, which changes the practical advice in §4.2 considerably — it does not eliminate the need to understand skew, but it does mean the first response to a straggler should be to check whether AQE is enabled. Chapter 21 §21.7 covers it. Tier 2 — version-specific; check the selector.

On replication and its lag

• PostgreSQL documentation, "High Availability, Load Balancing, and Replication" (Chapter 26 of the manual), and the monitoring views pg_stat_replication and pg_stat_wal_receiver. The primary source for the incident in Case Study 2. Read the section on how to measure lag specifically — there are several different lag figures (write, flush, replay) and they mean different things, which matters when you are setting the threshold that guards an extract. Tier 1.

• Douglas Terry et al., "Session Guarantees for Weakly Consistent Replicated Data" (1994). Where read-your-writes, monotonic reads, and the other session guarantees come from. Worth reading because these are the properties you actually want when you say "consistent," and naming them precisely is more useful than arguing about consistency in the abstract. Tier 2 — older paper, findable but not always where you expect.

On CAP and what replaced it

• Eric Brewer, "CAP Twelve Years Later: How the 'Rules' Have Changed" (2012), IEEE Computer. Brewer revisiting his own conjecture and explicitly correcting the "pick two of three" misreading that §4.4 complains about. Short, and authoritative in a way that secondary explanations are not. Tier 1.

• Daniel Abadi, "Consistency Tradeoffs in Modern Distributed Database System Design" (2012), IEEE Computer. The PACELC paper. The argument that the latency-versus-consistency trade-off in normal operation matters far more than the partition case is the single most useful correction to conventional CAP discussion, and it is what §4.4 is built on. Tier 1.

• Kyle Kingsbury's Jepsen reports (jepsen.io). Empirical testing of distributed databases' actual consistency guarantees against their marketing claims. Read two or three for systems you use. They are technical, occasionally funny, and they will permanently change how much weight you give a vendor's consistency claims — which is the disposition §4.5 is trying to instill. Tier 1.

On delivery semantics and exactly-once

• Neha Narkhede, "Exactly-once Semantics are Possible: Here's How Kafka Does it" (2017), Confluent blog. The clearest explanation of what Kafka's transactional producer actually provides. Read it for the mechanism and read it against the headline — the post is honest about the boundary in its body while the title is not, which makes it a useful object lesson in how this claim gets transmitted. §4.5's skepticism is aimed at the transmission, not at the engineering, which is genuinely good. Tier 2 — vendor blog; content moves.

• Jim Gray and Andreas Reuter, Transaction Processing: Concepts and Techniques (1992), on two-phase commit. Where the "why not just use distributed transactions" question gets its answer. 2PC works, is well understood, and is nearly absent from data pipelines because the coordinator becomes a single point of failure and the blocking behavior on coordinator failure is unacceptable. Worth knowing exists. Tier 2 — a large and expensive book; the 2PC material is summarized well in Kleppmann Chapter 9 if you want the short version.

On time

• Leslie Lamport, "Time, Clocks, and the Ordering of Events in a Distributed System" (1978), CACM. The foundational paper on ordering without a global clock, and the origin of the happened-before relation and logical clocks. Short, readable, and the source of this chapter's epigraph. It is the paper that explains why §4.6 tells you not to order cross-machine events by wall clock. Tier 1.

• Tyler Akidau et al., Streaming Systems (O'Reilly, 2018), Chapters 1–3. Event time versus processing time, watermarks, and windowing, treated rigorously. This is where the three-day late-arrival window in §4.6 comes from conceptually, and where to go before Chapter 29. Tier 1.

On the failure modes

• Marc Brooker's blog (brooker.co.za/blog) and the AWS Builders' Library (aws.amazon.com/builders-library). The best available practical writing on timeouts, retries, backoff and jitter, and load shedding — written by people operating systems at a scale where getting it wrong is measurable. The article "Timeouts, retries, and backoff with jitter" is the primary source for §4.7's insistence that jitter is not optional. Tier 2 — AWS content moves, but the Builders' Library has been stable.

• Richard Cook, "How Complex Systems Fail" (1998). Recommended in Chapter 1 and recommended again here for a different reason: proposition 3, that catastrophe requires multiple failures, is exactly Case Study 2 — a lag alert that was harmless alone and a fallback branch that was harmless alone. Four pages. Tier 1.

If you only read one thing

Read Kleppmann's Chapter 8, "The Trouble with Distributed Systems." About forty pages, and it is the chapter that converts "distributed systems are hard" from a slogan into a specific list of things that are hard and why. Unreliable networks, unreliable clocks, and process pauses — the third being the one nobody anticipates, and the reason a garbage collection pause can look exactly like a dead node.