Exercises: Distributed Systems Foundations

Most of these are diagnosis: here is a symptom, name the mechanism. That is the skill this chapter exists to build, because the symptoms never look like distributed systems problems.

Difficulty: ⭐ warm-up · ⭐⭐ standard · ⭐⭐⭐ deeper. Solutions: daggered (†) and odd-numbered problems are in appendices/answers-to-selected.md.

Part A — Warm-ups ⭐

4.1 † State the three possible outcomes of a remote request, and say which one does not exist on a single machine.

4.2 For each partition key, predict the distribution for Kestrel's clickstream and say whether it is usable: session_id, event_type, country_code, customer_id, date_trunc('hour', event_ts).

4.3 † Define replication lag in one sentence. Then explain, in two more, why it causes a watermark-based extract to lose rows.

4.4 State the quorum condition. With $N = 5$, list every $(W, R)$ pair that satisfies it, and say which you would choose for a write-heavy workload and which for a read-heavy one.

4.5 † What does CAP actually say? Name the two things people commonly get wrong about it.

4.6 Write out PACELC in full and give one Kestrel example of each of the two choices.

4.7 † Define at-most-once, at-least-once, and exactly-once. Then state, in one sentence, why this book never relies on the third.

4.8 Give an example from Kestrel where event time and processing time differ by more than an hour. What creates the gap?

Part B — Standard ⭐⭐

4.9 A nightly job's runtime is normally 22 minutes. It is now 71 minutes, roughly twice a week, on days that look otherwise unremarkable. The job reads a table partitioned by customer_id. (a) Name the most likely mechanism. (b) Write the single query that would confirm it. (c) Give two fixes and say what each costs.

4.10 † Kestrel's extract reads a replica and stores MAX(updated_at) as its watermark. Write out, as a timeline, exactly how four minutes of replication lag causes permanent row loss. Then evaluate the three fixes in §4.3 against your timeline: does each one fully solve it, and what does each cost?

4.11 An orchestrator submits a job over HTTP with a 30-second timeout and a retry policy of three attempts. Design the idempotency key. It must be: deterministic from information the orchestrator already has, stable across retries of the same logical work, and different for a legitimate re-run of the same day's data after a bug fix. That last requirement is harder than it looks — explain the tension and how you resolve it.

4.12 † Kestrel's clickstream topic has 12 partitions. Peak is 2,900 events/sec. (a) Using $L = \lambda W$ from Chapter 3 §3.5 with a p99 handling time of 300 ms, how many concurrent handlers are needed? (b) A Kafka partition is consumed by at most one consumer in a group. What does your answer imply about concurrency within a consumer, and what problem does that create for ordering? (c) If you needed 40 concurrent handlers per partition, what would you change?

4.13 Write the three-day late-arrival policy from §4.6 as a specification precise enough to implement: what gets reprocessed, when, what happens to events later than the window, and how a consumer knows whether a number is settled.

4.14 † A colleague proposes turning on Kafka's exactly-once semantics to eliminate the deduplication logic in a consumer that writes to PostgreSQL. Write a three-paragraph response that (a) concedes what exactly-once genuinely provides, (b) identifies precisely where the guarantee stops, and (c) proposes what to do instead.

4.15 For each symptom, name the mechanism from §4.7 and the first thing you would check: (a) Consumer lag rises and falls in a sawtooth; throughput is a third of what one consumer achieves alone. (b) 199 of 200 tasks complete in two minutes; the job takes 40. (c) A service recovers from a two-minute outage and immediately goes down again. (d) A long-running process's memory grows steadily until it is killed. (e) One slow dependency makes four unrelated pipelines fail.

Part C — Deeper ⭐⭐⭐

4.16 The §4.6 📐 callout sets Kestrel's late-arrival window at three days, citing 99.97% of events arriving within it. Suppose measurement showed 99.7% instead — 42,000 late events a day rather than 4,200. Redo the decision. At what completeness level would you extend the window, and what would you tell finance about numbers changing for a week?

4.17 † §4.5 claims exactly-once is achievable "within a closed system that controls both the message log and the state store." Explain precisely why external systems break it. Then describe the two-phase-commit approach that would in principle fix it, and explain why almost nobody uses it in data pipelines.

4.18 Design a skew detector for Kestrel: a scheduled check that catches a partition key going hot before it breaks the SLA. Specify what it measures, over what window, the threshold, and what it does when it fires. Then estimate its false-positive rate on a business with Black Friday in it.

4.19 † The 🧭 Version Note in §4.4 describes S3 becoming strongly consistent in 2020, obsoleting a body of practice. Find another example — in any system you know — where a property people designed around later changed. What signal, if any, told practitioners? What does the absence of such a signal imply about how you should write down assumptions?

Part D — The Kestrel Platform ⭐⭐⭐

4.20 — Increment 4: the failure inventory.

Create platform/docs/failure-modes.md. For each of the three Kestrel sources, work through the five failure modes in §4.7 and answer:

• Does this failure mode apply? If not, say why not — that is a real answer.
• What is the symptom a person would see first?
• What is the current defense, if any?
• What is the blast radius — what breaks, and does it fail loudly or plausibly?

Then, separately, complete the 🧪 Try It buffer audit from §4.7 for the platform as designed so far: every place data buffers, its bound, and what happens when it fills. Any queue whose bound you cannot state is an open item.

4.21 † Add platform/docs/idempotency-register.md: a table with one row per write operation the platform will perform, and columns for the target, the idempotency strategy (delete-insert, merge, partition replacement, deduplicate-on-read), and the key. Most rows will be blank now. Fill them in as you build, and check the register in Chapter 38 — a blank row at that point is a bug you have not found yet.

Reflection

A. §4.1 argues partial failure is a property of the universe rather than of your code. Does that change how you feel about a system that "usually works"? What would it take for you to trust a pipeline you did not write?

B. The chapter refuses to teach consensus algorithms, CRDTs, and formal consistency models. Do you agree with the exclusion? Name one you think should have been included and say what pipeline bug it would have helped you diagnose.