Chapter 4 — Key Takeaways (Distributed Systems Foundations)
The page to reach for when a bug does not make sense.
The one fact everything follows from
you ──── request ────▶ remote system ✓ executed
◀ ─ ─ ✗ no response ─ ─ ─
Was it executed? YOU CANNOT TELL.
Retry → may duplicate. Don't retry → may lose. No third option.
Therefore: idempotency exists so retrying is safe · delivery semantics name which side you chose · consistency models exist because "the current value" is ambiguous · a timeout cannot distinguish a dead machine from a slow one.
Any operation you might retry needs a client-generated key identifying the LOGICAL WORK, not the attempt. e.g.
f"{dag_id}:{task_id}:{logical_date}".
Partitioning
You pay for the MAX partition, not the average — a job finishes when its slowest task does.
| Scheme | Good for | Watch out |
|---|---|---|
| Range | range queries, time series | current partition is always hot |
| Hash | even distribution | range queries touch every partition |
| Consistent hashing | changing partition counts | mostly invisible in these tools |
Skew generators: low cardinality · natural popularity. Business dimensions usually have both.
| Kestrel key | Verdict |
|---|---|
session_id |
✅ chosen — high cardinality, and keeps a session ordered |
event_type |
❌ 7 values, page_view dominates |
country_code |
❌ US ≈ 80% |
warehouse_id |
❌ 3 values |
customer_id |
⚠️ looks safe until one B2B account hits 8% |
High cardinality ≠ even distribution. 1.9M customers, one at 8.1%, job 22 min → 71 min.
Diagnosing skew: look at max ÷ median task duration. Under 3× normal, over 10× skew. In the incident it was 350×. Fixes: salt hot keys · split genuinely-different populations · alert on skew ratio. Not a fix: more partitions. All rows for one key hash to one partition regardless.
Target sizes: Parquet file 128 MB–1 GB (256 MB default) · Spark partition 100–200 MB · Kafka partition sized by throughput.
Three things break partition pruning — all written by careful engineers producing correct
results:
1. a function on the partition column · 2. a type mismatch forcing a cast · 3. an unpushable subquery.
Check PartitionFilters first. An empty bracket is the most expensive punctuation in data
engineering.
Replication
Leader-follower: sync = no loss, slower writes. Async = fast writes, followers are behind by a varying amount.
Leaderless quorum: $W + R > N$ guarantees a read sees the latest write. $N=3, W=R=2$ is the common default.
⚠️ Never record a position from a lagging replica. Any process that stores "where I got to" from a replica skips whatever arrives in the lag window.
Three fixes, all three worth having: 1. Read the watermark from the primary ← the exact fix 2. Subtract a safety margin > worst observed lag (turns loss into duplicates → idempotent writes handle it) 3. Refuse to extract when lag exceeds a threshold ← tells you something upstream is wrong
Also: bound the range on both sides. WHERE ts > lower AND ts <= upper with an upper bound you
chose beats an open range closed by whatever was read.
Consistency
CAP says: during a network partition, choose availability or consistency. CAP does not say: "pick two of three." Partition tolerance is not optional. And partitions are rare, so CAP is nearly useless for daily work.
PACELC is the useful form:
if Partition → A or C; Else → Latency or Consistency
The second clause is the one you decide every day.
Eventually consistent promises convergence if writes stop. It does not promise when, or that reads move forward in time. A value can appear, disappear, and reappear.
🧭 S3 became strongly read-after-write consistent for all operations in 2020. Pre-2021 workaround advice is obsolete and still ranks well in search. A property you designed around can change and nothing will tell you.
Delivery semantics
| Loss | Duplicates | Use | |
|---|---|---|---|
| At-most-once | possible | never | high-volume low-value telemetry |
| At-least-once | never | possible | the default in every real system |
| Exactly-once | never | never | one hop, inside a closed system |
┌───────────────────────────────────┐
│ Kafka transactional boundary │ ┌──────────────┐
│ topic A ─▶ Streams ─▶ topic B │─────▶│ PostgreSQL │
│ exactly-once genuinely HERE │ │ outside it │
└───────────────────────────────────┘ └──────────────┘
at-least-once, and your problem
At-least-once delivery + idempotent writes = effectively-once processing. Works with any transport and any sink, with no distributed transaction coordinator.
Four idempotent write strategies:
| Strategy | Best for |
|---|---|
| Delete-insert | partitioned batch loads |
| Merge / upsert on a natural key | dimensions, mutable rows |
| Partition replacement | lakehouse and warehouse tables |
| Deduplicate on read | append-only bronze |
Time
No global clock. Never order cross-machine events by wall clock · never subtract cross-machine
timestamps · never assume created_at < updated_at. Use a monotonic clock for durations.
Event time = when it happened. Processing time = when you saw it. A mobile client in a tunnel makes them differ by hours.
Event time is right for analytics — and obliges you to handle late data. A watermark declares you no longer expect events older than a given event time, plus a stated policy for later ones.
Kestrel's policy: bronze partitioned by ingest_date (what the writer knows) · silver and gold
by event date · 3-day late window, trailing 3 days reprocessed nightly · later events to a side
table · every daily aggregate carries is_final. Silently changing history is far worse than
changing it visibly.
Order is guaranteed exactly as far as the partition boundary and no further. Two things that must be ordered relative to each other must share a partition key.
The five failure modes you will actually meet
| Symptom | Fix | |
|---|---|---|
| Consumer rebalance | sawtooth lag, throughput below one consumer | poll often, process async, tune max.poll.interval.ms |
| Straggler | 199/200 tasks in 2 min, job takes 40 | fix skew, speculative execution, even partitions |
| No backpressure | memory grows until crash; queue depth only rises | bound every queue and decide what happens when it fills |
| Thundering herd | recovers, immediately falls over again | exponential backoff with jitter — the jitter is the part people omit |
| Cascading failure | one slow dependency takes down four pipelines | timeouts on every remote call, circuit breakers, bulkheads |
The ten-minute buffer audit
For any pipeline, in writing:
- Where does data buffer? List every place.
- What is the bound on each? A number. "None" = an outage waiting for a slow day.
- What happens when each fills? Block, drop, spill, or crash. If you don't know: crash.
An unbounded queue is a decision to fail later, in a worse way.